► Integrations Bulk API Training ► Bulk API

Bulk API - Export

The Export API makes it possible to extract records in batches from 4me.

The most common use of this API is to automatically load data from one or more 4me accounts into a data warehouse or to load time entries and request info into another time registration system or into a billing tool.

With the Export API you can automatically prepare a UTF-8 encoded comma-separated values (CSV) file for each type of 4me record. These CSV files can then be picked up by a script that imports them into the data warehouse. The functionally is similar to the Export feature that is available to account administrators within the 4me user interface.

The Components of a Bulk Export API Request

The Bulk API is based on REST principles and is optimized for working with large sets of data. Bulk export is performed by initiating an export job, and then waiting for the job to make the export file available. These jobs are executed asynchronously, and can be polled to retrieve the status of the export.

A Bulk API request consist of 4 components: the url (the endpoint), the HTTP Verb (the method), the headers and the parameters.

The Bulk Export API Endpoints

These are the endpoints of the Bulk Export API:

Production enviroment: https://api.4me.com/v1/export
QA environment: https://api.4me.qa/v1/export
Demo environment: https://api.4me-demo.com/v1/export

The Bulk API Verbs

To initiate a Bulk export API you need to perform a POST Request. Polling the export progress is done with a GET Request.

The HTTP Headers

Each request to the 4me REST API must supply two headers to specify your personal access token and to specify the 4me account you need access to:

Authorization: Bearer <personal-access-token>
X-4me-Account: <accountID>

Be aware that you cannot export records from multiple 4me accounts with one command.

The Parameters

When you initiate an export you can define 2 parameters: type and from.

You need to POST these parameters using the Content-Type multipart/form-data according to RFC 2388. This sounds more complex than it actually is: in fact you will emulate a filled-in form in which the user has filled in the type and eventually the from parameter and then presses the submit button.

In CURL you can simply use the -F option. Within double quotes you can specify the parameter with the selected value(s). Examples:
-F "type=tasks,project_tasks"
-F "from=20220524T00:00:00"

The type parameter is mandatory: you can define one or more record types that you want to be included in the export. When more than one type is specified the export file will be of filetype .zip.

For some record types the number of records can be huge. That’s why most of the time an export integration is implemented as an incremental process: with the from parameter you have the possibility to tell the Export API that you only want to receive the records that were created or updated after a specific date and time. That will ensure that the size of the CSV files remains manageable, even when then total number of records (like requests) continues to grow in the account(s).

An export without the from parameter is called a Full Export. An export with the from parameter is called a Partial Export

Export Limits

A Full Export can be performed multiple times per day, unless the full export contains more than 10,000 records. If the export exceeds 10,000 records, the full export for that record type can be performed only once per day.

A Partial Export has unlimited frequency but is limited to a maximum of 2 months backwards.

Initiating an Export

In the following exercises you will perform a partial and a full export.

Exercises:

Can you export all requests of the Widget Data Center (id = wdc) account that were created or updated on or after the first of the previous month ?

Make sure you add the ‘Widget Data Center’ account to the scope of your personal access token and that you have added Export – Create to the scopes. Use the following CURL statement :
curl -X POST -i -H "Authorization: Bearer <personal-access-token>" -H "X-4me-Account: wdc" -F "type=requests" -F "from=yyyymmdd" "https://api.4me-demo.com/v1/export" with yyyymmdd the first of the previous month (for example 20210201).

Can you export all sites, teams and organizations of the ‘Widget N. America – IT’ account (id = wna-it)?

Make sure you add the ‘Widget N. America – Information Technology’ account to the scope of your personal access token. Use the following CURL statement :

curl -X POST -i -H "Authorization: Bearer <personal-access-token>" -H "X-4me-Account: wna-it" -F "type=sites,teams,organizations" "https://api.4me-demo.com/v1/export"

By not specifying a date, all records of the specified types will be exported.

When the initiation of the exports was successful you can find the results in the 4me specialist user interface. Login as howard.tanner@widget.com to the Widget Data Center account and go to the Settings console (5th icon in the top header bar). Look for the Export menu. You should find an entry for the partial export of requests that you have created via the Bulk API. When you login to the ‘Widget North-America, Information Technology’ account you should find the full export entry for sites, teams and organizations.
Export results in the 4me specialist user interface

Polling the Export Process and Downloading the File

So far so good. But your integration will need to take it at least one step further and download (and probably process) the export file. This is done with the polling. On regular intervals your integration will check if the export file is ready to be downloaded. When your export initiation was successfull you received a unique token in the response. With a GET request and the token specified in the endpoint you can poll the export process.

Production enviroment: https://api.4me.com/v1/export/token
QA environment: https://api.4me.qa/v1/export/token
Demo environment: https://api.4me-demo.com/v1/export/token

The polling response provides a state field. When the state is done the response will contain an URL that your integration can use to download the file.

Exercise:

Can you export all configuration items of the ‘Widget Europe – IT’ account (id = weu-it)? Define also the CURL command to poll the export progress.

Use the following CURL statement to initiate the export of the configuration items:

curl -X POST -i -H "Authorization: Bearer <personal-access-token>" -H "X-4me-Account: weu-it" -F "type=cis" "https://api.4me-demo.com/v1/export"

You can poll the progress of the export with the following CURL statement:

curl -X GET -i -H "Authorization: Bearer <personal-access-token>" -H "X-4me-Account: weu-it" "https://api.4me-demo.com/v1/export/<export_token>"

When the export is done check if you can download the file. Login as Howard Tanner to the ‘Widget Europe – Information Technology’ account, go to the settings console, select the Export menu and download the file with the configuration items.

You can also use the URL that is part of the response of the polling request. The response is in a JSON format and uses unicode for special characters. The ampersand  &  character is encoded as \u0026. A URL does not support this unicode format: you will need to replace \u0026 with  &  to get a valid URL.

Next Topic