REST API

Prerequisites

• An understanding of JSONPath and JSONPath expressions.

• You are assigned the Team Administrator, Team Collaborator, or Pipeline Administrator role in Hevo to create the Pipeline.

Configuring REST API as a Source

Perform the following steps to configure REST API as a Source in Hevo:

1. Click PIPELINES in the Navigation Bar.

2. Click + CREATE in the Pipeline List View.

3. In the Select Source Type page, select REST API.

4. Provide REST endpoint details in the Configure your REST API Source page:

   

   • Pipeline Name: A unique name for the Pipeline.

   • Method: The API method. This can be GET or POST.

   • URL: The fully qualified URL endpoint for the REST API. The endpoint must contain HTTP or HTTPS, whichever is applicable.

   • Request Body: The POST body for the REST API, if the API method is POST. The body must be a valid JSON or Form Data in a key-value pair format.

   • Authentication:

     • Basic Auth: Optional Username and Password for Basic Authentication to be provided when the API requires it.

       • Basic Auth User: Your API username or API key.

       • Basic Auth Password: Your API password or API secret.

     • OAuth 2.0:

       • Redirect URL: The URL where the Source authorization server sends the auth code which can then be exchanged for the access token.

       • Select Access Token: The access token used by Hevo to make API requests. You can:

         • Select an existing token that you created.

         • + Create New Token: A new access token for making API requests.

           • Token Name: A name for the access token.

           • Client ID: The ID retrieved from your Source application, to identify Hevo.

           • Client Secret: The secret key retrieved from your Source application, to authenticate Hevo to read your Source data.

           • Auth URL: The URL for the Source API authorization server, to retrieve the auth code.

           • Token URL: The URL for the Source API authentication server, to exchange an auth code for an access token.

           • Scopes: A set of permissions that you want to provide Hevo for accessing the Source data.

   • Headers: Custom headers needed for the API call.

   • Query Params: Query parameters to be appended to the URL while making a request. These are given as a key-value pair. Hevo supports the following type of query parameters:

     • Text: A static query parameter.

       • Key: The name of the parameter.

       • Value: The value of the parameter.

       For example, suppose you want to find the total number of records in an API response. Then, specify the Key as total and the Value as true.

     • Date: A dynamic Date type query parameter.

       • Key: The name of the parameter.

       • Format: The format in which the string value for the date must be generated. Hevo generates DateTime values according to Java’s SimpleDateFormat.

       • Offset Unit: The unit in which DateTime is generated.

       • Offset Value: A numerical value for the selected unit.

       The Offset Unit and Offset Value are optional parameters. They are used to generate DateTime values relative to the time at which the API call is made.

       For example, to generate a value that is 24 hours ago, provide the Offset Unit as Hours and the Offset Value as -24. A positive offset value generates future date times.

   • Data Root: The JSONPath expression to the data that you want to replicate. Hevo suggests possible data roots based on the response received from the URL specified in the connection settings.

     The Data Root field displays the first suggested data root, and the Sample Response from API section displays the parsed response from it. You can change the data root if you want and click CHECK PARSED RESPONSE to view the parsed data. For example, the image below displays the first suggested data root $.* and the parsed response for it.

     

     Note: The data root property can only contain a JSONPath expression that returns an array or an object, and each element of the data root should return a JSON object.

     Sample Response from API: A view-only field that displays the raw and parsed sample responses from the API. You can use the responses to identify your data root and view the Events present in the data root respectively.

     • Raw Response: The API response in JSON format, based on the URL specified in the connection settings. It also displays the data root for the Events. For example, in the image below, the data root is friends.

       

     • Parsed Events: The API response parsed based on the data root specified above. It displays the list of Events to be ingested from the Source.

       

     Read JSONPath Expression Examples for more information on writing valid expressions. Also, read Troubleshooting the REST API.

   • Pagination: The process of dividing the API response into pages. Default selection: No Pagination. Hevo supports the following types of pagination:

     • No Pagination: This option attempts to fetch the entire result set of the API in a single call. However, depending on the way pagination is implemented within the API, the entire response may be fetched in one page or be divided into multiple pages. In the latter case, you need to select a different pagination option to retrieve the entire result set.

       For example, consider a REST API that fetches a list of 1000 books and their reviews in a single call. If you configure this API as a Source with No Pagination, then, on each Pipeline run, the API fetches the details of all the 1000 books. Suppose the Source now has the data of 3000 books. Since the API only fetches 1000 records in a single call, you would need to select a different pagination option to fetch the remaining 2000 records; with no pagination, Hevo fetches the same list of 1000 books in every call.

       Note: If you are unsure of the pagination option supported by the API which you have configured as a Source, contact the API vendor or developer.

     • Session: One of the parameters serves as a pointer to the next set of records. This pointer can be derived using one of the following:

       • URL-based parameters: In this method, the next page is retrieved from a URL.

         Note: Hevo stops calling the API if the current API response does not contain a next page URL or returns zero results.

         • Is the next page URL present in the response?: Hevo extracts the next page URL from the parameter mentioned.

           • Next URL Field: The JSONPath expression to the field that contains the URL to the next set of records. Retrieve this from the API response.

         

       • Query-based parameters: This is the default method whereby the next page is retrieved using the query parameters.

         Note: Hevo stops calling the API if the API response does not return a next pointer value or returns zero results.

         

         • Session Query Param: The name of the query parameter that Hevo appends to the URL to fetch the next page from the second call onwards.

         • Session Response Field: The JSONPath expression to the field that contains the pointer to the next set of records.

         • Extract from Last Record in Response: Hevo retrieves the pointer to the next page from a field of the last record in the current API response.

           Note: The JSONPath expression to the field when you select the option, Extract from Last Record in Response, should be relative to the object being fetched.

           For example, suppose the API response contains a field id that is used to identify each record. And, the field after of the last record holds the information to the next set of records. Then, to retrieve all the pages in the API response, you must specify the value of Session Query Param as after, and the value of Session Response Field as id.

           

     • Page Number: Page number-based pagination works through a page number, either specified by you or extracted from the API response.

       Note: Hevo stops calling the API once it stops giving results.

       

       • Page Number Query Param: The name of the query parameter that Hevo appends to the URL to specify the page number.

       • Initial Page Number: The value of the page number in the API response from which Hevo fetches records. It increments the page number for each call till the API stops returning results. Default value: 1.

       • Extract Page Number from Response: Hevo extracts the page number from the response of the API call.

         

         • Page Response Field: The JSONPath expression to the field that contains the next page number.

         • Increment for next page: If enabled, Hevo increments the page number value when it makes the next API call.

     • Offset and Limit: Offset-based pagination works through an offset generated by the first API call to a URL.

       Note: Hevo stops calling the API once it stops giving results.

       

       • Offset Query Param: The name of the query parameter that specifies the offset. Retrieve this from the API response. Hevo appends this parameter to the URL.

       • Offset Starting Value: The value for the offset parameter from which the API starts fetching data.

       • Offset Increment Value: The number of records that the API must skip before it starts to fetch data from the second call onwards.

       • Limit Query Param: The name of the query parameter that Hevo appends to the URL to limit the number of records that the API fetches.

       • Limit Initial Value: The number of records fetched by the API on each call.

         For example, suppose an API result set contains 100 records. If you want to generate a sample data set that contains the first five records out of every 20 records, then set the Offset Query Param to offset, the Offset Starting Value to 0, the Offset Increment Value to 15, the Limit Query Param to limit, and the Limit Initial Value to 5.

         With this configuration, the API:

         1. Starts from an offset of zero, indicating the first record.

         2. It fetches the first five records.

         3. Next, it skips 15 records (offset is now 20).

         4. Again, it fetches five records (20-24) and skips the next 15 (offset is now 40), and so on.

         5. It repeats this pattern on every call to fetch five records of every 20 till it receives zero results.

         

5. Click TEST PAGINATION to see the API response paginated as per the settings you specified. For example, consider the following settings:

   • Pagination type: Page Number
   • Page Number Query Param: page
   • Initial Page Number: 1

   Click TEST PAGINATION.

   

   Hevo displays the following API responses. Click NEXT PAGE to retrieve the next page(s) in sequence till you reach the last page.

   

6. Click FINISH SETUP to proceed to setting up the Destination.