REST API

Hevo allows you to bring data from various Sources through its native connectors. However, for situations where you need to bring data from multiple different applications or from an in-house REST API, you can use the REST API Source.

For creating Pipelines using this Source, Hevo provides you a fully managed BigQuery data warehouse as a possible Destination. This option remains available till the time you set up your first BigQuery Destination irrespective of any other Destinations that you may have. With the managed warehouse, you are only charged the cost that Hevo incurs for your project in Google BigQuery. The invoice is generated at the end of each month and payment is recovered as per the payment instrument you have set up. You can now create your Pipeline and directly start analyzing your Source data. Read Hevo Managed Google BigQuery.


Prerequisites


Configuring REST API as a Source

Perform the following steps to configure REST API as a Source in Hevo:

  1. Click PIPELINES in the Asset Palette.

  2. Click + CREATE in the Pipeline List View.

  3. In the Select Source Type page, select REST API.

  4. Provide REST endpoint details in the Configure your REST API Source page:

    REST API configuration

    • Pipeline Name: A unique name for the Pipeline.

    • Method: The API method. This can be GET or POST.

    • URL: The fully qualified URL endpoint for the REST API. The endpoint must contain HTTP or HTTPS, whichever is applicable.

    • Request Body: The POST body for the REST API, if the API method is POST. The body must be a valid JSON or Form Data in a key-value pair format.

    • Authentication:

      • Basic Auth: Optional Username and Password for Basic Authentication to be provided when the API requires it.

        • Basic Auth User: Your API username.

        • Basic Auth Password: Your API password.

      • OAuth 2.0:

        • Redirect URL: The URL where the Source authorization server sends the auth code which can then be exchanged for the access token.

        • Select Access Token: The access token used by Hevo to make API requests. You can:

          • Select an existing token that you created.

          • + Create New Token: A new access token for making API requests.

            • Token Name: A name for the access token.

            • Client ID: The ID retrieved from your Source application, to identify Hevo.

            • Client Secret: The secret key retrieved from your Source application, to authenticate Hevo to read your Source data.

            • Auth URL: The URL for the Source API authorization server, to retrieve the auth code.

            • Token URL: The URL for the Source API authentication server, to exchange an auth code for an access token.

            • Scopes: A set of permissions that you want to provide Hevo for accessing the Source data.

    • Headers: Custom headers needed for the API call.

    • Query Params: Query parameters to be appended to the URL while making a request. These are given as a key-value pair. Hevo supports the following type of query parameters:

      • Text: A static query parameter.

        • Key: The name of the parameter.

        • Value: The value of the parameter.

        For example, suppose you want to find the total number of records in an API response. Then, specify the Key as total and the Value as true.

      • Date: A dynamic Date type query parameter.

        • Key: The name of the parameter.

        • Format: The format in which the string value for the date must be generated. Hevo generates DateTime values according to Java’s SimpleDateFormat.

        • Offset Unit: The unit in which DateTime is generated.

        • Offset Value: A numerical value for the selected unit.

        The Offset Unit and Offset Value are optional parameters. They are used to generate DateTime values relative to the time at which the API call is made.

        For example, to generate a value that is 24 hours ago, provide the Offset Unit as Hours and the Offset Value as -24. A positive offset value generates future date times.

    • Data Root: The JSONPath expression to the data that you want to replicate. You can retrieve the data root from the API response received based on the URL you entered above.

      Data Root Config

      Note: The Data Root property can only contain a JSONPath expression that returns an array or an object, and each element of the Data Root should return an object.

      • Suggested Data Roots: The data roots suggested based on the response received from the URL specified in the connection settings. Click and select one of the data roots or enter one manually. For example, in the image below, the suggested data roots are $.* and $.*.friends.*.

        Data Roots

    Sample Response from API: The raw and parsed sample responses from the API that can be used to identify your data root and view the Events present in the data root respectively.

    • Raw Response: The complete API response in JSON format, based on the URL specified in the connection settings. This also displays the data root for the Events. For example, in the image below, the data root is friends.

      Raw response

    • Parsed Events: The response that is displayed once you enter the correct data root. It includes the Events ingested from the Source based on the data root specified above.

      Parsed response

      Read JSONPath Expression Examples for more information on writing valid expressions. Also, read Troubleshooting the REST API.

    • Pagination: The process of dividing the API response into pages. Default selection: No Pagination. Hevo supports the following types of pagination:

      • No Pagination: This option attempts to fetch the entire result set of the API in a single call. However, depending on the way pagination is implemented within the API, the entire response may be fetched in one page or be divided into multiple pages. In the latter case, you need to select a different pagination option to retrieve the entire result set.

        For example, consider a REST API that fetches a list of 1000 books and their reviews in a single call. If you configure this API as a Source with No Pagination, then, on each Pipeline run, the API fetches the details of all the 1000 books. Suppose the Source now has the data of 3000 books. Since the API only fetches 1000 records in a single call, you would need to select a different pagination option to fetch the remaining 2000 records; with no pagination, Hevo fetches the same list of 1000 books in every call.

        Note: If you are unsure of the pagination option supported by the API which you have configured as a Source, contact the API vendor or developer.

      • Session: One of the parameters serves as a pointer to the next set of records. This pointer can be derived using one of the following:

        • URL-based parameters: In this method, the next page is retrieved from a URL.

          Note: Hevo stops calling the API if the current API response does not contain a next page URL or returns zero results.

          • Is the next page URL present in the response?: Hevo extracts the next page URL from the parameter mentioned.

            • Next URL Field: The JSONPath expression to the field that contains the URL to the next set of records. Retrieve this from the API response.

          Next page URL

        • Query-based parameters: This is the default method whereby the next page is retrieved using the query parameters.

          Note: Hevo stops calling the API if the API response does not return a next pointer value or returns zero results.

          Session default values

          • Session Query Param: The name of the query parameter that Hevo appends to the URL to fetch the next page from the second call onwards.

          • Session Response Field: The JSONPath expression to the field that contains the pointer to the next set of records.

          • Extract from Last Record in Response: Hevo retrieves the pointer to the next page from a field of the last record in the current API response.

            Note: The JSONPath expression to the field when you select the option, Extract from Last Record in Response, should be relative to the object being fetched.

            For example, suppose the API response contains a field id that is used to identify each record. And, the field after of the last record holds the information to the next set of records. Then, to retrieve all the pages in the API response, you must specify the value of Session Query Param as after, and the value of Session Response Field as id.

            Extract last record from response

      • Page Number: Page number-based pagination works through a page number, either specified by you or extracted from the API response.

        Note: Hevo stops calling the API once it stops giving results.

        Page Number

        • Page Number Query Param: The name of the query parameter that Hevo appends to the URL to specify the page number.

        • Initial Page Number: The value of the page number in the API response from which Hevo fetches records. It increments the page number for each call till the API stops returning results. Default value: 1.

        • Extract Page Number from Response: Hevo extracts the page number from the response of the API call.

          Extract page number from response

          • Page Response Field: The JSONPath expression to the field that contains the next page number.

          • Increment for next page: If enabled, Hevo increments the page number value when it makes the next API call.

      • Offset and Limit: Offset-based pagination works through an offset generated by the first API call to a URL.

        Note: Hevo stops calling the API once it stops giving results.

        Offset and Limit

        • Offset Query Param: The name of the query parameter that specifies the offset. Retrieve this from the API response. Hevo appends this parameter to the URL.

        • Offset Starting Value: The value for the offset parameter from which the API starts fetching data.

        • Offset Increment Value: The number of records that the API must skip before it starts to fetch data from the second call onwards.

        • Limit Query Param: The name of the query parameter that Hevo appends to the URL to limit the number of records that the API fetches.

        • Limit Initial Value: The number of records fetched by the API on each call.

          For example, suppose an API result set contains 100 records. If you want to generate a sample data set that contains the first five records out of every 20 records, then set the Offset Query Param to offset, the Offset Starting Value to 0, the Offset Increment Value to 15, the Limit Query Param to limit, and the Limit Initial Value to 5.

          With this configuration, the API:

          1. Starts from an offset of zero, indicating the first record.

          2. It fetches the first five records.

          3. Next, it skips 15 records (offset is now 20).

          4. Again, it fetches five records (20-24) and skips the next 15 (offset is now 40), and so on.

          5. It repeats this pattern on every call to fetch five records of every 20 till it receives zero results.

          OffsetnLimit-Example

  5. Click TEST PAGINATION to see the API response paginated as per the settings you specified. For example, consider the following settings:

    • Pagination type: Page Number
    • Page Number Query Param: page
    • Initial Page Number: 1

    Click TEST PAGINATION.

    Pagination settings

    Hevo displays the following API responses. Click NEXT PAGE to retrieve the next page(s) in sequence till you reach the last page.

    API Responses

  6. Click FINISH SETUP to proceed to setting up the Destination.

    Finish Setup


Data replication

Default Pipeline Frequency Minimum Pipeline Frequency Maximum Pipeline Frequency Custom Frequency Range (Hrs)
15 Mins 5 Mins 168 Hrs 1-168

Note: The custom frequency must be set in hours, as an integer value. For example, 1, 2, 3 but not 1.5 or 1.75.


Troubleshooting the Rest API

Refer to this section for possible causes for some of the errors while testing the API.

  • Error: Failed to get a response

    Reason: The specified URL is incorrect, causing the API to fail.

  • Error: Failed to parse response

    Reason: The specified data root is incorrect, due to which, the API call fails to locate the root of the data in the API response.



Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Sep-05-2022 NA Updated the See Also section to:
- Add a reference to the REST API FAQs page.
- Remove the reference to the pagination-related FAQ.
Aug-24-2022 1.96 Updated section, Configuring REST API as a Source to add information about data root auto-suggestion by Hevo.
May-10-2022 1.88 - Added information about authentication using OAuth 2.0 protocol.
- Removed section, Testing the API Response.
Dec-20-2021 1.78 Updated the screenshot in step 4 of the section, Configuring REST API as a Source to to reflect the latest UI changes.
Nov-09-2021 1.75 Updated:
- Step 4 in the section, Configuring REST API as a Source to explain the API behavior in case No Pagination is selected.
- The See Also section to add a link to a pagination-related FAQ.
Oct-25-2021 NA Added the Pipeline frequency information in the Data Replication section.
Added the See Also section.
Oct-04-2021 1.73 Updated step 4 in the section, Configuring REST API as a Source to to add:
- A new pagination option, Offset and Limit.
- An option Is the next page URL present in the response? in Session-based pagination.
- Screenshots for each pagination option.
Sep-20-2021 1.72 Updated step 4 in the section, Configuring REST API as a Source to add:
- The Request Body property for the POST method.
- A note in the Data Root property about expected JSONPath expressions.
Apr-06-2021 1.60 Added the following sections:
- Testing the REST API.
- Troubleshooting the REST API.
Updated the section Configuring REST API as a Source to include the field Extract from last record in response.
Last updated on 02 Sep 2022

Tell us what went wrong