REST API

Hevo allows you to bring data from various Sources through its native connectors. However, for situations where you need to bring data from multiple different applications or from an in-house REST API, you can use the REST API Source.

For creating Pipelines using this Source, Hevo provides you a fully managed BigQuery data warehouse as a possible Destination. This option remains available till the time you set up your first BigQuery Destination irrespective of any other Destinations that you may have. With the managed warehouse, you are only charged the cost that Hevo incurs for your project in Google BigQuery. The invoice is generated at the end of each month and payment is recovered as per the payment instrument you have set up. You can now create your Pipeline and directly start analyzing your Source data. Read Hevo Managed Google BigQuery.


Prerequisites


Configuring REST API as a Source

Perform the following steps to configure REST API as a Source in Hevo:

  1. Click PIPELINES in the Asset Palette.

  2. Click + CREATE in the Pipeline List View.

  3. In the Select Source Type page, select REST API.

  4. Provide REST endpoint details in the Configure your REST API Source page:

    REST API settings

    • Pipeline Name: A unique name for the Pipeline.

    • Method: The API method. This can be GET or POST.

    • URL: The fully qualified URL endpoint for the REST API. The endpoint must contain HTTP or HTTPS, whichever is applicable.

    • Request Body: The POST body for the REST API, if the API method is POST. The body must be a valid JSON or Form Data in a key-value pair format.

    • Data Root: The JSONPath expression to the root of the data as retrieved from the API response.

      Note: The Data Root property can only contain a JSONPath expression that returns an array or an object, and each element of the Data Root should return an object.

      Read JSONPath Expression Examples for more information on writing valid expressions. Also, read Troubleshooting the REST API.

    • Basic Auth: Optional Username and Password for Basic Authentication to be provided when the API requires it.

    • Headers: Arbitrary custom headers needed for the API call.

    • Query Params: Query parameters to be appended to the URL while making a request. These are given as a key-value pair. Hevo supports the following type of query parameters:

      • Text: A static query parameter.

        • Key: The name of the parameter.

        • Value: The value of the parameter.

        For example, suppose you want to find the total number of records in an API response. Then, specify the Key as total and the Value as true.

      • Date: A dynamic Date type query parameter.

        • Key: The name of the parameter.

        • Format: The format in which the string value for the date must be generated. Hevo generates DateTime values according to Java’s SimpleDateFormat.

        • Offset Unit: The unit in which DateTime is generated.

        • Offset Value: A numerical value for the selected unit.

        The Offset Unit and Offset Value are optional parameters. They are used to generate DateTime values relative to the time at which the API call is made.

        For example, to generate a value that is 24 hours ago, provide the Offset Unit as Hours and the Offset Value as -24. A positive offset value generates future date times.

    • Pagination: The process of dividing the API response into pages. Default selection: No Pagination. Hevo supports the following types of pagination:

      • No Pagination: This option attempts to fetch the entire result set of the API in a single call. However, depending on the way pagination is implemented within the API, the entire response may be fetched in one page or be divided into multiple pages. In the latter case, you need to select a different pagination option to retrieve the entire result set.

        For example, consider a REST API that fetches a list of 1000 books and their reviews in a single call. If you configure this API as a Source with No Pagination, then, on each Pipeline run, the API fetches the details of all the 1000 books. Suppose the Source now has the data of 3000 books. Since the API only fetches 1000 records in a single call, you would need to select a different pagination option to fetch the remaining 2000 records; with no pagination, Hevo fetches the same list of 1000 books in every call.

        Note: If you are unsure of the pagination option supported by the API which you have configured as a Source, contact the API vendor or developer.

      • Session: One of the parameters serves as a pointer to the next set of records. This pointer can be derived using one of the following:

        • URL-based parameters: In this method, the next page is retrieved from a URL.

          Note: Hevo stops calling the API if the current API response does not contain a next page URL or returns zero results.

          • Is the next page URL present in the response?: Hevo extracts the next page URL from the parameter mentioned.

            • Next URL Field: The JSONPath expression to the field that contains the URL to the next set of records. Retrieve this from the API response.

          Next page URL

        • Query-based parameters: This is the default method whereby the next page is retrieved using the query parameters.

          Note: Hevo stops calling the API if the API response does not return a next pointer value or returns zero results.

          Session default values

          • Session Query Param: The name of the query parameter that Hevo appends to the URL to fetch the next page from the second call onwards.

          • Session Response Field: The JSONPath expression to the field that contains the pointer to the next set of records.

          • Extract from Last Record in Response: Hevo retrieves the pointer to the next page from a field of the last record in the current API response.

            Note: The JSONPath expression to the field when you select the option, Extract from Last Record in Response, should be relative to the object being fetched.

            For example, suppose the API response contains a field id that is used to identify each record. And, the field after of the last record holds the information to the next set of records. Then, to retrieve all the pages in the API response, you must specify the value of Session Query Param as after, and the value of Session Response Field as id.

            Extract last record from response

      • Page Number: Page number-based pagination works through a page number, either specified by you or extracted from the API response.

        Note: Hevo stops calling the API once it stops giving results.

        Page Number

        • Page Number Query Param: The name of the query parameter that Hevo appends to the URL to specify the page number.

        • Initial Page Number: The value of the page number in the API response from which Hevo fetches records. It increments the page number for each call till the API stops returning results. Default value: 1.

        • Extract Page Number from Response: Hevo extracts the page number from the response of the API call.

          Extract page number from response

          • Page Response Field: The JSONPath expression to the field that contains the next page number.

          • Increment for next page: If enabled, Hevo increments the page number value when it makes the next API call.

      • Offset and Limit: Offset-based pagination works through an offset generated by the first API call to a URL.

        Note: Hevo stops calling the API once it stops giving results.

        Offset and Limit

        • Offset Query Param: The name of the query parameter that specifies the offset. Retrieve this from the API response. Hevo appends this parameter to the URL.

        • Offset Starting Value: The value for the offset parameter from which the API starts fetching data.

        • Offset Increment Value: The number of records that the API must skip before it starts to fetch data from the second call onwards.

        • Limit Query Param: The name of the query parameter that Hevo appends to the URL to limit the number of records that the API fetches.

        • Limit Initial Value: The number of records fetched by the API on each call.

          For example, suppose an API result set contains 100 records. If you want to generate a sample data set that contains the first five records out of every 20 records, then set the Offset Query Param to offset, the Offset Starting Value to 0, the Offset Increment Value to 15, the Limit Query Param to limit, and the Limit Initial Value to 5.

          With this configuration, the API:

          1. Starts from an offset of zero, indicating the first record.

          2. It fetches the first five records.

          3. Next, it skips 15 records (offset is now 20).

          4. Again, it fetches five records (20-24) and skips the next 15 (offset is now 40), and so on.

          5. It repeats this pattern on every call to fetch five records of every 20 till it receives zero results.

          OffsetnLimit-Example

  5. Click TEST & CONTINUE.

  6. Proceed to configuring the data ingestion and setting up the Destination.


Data replication

Default Pipeline Frequency Minimum Pipeline Frequency Maximum Pipeline Frequency Custom Frequency Range (Hrs)
15 Mins 5 Mins 168 Hrs 1-168

Note: The custom frequency must be set in hours, as an integer value. For example, 1, 2, 3 but not 1.5 or 1.75.


Testing the REST API

Hevo allows you to instantly test your configuration using the Test the REST API panel on the right side of the Configure your REST API Source page. This helps you avoid any errors that might occur due to incorrect configuration settings, such as, incorrect URL, API key, or data root specified while configuring REST API as a Source in Hevo. You can also view how the ingested Events look, and test the pagination in the API response through this panel.

Test the REST API

Retrieving API response

You can retrieve and view the API response for your specified settings by clicking GET RESPONSE in the Test the REST API panel.

Get the API Response

For example, consider the following settings:

  • Url: https://jsonplaceholder.typicode.com/posts

  • Data Root: $

When you click GET RESPONSE, the following API response is retrieved.

API Responses

You can change the connection settings and click GET RESPONSE to retrieve another API response. For each API response, a new row is added on the top in the API Responses section.

New Row

Click the Up/Down arrow next to an API response to view the complete raw response and the corresponding parsed Events:

  • Raw Response: The complete API response based on the URL specified in the connection settings.

  • Parsed Events: The records ingested from the Source based on the data root specified in the connection settings.

Expanded View

Testing pagination

Once you have successfully fetched your first API response, you can test the pagination within the API response by clicking TEST PAGINATION. If your API response spans multiple pages, then you can click TEST PAGINATION to retrieve the next page(s) in sequence till you reach the last page.

Test pagination

The TEST PAGINATION button is disabled if any one of the following is true:

  • You have not yet retrieved the API response.

  • The response has only one page.

  • You have scrolled to the last page in the response.


Troubleshooting the Rest API

Refer to this section for possible causes for some of the errors while testing the API.

  • Error: Failed to get a response

    Reason: The specified URL is incorrect, causing the API to fail.

  • Error: Failed to parse response

    Reason: The specified data root is incorrect, due to which, the API call fails to locate the root of the data in the API response.


See Also


Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Nov-09-2021 1.75 Updated:
- Step 4 in the section, Configuring REST API as a Source to explain the API behavior in case No Pagination is selected.
- The See Also section to add a link to a pagination-related FAQ.
Oct-25-2021 NA Added the Pipeline frequency information in the Data Replication section.
Added the See Also section.
Oct-04-2021 1.73 Updated step 4 in the section, Configuring REST API as a Source to to add:
- A new pagination option, Offset and Limit.
- An option Is the next page URL present in the response? in Session-based pagination.
- Screenshots for each pagination option.
Sep-20-2021 1.72 Updated step 4 in the section, Configuring REST API as a Source to add:
- The Request Body property for the POST method.
- A note in the Data Root property about expected JSONPath expressions.
Apr-06-2021 1.60 Added the following sections:
- Testing the REST API.
- Troubleshooting the REST API.
Updated the section Configuring REST API as a Source to include the field Extract from last record in response.
Last updated on 09 Nov 2021