REST API

Hevo allows you to bring data from various Sources through its native connectors. However, for situations where you need to bring data from multiple different applications or from an in-house REST API, you can use the REST API Source.

For creating Pipelines using this Source, Hevo provides you a fully managed BigQuery data warehouse as a possible Destination. This option remains available till the time you set up your first BigQuery Destination irrespective of any other Destinations that you may have. With the managed warehouse, you are only charged the cost that Hevo incurs for your project in Google BigQuery. The invoice is generated at the end of each month and payment is recovered as per the payment instrument you have set up. You can now create your Pipeline and directly start analyzing your Source data. Read Hevo Managed Google BigQuery.

Prerequisites

You need to have a decent understanding of JSON Paths.


Configuring REST API as a Source

Perform the following steps to configure REST API as a Source in Hevo:

  1. Click PIPELINES in the Asset Palette.

  2. Click + CREATE in the Pipeline List View.

  3. In the Select Source Type page, select REST API.

  4. Provide REST endpoint details in the Configure your REST API Source page:

    REST API settings

    • Pipeline Name - A unique name for the Pipeline.

    • Method - The API method. This can be GET or POST.

    • Url - The fully qualified URL endpoint for the REST API along with URL method (GET or POST). The endpoint must contain HTTP or HTTPS whichever applicable.

    • Data Root - The JSON Path to root of data in the API response. Read Writing JSONPath Expressions.

    • Basic Auth - Optional Username and Password for Basic Authentication to be provided when API requires it.

    • Headers - Arbitrary custom headers needed for the API call.

    • Query Params - Query parameters to be appended to the URL while making a request. The following type of Query Parameters are supported by Hevo:

      • Text - A static query parameter. Key is the name of the parameter and Value is a static value.

      • Date - A dynamic date type query parameter. Key is the name of the parameter. The format must be provided to generate a string value for the parameter. Hevo generates date time values according to Java’s SimpleDateFormat. Offset Unit and Offset Value can be provided to generate date time values relative to the time at which the API call is made. e.g. to generate a value 24 hours ago, provide the Offset Unit as Hours and the Offset Value as -24. Positive offset value will generate future date times.

    • Pagination - Process of diving the API response into pages. Hevo supports two types of paginations:

      • Session - Session-based pagination works through a session id generated by the first API call to a URL. Session Query Param is the name of the query parameter that Hevo will append to the url from the second call. Session Response Field is the JSON path to the field in the response which will contain the session id. Hevo will stop calling the API if the API doesn’t return a session id in the response or starts giving back zero results.

        • Extract from last record in response: Retrieves the next page by using an element from the last record in the current API response.
      • Page number - Page Number Query Param is the name of the query parameter that Hevo will append to the url to specify the page number. Page Number Value can be generated in two ways:

        • Increment on each call - Hevo will start calling the API with page number value as 1 and increment the value for each call till the API stops giving back results.

        • Extract from Response - Hevo will extract the page number from the response of the API call using the JSON path provided in the Page Response Field. You can enable Increment for next page if you want Hevo to increment the value while making the next API call. Hevo will stop calling the API once it stops giving results.

  5. Click TEST & CONTINUE.

  6. Proceed to configuring the data ingestion and setting up the Destination.


Testing the REST API

Hevo allows you to instantly test your configuration using the Test the REST API panel on the right side of the Configure your REST API Source page. This helps you avoid any errors that might occur due to incorrect configuration settings, such as, incorrect URL, API key, or data root specified while configuring REST API as a Source in Hevo. You can also view how the ingested Events look, and test the pagination in the API response through this panel.

Panel

Retrieving API response

You can retrieve and view the API response for your specified settings by clicking GET RESPONSE in the Test the REST API panel.

Get response

For example, consider the following settings:

  • Url: https://jsonplaceholder.typicode.com/posts

  • Data Root: $

When you click GET RESPONSE, the following API response is retrieved.

API Responses

You can change the connection settings and click GET RESPONSE to retrieve another API response. For each API response, a new row is added on the top in the API Responses section.

New Row

Click the Up/Down arrow next to an API response to view the complete raw response and the corresponding parsed Events:

  • Raw Response: The complete API response based on the URL specified in the connection settings.

  • Parsed Events: The records ingested from the Source based on the data root specified in the connection settings.

Expanded View

Testing pagination

Once you have successfully fetched your first API response, you can test the pagination within the API response by clicking TEST PAGINATION. If your API response spans multiple pages, then you can click TEST PAGINATION to retrieve the next page(s) in sequence till you reach the last page.

Test pagination

The TEST PAGINATION button is disabled if any one of the following is true:

  • You have not yet retrieved the API response.

  • The response has only one page.

  • You have scrolled to the last page in the response.


Troubleshooting the Rest API

Refer to this section for possible causes for some of the errors while testing the API.

  • Error: Failed to get a response

    Reason: The specified URL is incorrect, causing the API to fail.

  • Error: Failed to parse response

    Reason: The specified data root is incorrect, due to which, the API call fails to locate the root of the data in the API response.


Limitations

  • Hevo only supports query params and not body in the REST API.

Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Apr-06-2021 1.60 - Added the following sections:
a. Testing the REST API.
b. Troubleshooting the REST API.
- Updated the section Configuring REST API as a Source in Hevo to include the field Extract from last record in response.
Last updated on 26 Jul 2021