REST API

Hevo allows you to bring data from various Sources through its native connectors. However, for situations where you need to bring data from multiple different applications or from an in-house REST API, you can use the REST API Source.

Note: For Pipelines created with this Source, Hevo provides you a fully-managed BigQuery data warehouse Destination if you do not already have one set up. You are only charged the cost that Hevo incurs for your project in Google BigQuery. The invoice is generated at the end of each month and payment is recovered as per the payment instrument you have set up. You can now create your Pipeline and directly start analyzing your Source data. Read Hevo Managed Google BigQuery .

Prerequisites

You need to have a decent understanding of JSON Paths.

Configuring REST API as a Source

Perform the following steps to configure REST API as a Source in Hevo:

  1. Click PIPELINES in the Asset Palette.

  2. Click + CREATE in the Pipeline List View.

  3. In the Select Source Type page, select REST API.

  4. Provide REST endpoint details in the Configure your REST API Source page:

    REST API settings

    • Pipeline Name - A unique name for the Pipeline.
    • Method - The API method. This can be GET or POST.
    • Url - The fully qualified URL endpoint for the REST API along with URL method (GET or POST). The endpoint must contain HTTP or HTTPS whichever applicable.
    • Data Root - The JSON Path to root of data in the API response. Read Writing JSONPath Expressions.
    • Basic Auth - Optional Username and Password for Basic Authentication to be provided when API requires it.
    • Headers - Arbitrary custom headers needed for the API call.
    • Query Params - Query parameters to be appended to the URL while making a request. The following type of Query Parameters are supported by Hevo:
    • Text - A static query parameter. Key is the name of the parameter and Value is a static value.
    • Date - A dynamic date type query parameter. Key is the name of the parameter. The format must be provided to generate a string value for the parameter. Hevo generates date time values according to Java’s SimpleDateFormat. Offset Unit and Offset Value can be provided to generate date time values relative to the time at which the API call is made. e.g. to generate a value 24 hours ago, provide the Offset Unit as Hours and the Offset Value as -24. Positive offset value will generate future date times.
      • Pagination - Hevo supports two types of paginations:
    • Session - Session based pagination works through a session id generated by the first API call to a URL. Session Query Param is the name of the query parameter that Hevo will append to the url from the second call. Session Response Field is the JSON path to the field in the response which will contain the session id. Hevo will stop calling the API if the API doesn’t return a session id in the response or starts giving back zero results.
    • Page number - Page Number Query Param is the name of the query parameter that Hevo will append to the url to specify the page number. Page Number Value can be generated in two ways:
      • Increment on each call - Hevo will start calling the API with page number value as 1 and increment the value for each call till the API stops giving back results.
      • Extract from Response - Hevo will extract the page number from the response of the API call using the JSON path provided in the Page Response Field. You can enable Increment for next page if you want Hevo to increment the value while making the next API call. Hevo will stop calling the API once it stops giving results.
  5. Click TEST & CONTINUE.

  6. Proceed to configuring the data ingestion and setting up the Destination.

Limitations

  • Hevo only supports query params, and not body, in the REST API.
Last updated on 30 Dec 2020