ON THIS PAGE
Hevo allows you to bring data from tons of data sources through its native connectors. But, there are thousands of applications out there from which you might want to bring data into your data warehouse. Also, there are cases where you might want to bring data from an in-house REST API. For all such cases, the REST API source can be used to bring data into your data warehouse.
Note: For Pipelines created with this Source, Hevo provides you a fully-managed BigQuery data warehouse Destination if you do not already have one set up. You are only charged the cost that Hevo incurs for your project in Google BigQuery. The invoice is generated at the end of each month and payment is recovered as per the payment instrument you have set up. You can now create your Pipeline and directly start analyzing your Source data. Read Hevo Managed Google BigQuery .
You need to have a decent understanding of JSON Paths.
Provide REST endpoint details on the Connection Settings page. You will have the following options in the connection details block:
- Source Name - A unique name for this source.
- Url - The fully qualified URL endpoint for the REST API along with URL method (GET or POST). The endpoint must contain HTTP or HTTPS whichever applicable.
- Data Root - The JSON Path to root of data in the API response.
- Basic Auth - Optional Username and Password for Basic Authentication to be provided when API requires it.
- Headers - Arbitrary custom headers needed for the API call.
- Query Params - Query parameters to be appended to the URL while making a request. The following type of Query Parameters are supported by Hevo:
- Text - A static query parameter. Key is the name of the parameter and Value is a static value.
- Date - A dynamic date type query parameter. Key is the name of the parameter. The format must be provided to generate a string value for the parameter. Hevo generates date time values according to Java’s SimpleDateFormat. Offset Unit and Offset Value can be provided to generate date time values relative to the time at which the API call is made. e.g. to generate a value 24 hours ago, provide the Offset Unit as Hours and the Offset Value as -24. Positive offset value will generate future date times.
- Pagination - Hevo supports two types of paginations:
- Session - Session based pagination works through a session id generated by the first API call to a URL. Session Query Param is the name of the query parameter that Hevo will append to the url from the second call. Session Response Field is the JSON path to the field in the response which will contain the session id. Hevo will stop calling the API if the API doesn’t return a session id in the response or starts giving back zero results.
- Page number - Page Number Query Param is the name of the query parameter that Hevo will append to the url to specify the page number. Page Number Value can be generated in two ways:
- Increment on each call - Hevo will start calling the API with page number value as 1 and increment the value for each call till the API stops giving back results.
- Extract from Response - Hevo will extract the page number from the response of the API call using the JSON path provided in the Page Response Field. You can enable Increment for next page if you want Hevo to increment the value while making the next API call. Hevo will stop calling the API once it stops giving results.