Hevo allows you to bring data from tons of data sources through its native connectors. But, there are thousands of applications out there from which you might want to bring data into yourdata warehouse. Also, there are cases where you might want to bring data from an in-house REST API. For all such cases, the REST API source can be used to bring data into your data warehouse.
A pre-requisite to using REST API source is an understanding of JSON Paths.
1. Create a new Pipeline
Click on PIPELINES option in the left navigation bar and click on Create New Pipeline.
2. Select Source Type
Select REST API from the list on Select Source Type Screen.
3. Provide Connection Settings
Provide REST endpoint details on Connection Settings page. You will have following options in connection details block:
- Source Name - A unique name for this source.
- Url - The fully qualified URL endpoint for the REST API along with URL method (GET or POST). The endpoint must contain http or https whichever applicable.
- Data Root - The JSON Path to root of data in the API response.
- Basic Auth - Optional Username and Password for Basic Authentication to be provided when API requires it.
- Headers - Arbitrary custom headers needed for the API call.
- Query Params - Query parameters to be appended to the URL while making a request. There following type of Query Parameters supported by Hevo:
- Text - A static query parameter. Key is the name of the parameter and Value is a static value.
- Date - A dynamic date type query parameter. Key is the name of the parameter. The format must be provided to generate a string value for the parameter. Hevo generates date time values according to Java's SimpleDateFormat. Offset Unit and Offset value can be provided to generate date time values relative to the time at which the API call is made. e.g. to generate a value 24 hours ago, provide the Offset Unit as Hours and the Offset Value as -24. Positive offset value will generate future date times.
- Pagination - Hevo supports two types of paginations:
- Session - Session based pagination works through a session id generated by the first API call to a URL. Session Query Param is the name of the query parameter that Hevo will append to the url from the second call. Session Response Field is the JSON path to the field in the response which will contain the session id. Hevo will stop calling the API if the API doesn't return a session id in the response or starts giving back zero results.
- Page number - Page Number Query Param is the name of the query parameter that Hevo will append to the url to specify the page number. Page Number Value can be generated in two ways:
- Increment on each call - Hevo will start calling the API with page number value as 1 and increment the value for each call till the API stops giving back results.
- Extract from Response - Hevo will extract the page number from the response of the API call using the JSON path provided in the Page Response Field. You can enable Increment for next page if you want Hevo to increment the value while making the next API call. Hevo will stop calling the API once it stops giving results.
4. Select the Destination
Select the Destination where you want to replicate data from the REST API location or Click on NEW DESTINATION to create a new Destination. Check out How to add Destination tutorial for the detailed walkthrough on steps needed for adding new Destination.
5. Pipeline Created
Your Pipeline will be created when you enter this page and you will have an option to see Sample Data and Map Schema.
While Hevo tries to load schemas from REST API, you can select CONTINUE IN BACKGROUND if it is taking too much time. Click on CREATE SCHEMA MAPPING to map Source and Destination Schemas, check out Introduction to Schema Mapper to learn about Schema Mapper or you can select DO IT LATER to directly head to Pipeline page. You can map schemas later on Schema Mapper page in your pipeline.
Please note that your data will not start replicating in Destination tables until you map source and Destination schemas.