- Introduction
- Getting Started
- Data Ingestion
- Data Loading
- Loading Data in a Database Destination
- Loading Data to a Data Warehouse
- Optimizing Data Loading for a Destination Warehouse
- Manually Triggering the Loading of Events
- Scheduling Data Load for a Destination
- Loading Events in Batches
- Data Loading Statuses
- Data Spike Alerts
- Name Sanitization
- Table and Column Name Compression
- Parsing Nested JSON Fields in Events
- Pipelines
- Data Flow in a Pipeline
- Familiarizing with the Pipelines UI
- Working with Pipelines
- Managing Objects in Pipelines
-
Transformations
-
Python Code-Based Transformations
- Supported Python Modules and Functions
-
Transformation Methods in the Event Class
- Create an Event
- Retrieve the Event Name
- Rename an Event
- Retrieve the Properties of an Event
- Modify the Properties for an Event
- Fetch the Primary Keys of an Event
- Modify the Primary Keys of an Event
- Fetch the Data Type of a Field
- Check if the Field is a String
- Check if the Field is a Number
- Check if the Field is Boolean
- Check if the Field is a Date
- Check if the Field is a Time Value
- Check if the Field is a Timestamp
-
TimeUtils
- Convert date string to required format
- Convert date to required format
- Convert datetime string to required format
- Convert epoch time to a date
- Convert epoch time to a datetime
- Convert epoch to required format
- Convert epoch to a time
- Get time difference
- Parse date string to date
- Parse date string to datetime format
- Parse date string to time
- Utils
- Examples of Python Code-based Transformations
-
Drag and Drop Transformations
- Special Keywords
-
Transformation Blocks and Properties
- Add a Field
- Change Datetime Field Values
- Change Field Values
- Drop Events
- Drop Fields
- Find & Replace
- Flatten JSON
- Format Date to String
- Format Number to String
- Hash Fields
- If-Else
- Mask Fields
- Modify Text Casing
- Parse Date from String
- Parse JSON from String
- Parse Number from String
- Rename Events
- Rename Fields
- Round-off Decimal Fields
- Split Fields
- Examples of Drag and Drop Transformations
- Effect of Transformations on the Destination Table Structure
- Transformation Reference
- Transformation FAQs
-
Python Code-Based Transformations
-
Schema Mapper
- Using Schema Mapper
- Mapping Statuses
- Auto Mapping Event Types
- Manually Mapping Event Types
- Modifying Schema Mapping for Event Types
- Schema Mapper Actions
- Fixing Unmapped Fields
- Resolving Incompatible Schema Mappings
- Resizing String Columns in the Destination
- Schema Mapper Compatibility Table
- Limits on the Number of Destination Columns
- File Log
- Troubleshooting Failed Events in a Pipeline
- Mismatch in Events Count in Source and Destination
- Activity Log
-
Pipeline FAQs
- Does creation of Pipeline incur cost?
- Why are my new Pipelines in trial?
- Can multiple Sources connect to one Destination?
- What happens if I re-create a deleted Pipeline?
- Why is there a delay in my Pipeline?
- Can I delete skipped objects in a Pipeline?
- Can I change the Destination post-Pipeline creation?
- How does changing the query mode affect data ingestion?
- Why is my billable Events high with Delta Timestamp mode?
- Can I drop multiple Destination tables in a Pipeline at once?
- How does Run Now affect scheduled ingestion frequency?
- Will pausing some objects increase the ingestion speed?
- Can I sort Event Types listed in the Schema Mapper?
- Can I see the historical load progress?
- Why is my Historical Load Progress still at 0%?
- Why is historical data not getting ingested?
- How do I restart the historical load for all the objects?
- How do I set a field as a primary key?
- How can I load only filtered Events to the Destination?
- How do I ensure that records are loaded only once?
- Why do the Source and the Destination events count differ?
- Events Usage
- Sources
- Free Sources
-
Databases and File Systems
- Data Warehouses
-
Databases
- Connecting to a Local Database
- Amazon DocumentDB
- Amazon DynamoDB
- Elasticsearch
-
MongoDB
- Generic MongoDB
- MongoDB Atlas
- Support for Multiple Data Types for the _id Field
- Example - Merge Collections Feature
-
Troubleshooting MongoDB
-
Errors During Pipeline Creation
- Error 1001 - Incorrect credentials
- Error 1005 - Connection timeout
- Error 1006 - Invalid database hostname
- Error 1007 - SSH connection failed
- Error 1008 - Database unreachable
- Error 1011 - Insufficient access
- Error 1028 - Primary/Master host needed for OpLog
- Error 1029 - Version not supported for Change Streams
- SSL 1009 - SSL Connection Failure
- Troubleshooting MongoDB Change Streams Connection
- Troubleshooting MongoDB OpLog Connection
-
Errors During Pipeline Creation
- SQL Server
-
MySQL
- Amazon Aurora MySQL
- Amazon RDS MySQL
- Azure MySQL
- Google Cloud MySQL
- Generic MySQL
- MariaDB MySQL
-
Troubleshooting MySQL
-
Errors During Pipeline Creation
- Error 1003 - Connection to host failed
- Error 1006 - Connection to host failed
- Error 1007 - SSH connection failed
- Error 1011 - Access denied
- Error 1012 - Replication access denied
- Error 1017 - Connection to host failed
- Error 1026 - Failed to connect to database
- Error 1027 - Unsupported BinLog format
- Failed to determine binlog filename/position
- Schema 'xyz' is not tracked via bin logs
- Errors Post-Pipeline Creation
-
Errors During Pipeline Creation
- MySQL FAQs
- Oracle
-
PostgreSQL
- Amazon Aurora PostgreSQL
- Amazon RDS PostgreSQL
- Azure PostgreSQL
- Google Cloud PostgreSQL
- Generic PostgreSQL
- Heroku PostgreSQL
-
Troubleshooting PostgreSQL
-
Errors during Pipeline creation
- Error 1003 - Authentication failure
- Error 1006 - Connection settings errors
- Error 1011 - Access role issue for logical replication
- Error 1012 - Access role issue for logical replication
- Error 1014 - Database does not exist
- Error 1017 - Connection settings errors
- Error 1023 - No pg_hba.conf entry
- Error 1024 - Number of requested standby connections
- Errors Post-Pipeline Creation
-
Errors during Pipeline creation
- PostgreSQL FAQs
- Troubleshooting Database Sources
- File Storage
-
Engineering Analytics
- Apify
- Asana
- Buildkite
- GitHub
-
Streaming
- Android SDK
- Kafka
-
REST API
- Writing JSONPath Expressions
-
REST API FAQs
- Why does my REST API token keep changing?
- Can I use a bearer authorization token for authentication?
- Does Hevo’s REST API support API chaining?
- What is the maximum payload size returned by a REST API?
- How do I split an Event into multiple Event Types?
- How do I split multiple values in a key into separate Events?
- Webhook
- GitLab
- Jira Cloud
- Opsgenie
- PagerDuty
- Pingdom
- Trello
- Finance & Accounting Analytics
-
Marketing Analytics
- ActiveCampaign
- AdRoll
- Apple Search Ads
- AppsFlyer
- CleverTap
- Criteo
- Drip
- Facebook Ads
- Facebook Page Insights
- Firebase Analytics
- Freshsales
- Google Campaign Manager
- Google Ads
- Google Analytics
- Google Analytics 4
- Google Analytics 360
- Google Play Console
- Google Search Console
- HubSpot
- Instagram Business
- Klaviyo
- Lemlist
- LinkedIn Ads
- Mailchimp
- Mailshake
- Marketo
- Microsoft Advertising
- Onfleet
- Outbrain
- Pardot
- Pinterest Ads
- Pipedrive
- Recharge
- Segment
- SendGrid Webhook
- SendGrid
- Salesforce Marketing Cloud
- Snapchat Ads
- SurveyMonkey
- Taboola
- TikTok Ads
- Twitter Ads
- Typeform
- YouTube Analytics
- Product Analytics
- Sales & Support Analytics
-
Source FAQs
- From how far back can the Pipeline ingest data?
- Can I connect to a Source not listed in Hevo?
- Can I connect a local database as a Source?
- How can I push data to Hevo API?
- How do I connect a CSV file as a Source?
- Why are my selected Source objects not visible in the Schema Mapper?
- How can I transfer Excel files using Hevo?
- How does the Merge Table feature work?
- Destinations
- Familiarizing with the Destinations UI
- Databases
-
Data Warehouses
- Amazon Redshift
- Azure Synapse Analytics
- Databricks
- Firebolt
- Google BigQuery
- Hevo Managed Google BigQuery
- Snowflake
-
Destination FAQs
- Can I move data between SaaS applications using Hevo?
- Can I change the primary key in my Destination table?
- How do I change the data type of table columns?
- Can I change the Destination table name after creating the Pipeline?
- How can I change or delete the Destination table prefix?
- How do I resolve duplicate records in the Destination table?
- How do I enable or disable deduplication of records?
- Why does my Destination have deleted Source records?
- How do I filter deleted Events from the Destination?
- Does a data load regenerate deleted Hevo metadata columns?
- Can I load data to a specific Destination table?
- How do I filter out specific fields before loading data?
- How do I sort the data in the Destination?
- Transform
- Alerts
- Account Management
- Personal Settings
- Team Settings
-
Billing
- Pricing Plans
- Time-based Events Buffer
- Setting up Pricing Plans, Billing, and Payments
- On-Demand Purchases
- Billing Alerts
- Viewing Billing History
- Billing Notifications
-
Billing FAQs
- Can I get a plan apart from the Starter plan?
- Are free trial Events charged once I purchase a plan?
- For how long can I stay on the Free plan?
- How can I upgrade my plan?
- Is there a discount for non-profit organizations?
- Can I seek a refund of my payment?
- Do ingested Events count towards billing?
- Will Pipeline get paused if I exceed the Events quota?
- Will the initial load of data be free?
- Does the Hevo plan support multiple Destinations?
- Do rows loaded through Models count in my usage?
- Is Hevo subscription environment-specific?
- Can I pause billing if I have no active Pipelines?
- Can you explain the pricing plans in Hevo?
- Where do I get invoices for payments?
- Account Suspension and Restoration
- Account Management FAQs
- Activate
- Glossary
- Release Notes
- Release Version 2.13
- Release Version 2.12
- Release Version 2.11
- Release Version 2.10
- Release Version 2.09
- Release Version 2.08
- Release Version 2.07
- Release Version 2.06
- Release Version 2.05
- Release Version 2.04
- Release Version 2.03
- Release Version 2.02
- Release Version 2.01
- Release Version 2.00
- Release Version 1.99
- Release Version 1.98
- Release Version 1.97
- Release Version 1.96
- Release Version 1.95
- Release Version 1.93 & 1.94
- Release Version 1.92
- Release Version 1.91
- Release Version 1.90
- Release Version 1.89
- Release Version 1.88
- Release Version 1.87
- Release Version 1.86
- Release Version 1.84 & 1.85
- Release Version 1.83
- Release Version 1.82
- Release Version 1.81
- Release Version 1.80 (Jan-24-2022)
- Release Version 1.79 (Jan-03-2022)
- Release Version 1.78 (Dec-20-2021)
- Release Version 1.77 (Dec-06-2021)
- Release Version 1.76 (Nov-22-2021)
- Release Version 1.75 (Nov-09-2021)
- Release Version 1.74 (Oct-25-2021)
- Release Version 1.73 (Oct-04-2021)
- Release Version 1.72 (Sep-20-2021)
- Release Version 1.71 (Sep-09-2021)
- Release Version 1.70 (Aug-23-2021)
- Release Version 1.69 (Aug-09-2021)
- Release Version 1.68 (Jul-26-2021)
- Release Version 1.67 (Jul-12-2021)
- Release Version 1.66 (Jun-28-2021)
- Release Version 1.65 (Jun-14-2021)
- Release Version 1.64 (Jun-01-2021)
- Release Version 1.63 (May-19-2021)
- Release Version 1.62 (May-05-2021)
- Release Version 1.61 (Apr-20-2021)
- Release Version 1.60 (Apr-06-2021)
- Release Version 1.59 (Mar-23-2021)
- Release Version 1.58 (Mar-09-2021)
- Release Version 1.57 (Feb-22-2021)
- Release Version 1.56 (Feb-09-2021)
- Release Version 1.55 (Jan-25-2021)
- Release Version 1.54 (Jan-12-2021)
- Release Version 1.53 (Dec-22-2020)
- Release Version 1.52 (Dec-03-2020)
- Release Version 1.51 (Nov-10-2020)
- Release Version 1.50 (Oct-19-2020)
- Release Version 1.49 (Sep-28-2020)
- Release Version 1.48 (Sep-01-2020)
- Release Version 1.47 (Aug-06-2020)
- Release Version 1.46 (Jul-21-2020)
- Release Version 1.45 (Jul-02-2020)
- Release Version 1.44 (Jun-11-2020)
- Release Version 1.43 (May-15-2020)
- Release Version 1.42 (Apr-30-2020)
- Release Version 1.41 (Apr-2020)
- Release Version 1.40 (Mar-2020)
- Release Version 1.39 (Feb-2020)
- Release Version 1.38 (Jan-2020)
- Upcoming Features
REST API
Hevo allows you to bring data from various Sources through its native connectors. However, for situations where you need to bring data from multiple different applications or from an in-house REST API, you can use the REST API Source.
For creating Pipelines using this Source, Hevo provides you a fully managed BigQuery data warehouse as a possible Destination. This option remains available till the time you set up your first BigQuery Destination irrespective of any other Destinations that you may have. With the managed warehouse, you are only charged the cost that Hevo incurs for your project in Google BigQuery. The invoice is generated at the end of each month and payment is recovered as per the payment instrument you have set up. You can now create your Pipeline and directly start analyzing your Source data. Read Hevo Managed Google BigQuery.
Prerequisites
-
An understanding of JSONPath and JSONPath expressions.
-
You are assigned the Team Administrator, Team Collaborator, or Pipeline Administrator role in Hevo to create the Pipeline.
Configuring REST API as a Source
Perform the following steps to configure REST API as a Source in Hevo:
-
Click PIPELINES in the Navigation Bar.
-
Click + CREATE in the Pipeline List View.
-
In the Select Source Type page, select REST API.
-
Provide REST endpoint details in the Configure your REST API Source page:
-
Pipeline Name: A unique name for the Pipeline.
-
Method: The API method. This can be GET or POST.
-
URL: The fully qualified URL endpoint for the REST API. The endpoint must contain HTTP or HTTPS, whichever is applicable.
-
Request Body: The POST body for the REST API, if the API method is POST. The body must be a valid JSON or Form Data in a key-value pair format.
-
Authentication:
-
Basic Auth: Optional Username and Password for Basic Authentication to be provided when the API requires it.
-
Basic Auth User: Your API username or API key.
-
Basic Auth Password: Your API password or API secret.
-
-
OAuth 2.0:
-
Redirect URL: The URL where the Source authorization server sends the auth code which can then be exchanged for the access token.
-
Select Access Token: The access token used by Hevo to make API requests. You can:
-
Select an existing token that you created.
-
+ Create New Token: A new access token for making API requests.
-
Token Name: A name for the access token.
-
Client ID: The ID retrieved from your Source application, to identify Hevo.
-
Client Secret: The secret key retrieved from your Source application, to authenticate Hevo to read your Source data.
-
Auth URL: The URL for the Source API authorization server, to retrieve the auth code.
-
Token URL: The URL for the Source API authentication server, to exchange an auth code for an access token.
-
Scopes: A set of permissions that you want to provide Hevo for accessing the Source data.
-
-
-
-
-
Headers: Custom headers needed for the API call.
-
Query Params: Query parameters to be appended to the URL while making a request. These are given as a key-value pair. Hevo supports the following type of query parameters:
-
Text: A static query parameter.
-
Key: The name of the parameter.
-
Value: The value of the parameter.
For example, suppose you want to find the total number of records in an API response. Then, specify the Key as total and the Value as true.
-
-
Date: A dynamic Date type query parameter.
-
Key: The name of the parameter.
-
Format: The format in which the string value for the date must be generated. Hevo generates DateTime values according to Java’s SimpleDateFormat.
-
Offset Unit: The unit in which DateTime is generated.
-
Offset Value: A numerical value for the selected unit.
The Offset Unit and Offset Value are optional parameters. They are used to generate DateTime values relative to the time at which the API call is made.
For example, to generate a value that is 24 hours ago, provide the Offset Unit as Hours and the Offset Value as -24. A positive offset value generates future date times.
-
-
-
Data Root: The JSONPath expression to the data that you want to replicate. Hevo suggests possible data roots based on the response received from the URL specified in the connection settings.
The Data Root field displays the first suggested data root, and the Sample Response from API section displays the parsed response from it. You can change the data root if you want and click CHECK PARSED RESPONSE to view the parsed data. For example, the image below displays the first suggested data root
$.*
and the parsed response for it.Note: The data root property can only contain a JSONPath expression that returns an array or an object, and each element of the data root should return a JSON object.
Sample Response from API: A view-only field that displays the raw and parsed sample responses from the API. You can use the responses to identify your data root and view the Events present in the data root respectively.
-
Raw Response: The API response in JSON format, based on the URL specified in the connection settings. It also displays the data root for the Events. For example, in the image below, the data root is friends.
-
Parsed Events: The API response parsed based on the data root specified above. It displays the list of Events to be ingested from the Source.
Read JSONPath Expression Examples for more information on writing valid expressions. Also, read Troubleshooting the REST API.
-
-
Pagination: The process of dividing the API response into pages. Default selection: No Pagination. Hevo supports the following types of pagination:
-
No Pagination: This option attempts to fetch the entire result set of the API in a single call. However, depending on the way pagination is implemented within the API, the entire response may be fetched in one page or be divided into multiple pages. In the latter case, you need to select a different pagination option to retrieve the entire result set.
For example, consider a REST API that fetches a list of 1000 books and their reviews in a single call. If you configure this API as a Source with No Pagination, then, on each Pipeline run, the API fetches the details of all the 1000 books. Suppose the Source now has the data of 3000 books. Since the API only fetches 1000 records in a single call, you would need to select a different pagination option to fetch the remaining 2000 records; with no pagination, Hevo fetches the same list of 1000 books in every call.
Note: If you are unsure of the pagination option supported by the API which you have configured as a Source, contact the API vendor or developer.
-
Session: One of the parameters serves as a pointer to the next set of records. This pointer can be derived using one of the following:
-
URL-based parameters: In this method, the next page is retrieved from a URL.
Note: Hevo stops calling the API if the current API response does not contain a next page URL or returns zero results.
-
Is the next page URL present in the response?: Hevo extracts the next page URL from the parameter mentioned.
- Next URL Field: The JSONPath expression to the field that contains the URL to the next set of records. Retrieve this from the API response.
-
-
Query-based parameters: This is the default method whereby the next page is retrieved using the query parameters.
Note: Hevo stops calling the API if the API response does not return a next pointer value or returns zero results.
-
Session Query Param: The name of the query parameter that Hevo appends to the URL to fetch the next page from the second call onwards.
-
Session Response Field: The JSONPath expression to the field that contains the pointer to the next set of records.
-
Extract from Last Record in Response: Hevo retrieves the pointer to the next page from a field of the last record in the current API response.
Note: The JSONPath expression to the field when you select the option, Extract from Last Record in Response, should be relative to the object being fetched.
For example, suppose the API response contains a field
id
that is used to identify each record. And, the fieldafter
of the last record holds the information to the next set of records. Then, to retrieve all the pages in the API response, you must specify the value of Session Query Param as after, and the value of Session Response Field as id.
-
-
-
Page Number: Page number-based pagination works through a page number, either specified by you or extracted from the API response.
Note: Hevo stops calling the API once it stops giving results.
-
Page Number Query Param: The name of the query parameter that Hevo appends to the URL to specify the page number.
-
Initial Page Number: The value of the page number in the API response from which Hevo fetches records. It increments the page number for each call till the API stops returning results. Default value: 1.
-
Extract Page Number from Response: Hevo extracts the page number from the response of the API call.
-
Page Response Field: The JSONPath expression to the field that contains the next page number.
-
Increment for next page: If enabled, Hevo increments the page number value when it makes the next API call.
-
-
-
Offset and Limit: Offset-based pagination works through an offset generated by the first API call to a URL.
Note: Hevo stops calling the API once it stops giving results.
-
Offset Query Param: The name of the query parameter that specifies the offset. Retrieve this from the API response. Hevo appends this parameter to the URL.
-
Offset Starting Value: The value for the offset parameter from which the API starts fetching data.
-
Offset Increment Value: The number of records that the API must skip before it starts to fetch data from the second call onwards.
-
Limit Query Param: The name of the query parameter that Hevo appends to the URL to limit the number of records that the API fetches.
-
Limit Initial Value: The number of records fetched by the API on each call.
For example, suppose an API result set contains 100 records. If you want to generate a sample data set that contains the first five records out of every 20 records, then set the Offset Query Param to offset, the Offset Starting Value to 0, the Offset Increment Value to 15, the Limit Query Param to limit, and the Limit Initial Value to 5.
With this configuration, the API:
-
Starts from an offset of zero, indicating the first record.
-
It fetches the first five records.
-
Next, it skips 15 records (offset is now 20).
-
Again, it fetches five records (20-24) and skips the next 15 (offset is now 40), and so on.
-
It repeats this pattern on every call to fetch five records of every 20 till it receives zero results.
-
-
-
-
-
Click TEST PAGINATION to see the API response paginated as per the settings you specified. For example, consider the following settings:
- Pagination type: Page Number
- Page Number Query Param: page
- Initial Page Number: 1
Click TEST PAGINATION.
Hevo displays the following API responses. Click NEXT PAGE to retrieve the next page(s) in sequence till you reach the last page.
-
Click FINISH SETUP to proceed to setting up the Destination.
Data replication
Default Pipeline Frequency | Minimum Pipeline Frequency | Maximum Pipeline Frequency | Custom Frequency Range (Hrs) |
---|---|---|---|
15 Mins | 5 Mins | 168 Hrs | 1-168 |
Note: The custom frequency must be set in hours, as an integer value. For example, 1, 2, 3 but not 1.5 or 1.75.
Additional Information
Read the detailed Hevo documentation for the following related topics:
Troubleshooting the Rest API
Refer to this section for possible causes for some of the errors while testing the API.
-
Error: Failed to get a response
Reason: The specified URL is incorrect, causing the API to fail.
-
Error: Failed to parse response
Reason: The specified data root is incorrect, due to which, the API call fails to locate the root of the data in the API response.
-
Error: HTTP ERROR 404
Reason: A private API is configured with Hevo, but Hevo’s IP addresses are not whitelisted.
Resolution: Whitelist the Hevo IP address for your region for private APIs to work with Hevo.
Limitations
- By default, Hevo uses the __hevo_id metadata column as the primary key to load the REST API Source data to the Destination. This may lead to data duplication in the Destination in case you have mutable data present in your Source. To avoid this, you can set another column as the primary key.
Revision History
Refer to the following table for the list of key updates made to this page:
Date | Release | Description of Change |
---|---|---|
Dec-19-2022 | 2.04 | Updated the section, Configuring REST API as a Source to add information about enhanced data root auto-suggestion functionality. |
Nov-23-2022 | NA | Added section, Limitations. |
Nov-11-2022 | NA | Updated section, Troubleshooting the Rest API to include the troubleshooting scenario for Error 404. |
Sep-05-2022 | NA | Updated the See Also section to: - Add a reference to the REST API FAQs page. - Remove the reference to the pagination-related FAQ. |
Aug-24-2022 | 1.96 | Updated section, Configuring REST API as a Source to add information about data root auto-suggestion by Hevo. |
May-10-2022 | 1.88 | - Added information about authentication using OAuth 2.0 protocol. - Removed section, Testing the API Response. |
Dec-20-2021 | 1.78 | Updated the screenshot in step 4 of the section, Configuring REST API as a Source to to reflect the latest UI changes. |
Nov-09-2021 | 1.75 | Updated: - Step 4 in the section, Configuring REST API as a Source to explain the API behavior in case No Pagination is selected. - The See Also section to add a link to a pagination-related FAQ. |
Oct-25-2021 | NA | Added the Pipeline frequency information in the Data Replication section. Added the See Also section. |
Oct-04-2021 | 1.73 | Updated step 4 in the section, Configuring REST API as a Source to to add: - A new pagination option, Offset and Limit. - An option Is the next page URL present in the response? in Session-based pagination. - Screenshots for each pagination option. |
Sep-20-2021 | 1.72 | Updated step 4 in the section, Configuring REST API as a Source to add: - The Request Body property for the POST method. - A note in the Data Root property about expected JSONPath expressions. |
Apr-06-2021 | 1.60 | Added the following sections: - Testing the REST API. - Troubleshooting the REST API. Updated the section Configuring REST API as a Source to include the field Extract from last record in response. |