Ingestion Modes and Query Modes

Last updated on Nov 08, 2022
On This Page

Hevo Pipelines ingest data from your Source and load it to a Destination. The existing data in your Source is ingested as historical load and any data changes thereafter are ingested as incremental data.

To ingest data from your Sources, Hevo uses:

  • Ingestion modes: Ingestion modes are applicable for database Sources such as MySQL, MS SQL Server, Oracle, MongoDB and PostgreSQL. The Ingestion mode, also called Pipeline Mode, defines how Hevo fetches data from a database Source. Hevo provides three modes of data ingestion: Log-based, Table, and Custom SQL. Hevo can fetch data using the logs maintained by your Source, write custom SQL queries, or ingest full or selective data from the database tables. Read Ingestion Modes to know the details of each mode.
    Read Events Usage for Billing and Best Practices for Creating Database Pipelines to select a suitable ingestion mode.

  • Query Modes: Query modes are applicable only for log-based and table-based Pipelines created with database Sources, such as MySQL, PostgreSQL, Oracle, and MS SQL. These define how Hevo must fetch data from the Source database. Hevo allows you to choose the query mode for every object in your Source database. Read Query modes to know the details of each mode. Query modes keep record of incrementing data, by creating new columns in the Destination database. You can replicate the full table in a single run using Full Load, or you can replicate individual incrementing rows and columns using different query modes, such as Unique incrementing Append only, Delta Timestamp, Change data capture, Change tracking, and XMIN. Read Query Modes and Events Quota Consumption to select a suitable query mode.

Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Sep-21-2022 NA New document.

Tell us what went wrong