Log-based Pipelines

Last updated on Jun 21, 2024
On This Page

A log-based Pipeline moves data from a Source to a Destination by trimming through logs, which consist of Events describing changes made to a database. These logs are generally maintained for replication or recovery of data.

You must select the appropriate Pipeline mode depending on the Source type to create a log-based Pipeline:

  • A MySQL Source configured using the BinLog ingestion mode.

  • A PostgreSQL Source configured using the Logical replication ingestion mode.

  • An Amazon DynamoDB Source using Streams.

  • A MongoDB Source configured using either the Change Streams or OpLog ingestion mode.

  • An SQL Server with individual objects configured with Change Tracking as the query mode, the ingestion mode, or both.

  • An Oracle Source configured using the Redo Logs ingestion mode.

From Release 2.21 onwards, new and existing teams can create a streaming Pipeline for the Source types (all variants) listed above, except SQL Server (all variants) and Amazon DynamoDB. To enable streaming for your log-based Pipeline, you should contact the Hevo Sales team.

Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Jun-21-2024 NA Updated information about streaming Pipelines to reflect supported functionality.

Tell us what went wrong