Log-based Pipelines

A log-based Pipeline moves data from a Source to a Destination by trimming through logs which consist of Events describing changes made to a database, wherein data is ingested at a fixed interval.

These logs are generally maintained for replication or recovery of data.

You must select the appropriate Pipeline mode depending on the Source type to create a log-based Pipeline:

• A MySQL Source configured using BinLog mode.

• A PostgreSQL Source configured using Logical replication mode.

• An Amazon DynamoDB Source using Streams.

• A MongoDB Source configured using either Change Stream or OpLog mode.

• An Amazon Aurora MySQL Source configured using BinLog mode.

• An SQL Server with individual objects configured with Change Tracking as the Query mode or the Pipeline mode or both.

• An Oracle Source configured using Redo Logs mode.

See Also

• Data Loss Prevention
• Can I see the percentage completion of the historical load?

• Articles in this section
  • Pausing a Log-based Pipeline
  • Handling Deletes in Log-based Pipelines