Log-based Pipelines

A log-based Pipeline moves data from a Source to a Destination by trimming through logs which consist of Events describing changes made to a database, wherein data is ingested at a fixed interval.

These logs are generally maintained for replication or recovery of data.

You must select the appropriate Pipeline mode depending on the Source type to create a log-based Pipeline:

  • A MySQL Source configured using BinLog mode.
  • A PostgreSQL Source configured using Logical replication mode.
  • An Amazon DynamoDB Source using Streams.
  • A MongoDB Source configured using either Change Stream or OpLog mode.
  • An Amazon Aurora Source configured using BinLog mode.
  • A SQL Server with individual jobs configured with `Change Tracking` as the query mode or the Change Tracking Pipeline mode.
  • An Oracle Source configured using Redo Logs mode.

See Also

Last updated on 02 Mar 2021