Handling Deletes in Log-based Pipelines

Log-based Pipelines are the Pipelines made from any of the following Sources:

• MySQL BinLog

• Postgres WAL

• MongoDB OpLog

• MongoDB Change Streams

• SQL Server Change Tracking

• Oracle Redo Log

• AWS DynamoDB Streams

Hevo reads the DB log to replicate the database. The DB log contains information about new insertions, updations, and deletions. When Hevo encounters a delete action in log, it replicates the information in the destination under a boolean column __hevo_marked_deleted.

Users can use the column accordingly in their query to ignore it from their analysis or maybe perform deletes later on. It is to be noted that Hevo does not perform any delete operations in the Destination.

About the column

• Name: __hevo_marked_deleted

• Data Type: Boolean

• Nullable: Yes

• Possible Values:

  • null: This happens when the column is being added into the existing table, All data with null value should be considered not deleted.

  • true: Data was deleted in the source

  • false: Data was not deleted in the source

Note: This field used to be visible in the Schema Mapper till before 3rd September 2019. It will not be visible in the Schema Mapper anymore but will be added to destination tables automatically.