Log-based pipelines are the Pipelines made from any of the following sources
- MySQL BinLog
- Postgres WAL
- Mongo OpLog
- Mongo ChangeStreams
- MSSQL Change Tracker
- Oracle Redo Log
- AWS DynamoDB
Hevo reads the DB log to replicate the database. The DB log contains information about new Insertions, updations, and deletions. When Hevo encounters a delete action in log it replicates the information in the destination under a boolean column __hevo_marked_deleted.
Users can use the column accordingly in their query to ignore it from their analysis or maybe perform deletes later on. It is to be noted that Hevo does not perform any delete operations in the destination.
About the column
Data Type: Boolean
null: This will happen when the column is being added into the existing table, All data with null value should be considered not deleted.
true: Data was deleted in the source
false: Data was not deleted in the source