Events not Found for future timestamp values in CDC mode

Potential Causes

This error may occur for a Pipeline that is created with the CustomSQL or Table ingestion mode and is using the Change Data Capture (CDC) query mode.

The Change Data Capture (CDC) query mode uses the now() function to get new records and updates that are newer than the last ingested record. The now() function is a timestamp function that gets the exact timestamp of the moment at which the function is run. Read MySQL NOW() Function for more information.

If the Source timestamp values are in the future, that is, the selected timestamp column contains values that are in the future with respect to the Source database or server timezone, then there are no records older than the timestamp fetched by the now() function. Hence, Hevo does not have any records to ingest.

Suggested Action(s)

Set the Milliseconds to wait value to a negative value, for example, -25200000 for 7 hours, or a value as per the difference you expect the data to have from the current time of the Source database or server, using the Edit config option for the concerned object.

The Pipeline adds this value to the timestamp of the now() function to identify the records to ingest. The Milliseconds to wait setting enables the Pipeline to check the configured timestamp field’s value in the past for a positive value, and in the future for a negative value.
For example, if your Source has timestamp values 7 hours ahead of the current time, the value, -25200000 tells the Pipeline to check for timestamp values 7 hours in the future.

Revision History