Why is my billable Events count so high even though I selected Delta Timestamp as the query mode?

This issue may occur due to the settings that you define for the query mode while configuring the Source for your Pipeline.

The Delta Timestamp query mode uses the Update Timestamp Column of an object to keep track of the Events it has ingested. If this column has duplicate values, and the count of these duplicate values is more than the ingestion batch size of the Pipeline, some of these may fail to get ingested within the same batch and hence, are not loaded to the Destination. The default ingestion batch size for a Pipeline is 1M Events; however, you can get this modified on request.

Let us suppose that you have specified Export Time as the Update Timestamp Column in your sales data, and there are 1.1M Events with the same Export Time value, say T1.

In each run, the Pipeline queries the Source as follows,

SELECT *
FROM   table
WHERE  `updated_timestamp_column` >= last_polled_time
       AND `updated_timestamp_column` < Now() - delay
ORDER  BY `time_column` ASC
LIMIT  500;

to identify Events where Export Time >= the last ingested Event’s timestamp, and ingests 1M Events from the result set. As the Pipeline runs occur, an ingestion batch may contain some Events with export time as T1. Or, an entire batch may comprise Events with export time as T1. In either case, once the Pipeline queries for Events where Export Time >= T1, the entire 1.2M Events are found, but the Pipeline is able to ingest only 1M of these.

As a result, the Pipeline goes into an endless loop trying to ingest the Events with Export Time as T1, and keeps loading the data to the Destination, leading to a high billable Events count.

Such an issue can be avoided through careful selection and configuration of the query mode. Read Factors Affecting Event Usage - Query Modes to know more.

Note: You can find the Events usage information in the Load Status page for the Pipeline.

Revision History