Why is there a mismatch in the count of Events in the Source and the Destination even though I selected the Unique and Incrementing query mode?

This issue may occur due to the settings that you define for the query mode while configuring the Source for your Pipeline.

The Unique and Incrementing query mode uses the Auto-Incrementing Column of an object to keep track of the last ingested Event. If the auto-incrementing column that you specify has duplicate values, some of these may fail to get ingested, and hence, loaded to the Destination.

Let us suppose that you have specified Lot Number as the auto-incrementing column in your sales data, and the data has details of three different business units, all of whom have used the same numbering model. As a result, the Lot Number column has duplicate values, such as, 1, 2, 3, 1, 2, 3, 1, 2, 3, and so on.

Further, let us suppose that there are 100 occurrences of Lot Number value 2.

As per the configured query mode, in each Pipeline run, Hevo queries the Source for Events where Lot Number is > n, where n is an integer value, and ingests 1M Events from the result set. As the Pipeline runs occur, a batch may end up containing a few Events with lot number as 2. In the subsequent Pipeline run, Hevo queries for the next incremented value: Events where Lot Number is > 2, and the result set includes Events where lot number would be 3 or higher.

As a result, all Events with Lot Number as 2 may not get ingested, and hence, the Events in the Destination are fewer than in the Source.

Such an issue can be avoided through careful selection and configuration of the query mode. Read Factors Affecting Event Usage - Query Modes to know more.

Note: You can find the Events usage information in the Load Status page for the Pipeline.

Revision History