Mismatch in Events Count in Source and Destination

There are multiple factors that determine how Hevo ingests the data from your Source and how it loads it to the Destination. These can have an impact on the number of Events finally loaded to your Destination.

These include:

• Failed and sidelined Events. Sometimes Events may fail to be processed or may be sidelined and this may cause fewer Events to show up in the Destination for some time till such Events are replayed and loaded. Read Reasons for Event Failures for more information.

• Events deleted in the Source. Hevo updates the __is_deleted field in the Destination table to True for fields that have been deleted in the Source and does not delete the Event. As a result, Event count in the Destination may be lower.

• Append Rows on Update setting. In case of Google BigQuery and Snowflake, based on whether data was deduplicated (default behavior) or the Append Rows on Update option was selected for the object post-Pipeline creation, the count in the Destination may be less or more than the ingested Events count, respectively. If you disable this option after having enabled it previously, uniqueness is ensured only for future records. Therefore, both old and new versions of the same record may still exist.

  

• Any restarts done for an object. The Pipeline overview page displays the overall data ingested for the object. If an object was restarted multiple times, the overview page shows more Events.

• Re-ingestion of data on each Pipeline run. For some Sources, data is re-ingested every time as the Source APIs do not provide a way to identify changed data. For example, Google Sheets. In such cases, the loaded Events count is higher.

• Transformations applied on the ingested data. Data may be split into new Events, some Events may be dropped, or new Events may be added using Transformations.

• Sub-optimal query mode setup. Read Why is my billable Events high with Delta Timestamp Column? for more information.

• Slower Destination load schedule. If the frequency of ingestion is higher than the load frequency, the Events may be parked in the staging tables or Hevo’s S3 buckets for some time.

You can differentiate between the billable and historical Events that are ingested and loaded, by hovering over the different graphs on the Pipeline Overview page or the graphs for each object in the Destination Overview page.

Revision History