Ingestion Frequency and Data Synchronization

Ingestion frequency, or Pipeline frequency, is the frequency at which Hevo ingests data from the Source. It affects how your Events quota is consumed.

Pipeline frequency does not affect the quota consumed for historical data as all historical loads are free in Hevo. Moreover, the historical data is ingested once when you create the Pipeline and is not ingested on every subsequent run of the Pipeline unless you restart it. However, in case of the periodic data refresh, as done for ad-based Sources, the Events ingested during a data refresh count towards your Events quota consumption. As data refresh happens with each run of the Pipeline, optimizing the latter can impact your quota consumption.

Incremental data ingestion impacts your quota consumption depending on the type of the Source:

• In case of webhook Sources, Sources using log-based replication, and some SaaS Sources, only new and modified Events are ingested in each run of the Pipeline. Therefore, Pipeline frequency does not impact your quota consumption.

• In case of some SaaS Sources, where you may end up ingesting duplicate Events based on the Source API’s frequency vs the Pipeline frequency, excessive quota may get consumed.

  Example:

  In Microsoft Ads, the smallest granularity available in the API request is Daily (Hourly is not supported). This means that every time Hevo makes a request to extract the data from Microsoft Ads, it receives the data for the entire day. Thus, Hevo must ingest the incremental data for the entire day on every run of the Pipeline. So, along with the new Events, any Events that were already ingested previously are re-ingested and count towards your quota consumption.

  Read Factors Affecting Event Usage - Pipeline Frequency.

Revision History