Sync Frequency (Edge)

Hevo Pipelines ingest data from your Sources and synchronize it with a Destination of your choice on a single schedule, which is run as per the Sync Frequency. A single schedule removes the need to set up a separate ingestion frequency for ingesting data from the Source. Similarly, you are not required to configure a loading frequency for your Destination.

In a Pipeline configured to run on a sync frequency, data ingestion via the Source Connectors and data loading using the Consumers are synchronous activities. The Source connector job, which fetches data, is triggered at the start of the sync schedule. This job comprises multiple tasks, which ingest data from the Source and prepare it for loading. The consumer job, which loads data to the Destination, is triggered after the connector job is completed.

Subsequent connector jobs are triggered only on completion of the previous ones. If an earlier job is in progress at the following scheduled start, Hevo does not start another connector job, and the scheduled job is marked as SKIPPED.

Sync frequency and scheduling

By default, Hevo syncs data from your Source with the Destination every hour. You can change the sync frequency while creating the Pipeline by selecting one of the values provided by Hevo or specifying a custom value; however, this value must not exceed 24 hours.

Once the Pipeline is created, Hevo starts the Source connector job to ingest data according to the set frequency. If at the start of the following schedule, a previous job is in progress, Hevo skips initiating the ingestion job by an interval equal to the sync frequency.

The following is an example of a Pipeline configured to sync data every hour:

As seen above, Job 1 and Job 2 start at the scheduled time. However, Job 3 is skipped, as Job 2 ended at 10:25, which is beyond the scheduled start time (10:00) for Job 3. Further, as no job is in progress at 11:00, the next scheduled start time, Job 4 is run at 11:00.