Sync Frequency and Jobs (Edge)
Edge Pipeline is currently available under Early Access. You can request access to evaluate and test its features.
Hevo syncs data from the Source with the Destination as per the schedule defined by the sync frequency. Once the Pipeline is created, a sync job is triggered to ingest any existing (historical) data from the Source. No other job is triggered in the Pipeline until the historical load completes. Data is loaded to the Destination only after data ingestion for the selected historical sync duration is completed.
Hevo uses the sync frequency primarily to:
Trigger the Source connector Job
A job is triggered as per the sync interval for ingesting data (ingestor task), preparing the ingested data for loading, and finally loading it into the Destination tables. The loader task, which writes the data into the Destination table(s), is triggered only after the ingestor task is completed. A new job is not triggered until the previous job finishes loading data to the Destination. All jobs scheduled to run while the earlier job is in progress are skipped.
Suppose a Pipeline is set to sync data with the Destination every two hours. The following table shows the Pipeline schedule:
| Job # | Scheduled Start Time | Actual Start Time | Job End Time |
|---|---|---|---|
| 1 | 00:00 | 00:00 | 02:00 |
| 2 | 02:00 | 02:00 | 04:30 |
| 3 | 04:00 | Job is skipped | Job was skipped |
| 4 | 06:00 | 06:00 |
Display Pipeline statistics
In the case of log-based Pipelines, where data is ingested in a streaming manner, the sync frequency acts as a window to display the amount of work done by Hevo. For example, suppose the Pipeline is set to sync data with the Destination every hour. Here, the sync frequency is 1 Hour, and the statistics, such as the number of Events ingested, processed by the Pipeline, and those loaded to the Destination, are refreshed every hour.