Sync Frequency (Edge)

Hevo Pipelines ingest data from your Sources and synchronize it with a Destination of your choice on a single schedule, which is run as per the Sync Frequency. A single schedule removes the need to set up a separate ingestion frequency for ingesting data from the Source. Similarly, you are not required to configure a loading frequency for your Destination.

In a Pipeline configured to run on a sync frequency, data ingestion via the Source Connectors and data loading using the Consumers are synchronous activities. The Source connector job, which fetches data, is triggered at the start of the sync schedule. This job comprises multiple tasks, which ingest data from the Source and prepare it for loading. The consumer job, which loads data to the Destination, is triggered after the connector job is completed.

Subsequent connector jobs are triggered only on completion of the previous ones. If an earlier job is in progress at the following scheduled start, Hevo does not start another connector job, and the scheduled job is marked as SKIPPED.

Sync frequency and scheduling

By default, Hevo syncs data from your Source to the Destination every 6 hours. You can change the sync frequency while creating or editing a Pipeline by choosing a preset interval or entering a custom value.

• Preset intervals
  5, 10, 15, 20, 30, or 45 minutes

• Custom frequency
  Enter any integer between 1 and 24 to set a custom hourly frequency. If you set the frequency to 24 hours, you can also specify a fixed start time in hh:mm (UTC) format. Hevo will run the Pipeline once a day at the specified time.

Once the Pipeline is created, Hevo starts the Source connector job to ingest data at the configured frequency. If a previous job is in progress at the start of the next scheduled run, Hevo skips the ingestion job for that run.

The following is an example of a Pipeline configured to sync data every hour:

As seen above, Job 1 and Job 2 start at the scheduled time. However, Job 3 is skipped because Job 2 ran past its scheduled end time and was still in progress at 10:00. Hevo does not run two jobs simultaneously. Once Job 2 completes, Hevo picks up the next available slot, and Job 4 runs as scheduled at 11:00.

Retry Behavior on Job failure

When a job fails, Hevo’s retry behavior depends on the Pipeline sync frequency.

• Sync frequency under 1 hour: Retries occur at the next scheduled run.

• Sync frequency of 1 hour or more: Retries occur sooner, using a shorter retry interval defined by the Pipeline sync frequency, so data ingestion resumes without waiting for the next scheduled run.

The retry interval varies depending on the sync frequency, as listed in the table below:

The start time for the next job is calculated as follows:

Time of job failure + Retry interval

For example, consider a Pipeline configured to sync data every hour. If a job starts at 11:00 AM and fails at 11:10 AM, Hevo schedules the next attempt at 11:30 AM, which is 20 minutes after the failure, instead of waiting for the next scheduled run at 12:00 PM.

If this attempt also fails, subsequent jobs run as per the sync frequency.

Disabling Pipelines after seven days of continuous failures

If a Pipeline remains in a Failed state for seven days, Hevo disables it and notifies the users associated with the Pipeline and any users subscribed to alerts.

This prevents repeated failed ingestion attempts from consuming resources in the Source and Destination systems, and ensures that the underlying issues are addressed before data ingestion continues. Refer to Disabling your Edge Pipeline for more information about the actions Hevo performs when a Pipeline is disabled.

In such cases, resolve the issue and then manually enable the Pipeline to resume data ingestion.

---

Revision History