Scheduling a Pipeline

On This Page

A Pipeline in Hevo may run multiple ingestion jobs to fetch data from your Source. All these jobs run as per a schedule defined for them. The schedule is displayed below the Source name in the Pipeline Summary.

Pipeline Schedule

You can change the data ingestion schedule from the Pipeline’s Action menu. Every Source can have a different minimum and maximum ingestion frequency.

The changed schedule is applicable immediately after change. Let us suppose you change the schedule at 1.00 p.m. (UTC) and set a 4-hour ingestion frequency. The Events are now ingested at 1.00 p.m. (UTC), 5.00 p.m. (UTC), 9.00 p.m. (UTC), and so on.

The Change Schedule feature is not available for the following:

  • Log-based Pipelines: A high ingestion frequency causes accumulation of logs leading to their expiry.

  • Webhooks: As Events are pushed in real-time from the Source, the concept of schedule does not apply to Webhooks.

  • Sources that do not allow modifications to the ingestion frequency.

Other than the scheduled ingestion run, ingestion may be triggered explicitly via the Run Now option. To run the ingestion for the entire Pipeline, click Run Now in the Actions menu. To run the ingestion for a particular object, click Run Now in the Actions menu for the object.

Run ingestion for an object

NOTE: The frequency of a Pipeline determines the frequency at which data must be ingested from the Source. It may take slightly longer for the data to be visible in your Destination.

Similar to ingestion, you can also define the schedule for loading data into your Destination. The schedule applies to all the Pipelines that may be configured with that Destination. Due to this, the schedule cannot be controlled at the Pipeline level. Read Scheduling Data Load for a Destination.

See Also

Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Oct-04-2021 1.73 Added information about how the Change Schedule feature can be used for scheduling a Pipeline.
Sep-20-2021 NA Added the last paragraph about scheduling data loads at the Destination level.
Last updated on 07 Oct 2021