Scheduling a Pipeline
On This Page
A Pipeline in Hevo may run multiple ingestion jobs to fetch data from your Source. All these jobs run as per a schedule defined for them. The schedule is displayed below the Source name in the Pipeline Summary.
You can change the data ingestion schedule from the Pipeline’s Action menu. Every Source can have a different minimum and maximum ingestion frequency.
The changed schedule is applicable immediately after change. Let us suppose you change the schedule at 1.00 p.m. (UTC) and set a 4-hour ingestion frequency. The Events are now ingested at 1.00 p.m. (UTC), 5.00 p.m. (UTC), 9.00 p.m. (UTC), and so on.
The Change Schedule feature is not available for the following:
Log-based Pipelines: A high ingestion frequency causes accumulation of logs leading to their expiry.
Webhooks: As Events are pushed in real-time from the Source, the concept of schedule does not apply to Webhooks.
Sources that do not allow modifications to the ingestion frequency.
Other than the scheduled ingestion run, ingestion may be triggered explicitly via the Run Now option. To run the ingestion for the entire Pipeline, click Run Now in the Actions menu. To run the ingestion for a particular object, click Run Now in the Actions menu for the object.
NOTE: The frequency of a Pipeline determines the frequency at which data must be ingested from the Source. It may take slightly longer for the data to be visible in your Destination.
Similar to ingestion, you can also define the schedule for loading data into your Destination. The schedule applies to all the Pipelines that may be configured with that Destination. Due to this, the schedule cannot be controlled at the Pipeline level. Read Scheduling Data Load for a Destination.
Refer to the following table for the list of key updates made to this page:
|Date||Release||Description of Change|
|Oct-04-2021||1.73||Added information about how the Change Schedule feature can be used for scheduling a Pipeline.|
|Sep-20-2021||NA||Added the last paragraph about scheduling data loads at the Destination level.|