Scheduling a Pipeline

The Pipeline schedule determines how frequently your Hevo Pipeline fetches the data from a Source. (It may take slightly longer for the data to be visible in your Destination). The Pipeline frequency also impacts the rate at which your Events quota gets consumed. Read Pipeline Frequency and Events Quota Consumption for more information.

A Pipeline may run multiple ingestion tasks to fetch data from your Source. All these tasks run as per a schedule defined for them. The schedule is displayed below the Source name in the Pipeline Summary.

For a Pipeline, depending on the Source and the Pipeline mode, you can:

• Change the default ingestion frequency of the Pipeline.

• Manually trigger data ingestion for the Pipeline or specific objects.

• Create a custom data ingestion schedule.

Note: You must be assigned the Team Administrator, Team Collaborator, Pipeline Administrator, or Pipeline Collaborator role in Hevo to modify Pipeline schedule.

Read through the following sections for the details.

---

Changing the Default Ingestion Frequency

You can change the default ingestion schedule of the Pipeline by selecting a suggested value between the maximum and minimum frequency supported by the Source. Once you have selected the ingestion schedule, the ingestion of Events starts as per the new frequency.

Suppose you change the schedule at 11 AM (UTC) and set a 12-hour ingestion frequency. Then, the Events are ingested at a 12-hour interval starting immediately, which means at 11 AM (UTC) and 11 PM (UTC) daily.

Read the Data Replication section of the respective Source for these values. For some Sources, you can also set a custom ingestion frequency. Refer to section, Creating a Custom Ingestion Schedule.

Note: You can change the ingestion frequency at the Pipeline level only and not for individual objects.

To change the default ingestion frequency:

1. In your Pipeline’s detailed view, click the More ( ) icon to open the Pipeline’s Action menu, and click Change Schedule.

   

2. In the Change the Pipeline Ingestion Schedule pop-up window, select the desired frequency and click UPDATE.

   

The changed schedule is applied immediately. Let us suppose you change the schedule at 1:00 PM (UTC) and set a 3 Hrs ingestion frequency. The Events are now ingested at 1:00 PM (UTC), 4:00 PM (UTC), 7:00 PM (UTC), and so on.

For teams created after Release 2.21, Hevo provides the option to change the Pipeline ingestion schedule for most Sources and suggests a standard set of ingestion frequencies. The following image displays these frequencies:

Note: For teams created after Release 2.21, the maximum ingestion frequency for log-based Pipelines is 12 hours.

Refer to the following table to see if the Pipeline ingestion schedule for your Source type can be changed:

Source Type	Can Change Schedule?
Database Sources	Yes Note: Starting Release 2.21, you can Change Schedule from the Pipeline overview for new and existing log-based Pipelines.
Data Warehouse Sources	Yes
File-based Sources	Yes
SaaS Sources	Yes
Webhook Sources	No
Kafka-based Sources	No

Note: As Events are pushed in real-time from Webhook and Kafka-based Sources, the ingestion schedule for Pipelines created with these Source types cannot be changed. Also, it is not possible to change the ingestion schedule for some SaaS Sources with strict API rate limits.

---

Manually Triggering Data Ingestion

In addition to the scheduled ingestion run, you can explicitly trigger the ingestion via the Run Now option:

• To run the ingestion for the entire Pipeline, click Run Now in the Pipeline Actions menu.

• To start the ingestion for a particular object, click Run Now in the Actions menu for the object.

  

  Note: Triggering ingestion using Run Now does not affect the scheduled ingestion for the object. The subsequent ingestion occurs as per the frequency set for the Pipeline.

---

Creating a Custom Ingestion Schedule

Some Sources allow you to customize the data ingestion schedule. Read the Data Replication section of the respective Source to know more.

You can schedule the ingestion to run:

• At a fixed interval: You can ingest data every n minutes or n hours, where n is an integer value. For example, you may want to ingest data from your Facebook Ads every two hours instead of the default one hour.

• Daily: You can run the ingestion task as per a fixed schedule every day. For example, if you want to ingest data every two hours, after peak hours, say at 6 PM (UTC), then set the schedule to start ingestion at 6 PM (UTC), 8 PM (UTC), 10 PM (UTC), and so on.

Perform the following steps to create a custom ingestion schedule for your Pipeline:

1. In your Pipeline’s detailed view, click the More ( ) icon to open the Pipeline’s Action menu, and click Change Schedule.

   

2. In the Change the Pipeline Ingestion Schedule pop-up window, select Custom, and then one of the following options:

   

   • Run at fixed interval: Select this option to set the ingestion frequency to a value other than those provided by Hevo. Specify the number of hours, and click UPDATE to save the custom schedule.

     

     The changed schedule is applied immediately and is visible in the Pipeline Summary.

     

   • Run Daily: Select this option to set a fixed ingestion schedule every day. To configure:

     

     1. Select a time from the drop-down. To add multiple custom time(s), click + ADD MORE. You can add up to 24 custom time values to your schedule.

     2. Click UPDATE to save your custom schedule.

     The changed schedule is applied immediately and is visible in the Pipeline Summary.

     

Similar to ingestion, you can also define the schedule for loading data into your Destination. The schedule applies to all the Pipelines that may be configured with that Destination. Due to this, the schedule cannot be controlled at the Pipeline level. Read Scheduling Data Load for a Destination.

---

Revision History