Scheduling a Pipeline

The Pipeline schedule determines how frequently your Hevo Pipeline fetches the data from a Source. (It may take slightly longer for the data to be visible in your Destination). The Pipeline frequency also impacts the rate at which your Events quota gets consumed. Read Pipeline Frequency and Events Quota Consumption for more information.

A Pipeline may run multiple ingestion tasks to fetch data from your Source. All these tasks run as per a schedule defined for them. The schedule is displayed below the Source name in the Pipeline Summary.

For a Pipeline, depending on the Source and the Pipeline mode, you can:

Note: You must be assigned the Team Administrator, Team Collaborator, Pipeline Administrator, or Pipeline Collaborator role in Hevo, to modify Pipeline schedule.

• Change the default ingestion frequency of the Pipeline.

• Manually trigger data ingestion for the Pipeline or specific objects.

• Create a custom data ingestion schedule.

Read through the following sections for the details.

---

Changing the Default Ingestion Frequency

You can change the default ingestion schedule of the Pipeline by selecting a suggested value between the maximum and minimum frequency supported by the Source. Once you have selected the ingestion schedule, the ingestion of Events starts as per the new frequency.

Suppose you change the schedule at 11 AM (UTC) and set a 12-hour ingestion frequency. Then, the Events are ingested at a 12-hour interval starting immediately, which means, at 11 AM (UTC) and 11 PM (UTC) daily.

Read the Data Replication section of the respective Source for these values. For some Sources, you can also set a custom ingestion frequency. Refer to section, Creating a Custom Ingestion Schedule.

Note: You can change the ingestion frequency at the Pipeline level only, and not for individual objects.

To change the default ingestion frequency:

1. In your Pipeline’s detailed view, click the More ( ) icon to open the Pipeline’s Action menu, and click Change Schedule.

   

2. In the CHANGE PIPELINE SCHEDULE pop-up dialog, select the desired frequency, and click SCHEDULE.

   

The changed schedule is applied immediately. Let us suppose you change the schedule at 1:00 PM (UTC) and set a 3 hours ingestion frequency. The Events are now ingested at 1:00 PM (UTC), 4:00 PM (UTC), 7:00 PM (UTC), and so on.

The Change Schedule feature is not available for the following:

• Log-based Pipelines: A low ingestion frequency may cause the logs to accumulate, leading to their expiry. Therefore, to avoid data loss, the ingestion frequency for Log-based Pipelines cannot be altered.

• Webhooks and Kafka-based Pipelines: As Events are pushed in real-time from the Source, the schedule of these Pipelines cannot be changed.

• Some SaaS Sources: Some SaaS Sources impose strict API limits to restrict too frequent data reads. As a result, the ingestion frequency for these SaaS sources cannot be modified.

---

Manually Triggering Data Ingestion

In addition to the scheduled ingestion run, you can explicitly trigger the ingestion via the Run Now option:

• To run the ingestion for the entire Pipeline, click Run Now in the Pipeline Actions menu.

• To start the ingestion for a particular object, click Run Now in the Actions menu for the object.

  

  Note: Triggering ingestion using Run Now does not affect the scheduled ingestion for the object. The subsequent ingestion occurs as per the frequency set for the Pipeline.

---

Creating a Custom Ingestion Schedule

Some Sources allow you to customize the data ingestion schedule. Read the Data Replication section of the respective Source to know more.

You can schedule the ingestion to run:

• At a fixed interval: You can ingest data every n minutes or n hours, where n is an integer value. For example, you may want to ingest data from your Facebook Ads every two hours instead of the default one hour.

• Daily: You can run the ingestion task as per a fixed schedule every day. For example, if you want to ingest data every two hours, after peak hours, say at 6 PM (UTC), then set the schedule to start ingestion at 6 PM (UTC), 8 PM (UTC), 10 PM (UTC), and so on.

Perform the following steps to create a custom ingestion schedule for your Pipeline:

1. In your Pipeline’s detailed view, click the More ( ) icon to open the Pipeline’s Action menu, and click Change Schedule.

   

2. In the CHANGE PIPELINE SCHEDULE pop-up window, select Custom, and then one of the following options:

   

   • Run at fixed interval: Select this option to set the ingestion frequency to a value other than those provided by Hevo. Specify the number of hours, and click SCHEDULE to save the custom schedule.

     

     The changed schedule is applied immediately and is visible in the Pipeline Summary.

     

   • Run Daily: Select this option to set a fixed ingestion schedule every day. To configure:

     

     1. Select a time from the drop-down. To add multiple custom time(s), click + ADD MORE. You can add up to 24 custom time values to your schedule.

     2. Click SCHEDULE to save your custom schedule.

     The changed schedule is applied immediately and is visible in the Pipeline Summary.

     

Similar to ingestion, you can also define the schedule for loading data into your Destination. The schedule applies to all the Pipelines that may be configured with that Destination. Due to this, the schedule cannot be controlled at the Pipeline level. Read Scheduling Data Load for a Destination.

---

Revision History