Scheduling a Pipeline

Last updated on Feb 27, 2024

The Pipeline schedule determines how frequently your Hevo Pipeline fetches the data from a Source. (It may take slightly longer for the data to be visible in your Destination). The Pipeline frequency also impacts the rate at which your Events quota gets consumed. Read Pipeline Frequency and Events Quota Consumption for more information.

A Pipeline may run multiple ingestion tasks to fetch data from your Source. All these tasks run as per a schedule defined for them. The schedule is displayed below the Source name in the Pipeline Summary.

Change Pipeline Schedule

For a Pipeline, depending on the Source and the Pipeline mode, you can:

Note: You must be assigned the Team Administrator, Team Collaborator, Pipeline Administrator, or Pipeline Collaborator role in Hevo, to modify Pipeline schedule.

Read through the following sections for the details.


Changing the Default Ingestion Frequency

You can change the default ingestion schedule of the Pipeline by selecting a suggested value between the maximum and minimum frequency supported by the Source. Once you have selected the ingestion schedule, the ingestion of Events starts as per the new frequency.

Suppose you change the schedule at 11 AM (UTC) and set a 12-hour ingestion frequency. Then, the Events are ingested at a 12-hour interval starting immediately, which means, at 11 AM (UTC) and 11 PM (UTC) daily.

Read the Data Replication section of the respective Source for these values. For some Sources, you can also set a custom ingestion frequency. Refer to section, Creating a Custom Ingestion Schedule.

Note: You can change the ingestion frequency at the Pipeline level only, and not for individual objects.

To change the default ingestion frequency:

  1. In your Pipeline’s detailed view, click the More ( More icon ) icon to open the Pipeline’s Action menu, and click Change Schedule.

    Change Schedule

  2. In the CHANGE PIPELINE SCHEDULE pop-up window, select the desired frequency, and click SCHEDULE.

    Change the default frequency

The changed schedule is applied immediately. Let us suppose you change the schedule at 1:00 PM (UTC) and set a 3 hours ingestion frequency. The Events are now ingested at 1:00 PM (UTC), 4:00 PM (UTC), 7:00 PM (UTC), and so on.

The Change Schedule feature is not available for the following:

  • Log-based Pipelines: A low ingestion frequency may cause the logs to accumulate, leading to their expiry. Therefore, to avoid data loss, the ingestion frequency for Log-based Pipelines cannot be altered.

  • Webhooks and Kafka-based Pipelines: As Events are pushed in real-time from the Source, the schedule of these Pipelines cannot be changed.

  • Some SaaS Sources: Some SaaS Sources impose strict API limits to restrict too frequent data reads. As a result, the ingestion frequency for these SaaS sources cannot be modified.


Manually Triggering Data Ingestion

In addition to the scheduled ingestion run, you can explicitly trigger the ingestion via the Run Now option:

  • To run the ingestion for the entire Pipeline, click Run Now in the Pipeline Actions menu.

  • To start the ingestion for a particular object, click Run Now in the Actions menu for the object.

    Trigger Data Ingestion

    Note: Triggering ingestion using Run Now does not affect the scheduled ingestion for the object. The subsequent ingestion occurs as per the frequency set for the Pipeline.


Creating a Custom Ingestion Schedule

Some Sources allow you to customize the data ingestion schedule. Read the Data Replication section of the respective Source to know more.

You can schedule the ingestion to run:

  • At a fixed interval: You can ingest data every n minutes or n hours, where n is an integer value. For example, you may want to ingest data from your Facebook Ads every two hours instead of the default one hour.

  • Daily: You can run the ingestion task as per a fixed schedule every day. For example, if you want to ingest data every two hours, after peak hours, say at 6 PM (UTC), then set the schedule to start ingestion at 6 PM (UTC), 8 PM (UTC), 10 PM (UTC), and so on.

Perform the following steps to create a custom ingestion schedule for your Pipeline:

  1. In your Pipeline’s detailed view, click the More ( More icon ) icon to open the Pipeline’s Action menu, and click Change Schedule.

    Change Schedule

  2. In the CHANGE PIPELINE SCHEDULE pop-up window, select Custom, and then one of the following options:

    Create a Custom Schedule

    • Run at fixed interval: Select this option to set the ingestion frequency to a value other than those provided by Hevo. Specify the number of hours, and click SCHEDULE to save the custom schedule.

      Run at fixed interval

      The changed schedule is applied immediately and is visible in the Pipeline Summary.

      Fixed Interval -  Pipeline Summary

    • Run Daily: Select this option to set a fixed ingestion schedule every day. To configure:

      Custom Schedule - Run daily

      1. Select a time from the drop-down. To add multiple custom time(s), click + ADD MORE. You can add up to 24 custom time values to your schedule.

      2. Click SCHEDULE to save your custom schedule.

      The changed schedule is applied immediately and is visible in the Pipeline Summary.

      Run Daily -  Pipeline Summary

Similar to ingestion, you can also define the schedule for loading data into your Destination. The schedule applies to all the Pipelines that may be configured with that Destination. Due to this, the schedule cannot be controlled at the Pipeline level. Read Scheduling Data Load for a Destination.



Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Mar-09-2023 NA Merged FAQs, How does the timing of Change Schedule work? and How do I change the ingestion frequency for individual tables? into this document.
Oct-17-2022 NA Updated screenshots in the page to reflect the latest UI.
Feb-21-2022 1.82 Organized the existing content into sections:
- Changing the Default Ingestion Frequency.
- Manually Triggering Data Ingestion.
- Added section, Creating a Custom Ingestion Schedule to explain the steps to create a custom ingestion schedule.
Nov-10-2021 NA - Added information in the Overview section about how Pipeline frequency impacts the Events Quota Consumption.
- Added a See Also link to the Pipeline Frequency and Events Quota Consumption page.
Oct-04-2021 1.73 Added information about how the Change Schedule feature can be used for scheduling a Pipeline.
Sep-20-2021 NA Added the last paragraph about scheduling data loads at the Destination level.

Tell us what went wrong