Scheduling a Pipeline

Last updated on Jun 21, 2024

The Pipeline schedule determines how frequently your Hevo Pipeline fetches the data from a Source. (It may take slightly longer for the data to be visible in your Destination). The Pipeline frequency also impacts the rate at which your Events quota gets consumed. Read Pipeline Frequency and Events Quota Consumption for more information.

A Pipeline may run multiple ingestion tasks to fetch data from your Source. All these tasks run as per a schedule defined for them. The schedule is displayed below the Source name in the Pipeline Summary.

Change Pipeline Schedule

For a Pipeline, depending on the Source and the Pipeline mode, you can:

Note: You must be assigned the Team Administrator, Team Collaborator, Pipeline Administrator, or Pipeline Collaborator role in Hevo to modify Pipeline schedule.

Read through the following sections for the details.


Changing the Default Ingestion Frequency

You can change the default ingestion schedule of the Pipeline by selecting a suggested value between the maximum and minimum frequency supported by the Source. Once you have selected the ingestion schedule, the ingestion of Events starts as per the new frequency.

Suppose you change the schedule at 11 AM (UTC) and set a 12-hour ingestion frequency. Then, the Events are ingested at a 12-hour interval starting immediately, which means at 11 AM (UTC) and 11 PM (UTC) daily.

Read the Data Replication section of the respective Source for these values. For some Sources, you can also set a custom ingestion frequency. Refer to section, Creating a Custom Ingestion Schedule.

Note: You can change the ingestion frequency at the Pipeline level only and not for individual objects.

To change the default ingestion frequency:

  1. In your Pipeline’s detailed view, click the More ( ) icon to open the Pipeline’s Action menu, and click Change Schedule.

    Change Schedule

  2. In the Change the Pipeline Ingestion Schedule pop-up window, select the desired frequency and click UPDATE.

    Change the default frequency

The changed schedule is applied immediately. Let us suppose you change the schedule at 1:00 PM (UTC) and set a 3 Hrs ingestion frequency. The Events are now ingested at 1:00 PM (UTC), 4:00 PM (UTC), 7:00 PM (UTC), and so on.

For teams created after Release 2.21, Hevo provides the option to change the Pipeline ingestion schedule for most Sources and suggests a standard set of ingestion frequencies. The following image displays these frequencies:

New Ingestion Frequencies

Note: For teams created after Release 2.21, the maximum ingestion frequency for log-based Pipelines is 12 hours.

Refer to the following table to see if the Pipeline ingestion schedule for your Source type can be changed:

Source Type Can Change Schedule?
Database Sources Yes

Note: Starting Release 2.21, you can Change Schedule from the Pipeline overview for new and existing log-based Pipelines.
Data Warehouse Sources Yes
File-based Sources Yes
SaaS Sources Yes
Webhook Sources No
Kafka-based Sources No

Note: As Events are pushed in real-time from Webhook and Kafka-based Sources, the ingestion schedule for Pipelines created with these Source types cannot be changed. Also, it is not possible to change the ingestion schedule for some SaaS Sources with strict API rate limits.


Manually Triggering Data Ingestion

In addition to the scheduled ingestion run, you can explicitly trigger the ingestion via the Run Now option:

  • To run the ingestion for the entire Pipeline, click Run Now in the Pipeline Actions menu.

  • To start the ingestion for a particular object, click Run Now in the Actions menu for the object.

    Trigger Data Ingestion

    Note: Triggering ingestion using Run Now does not affect the scheduled ingestion for the object. The subsequent ingestion occurs as per the frequency set for the Pipeline.


Creating a Custom Ingestion Schedule

Some Sources allow you to customize the data ingestion schedule. Read the Data Replication section of the respective Source to know more.

You can schedule the ingestion to run:

  • At a fixed interval: You can ingest data every n minutes or n hours, where n is an integer value. For example, you may want to ingest data from your Facebook Ads every two hours instead of the default one hour.

  • Daily: You can run the ingestion task as per a fixed schedule every day. For example, if you want to ingest data every two hours, after peak hours, say at 6 PM (UTC), then set the schedule to start ingestion at 6 PM (UTC), 8 PM (UTC), 10 PM (UTC), and so on.

Perform the following steps to create a custom ingestion schedule for your Pipeline:

  1. In your Pipeline’s detailed view, click the More ( More icon ) icon to open the Pipeline’s Action menu, and click Change Schedule.

    Change Schedule

  2. In the Change the Pipeline Ingestion Schedule pop-up window, select Custom, and then one of the following options:

    Create a Custom Schedule

    • Run at fixed interval: Select this option to set the ingestion frequency to a value other than those provided by Hevo. Specify the number of hours, and click UPDATE to save the custom schedule.

      Run at fixed interval

      The changed schedule is applied immediately and is visible in the Pipeline Summary.

      Fixed Interval - Pipeline Summary

    • Run Daily: Select this option to set a fixed ingestion schedule every day. To configure:

      Custom Schedule - Run daily

      1. Select a time from the drop-down. To add multiple custom time(s), click + ADD MORE. You can add up to 24 custom time values to your schedule.

      2. Click UPDATE to save your custom schedule.

      The changed schedule is applied immediately and is visible in the Pipeline Summary.

      Run Daily - Pipeline Summary

Similar to ingestion, you can also define the schedule for loading data into your Destination. The schedule applies to all the Pipelines that may be configured with that Destination. Due to this, the schedule cannot be controlled at the Pipeline level. Read Scheduling Data Load for a Destination.



Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Jun-21-2024 NA Updated information about maximum ingestion frequency for log-based Pipelines.
May-27-2024 NA Removed information about not being able to change ingestion frequency for Amazon DynamoDB.
Mar-05-2024 2.21 Removed information about not being able to change ingestion frequency for log-based Sources.
Mar-09-2023 NA Merged FAQs, How does the timing of Change Schedule work? and How do I change the ingestion frequency for individual tables? into this document.
Oct-17-2022 NA Updated screenshots in the page to reflect the latest UI.
Feb-21-2022 1.82 Organized the existing content into sections:
- Changing the Default Ingestion Frequency.
- Manually Triggering Data Ingestion.
- Added section, Creating a Custom Ingestion Schedule to explain the steps to create a custom ingestion schedule.
Nov-10-2021 NA - Added information in the Overview section about how Pipeline frequency impacts the Events Quota Consumption.
- Added a See Also link to the Pipeline Frequency and Events Quota Consumption page.
Oct-04-2021 1.73 Added information about how the Change Schedule feature can be used for scheduling a Pipeline.
Sep-20-2021 NA Added the last paragraph about scheduling data loads at the Destination level.

Tell us what went wrong