Scheduling a Pipeline

The Pipeline schedule determines how frequently Hevo fetches the data from a Source. (It may take slightly longer for the data to be visible in your Destination). The Pipeline frequency also impacts the rate at which your Events quota gets consumed. Read Pipeline Frequency and Events Quota Consumption for more information.

A Pipeline may run multiple ingestion tasks to fetch data from your Source. All these tasks run as per a schedule defined for them. The schedule is displayed below the Source name in the Pipeline Summary.

Change Pipeline Schedule

For a Pipeline, depending on the Source and the Pipeline mode, you can:

Read through the following sections for the details.


Changing the Default Ingestion Frequency

You can change the default ingestion schedule of the Pipeline by selecting a suggested value between the maximum and minimum frequency supported by the Source. Read the Data Replication section of the respective Source for these values. For some Sources, you can also set a custom ingestion frequency. Refer to section, Creating a Custom Ingestion Schedule.

To change the default ingestion frequency:

  1. In your Pipeline’s detailed view, click the More ( More icon ) icon to open the Pipeline’s Action menu, and click Change Schedule.

    Change Schedule

  2. In the CHANGE PIPELINE SCHEDULE pop-up dialog, select the desired frequency, and click SCHEDULE.

    Change the default frequency

The changed schedule is applied immediately. Let us suppose you change the schedule at 1:00 PM (UTC) and set a 3 hours ingestion frequency. The Events are now ingested at 1:00 PM (UTC), 4:00 PM (UTC), 7:00 PM (UTC), and so on.

The Change Schedule feature is not available for the following:

  • Log-based Pipelines: A low ingestion frequency may cause the logs to accumulate, leading to their expiry. Therefore, to avoid data loss, the ingestion frequency for Log-based Pipelines cannot be altered.

  • Webhooks and Kafka-based Pipelines: As Events are pushed in real-time from the Source, the schedule of these Pipelines cannot be changed.

  • SaaS Sources: Some SaaS Sources, such as Mixpanel, impose strict API limits to restrict too frequent data reads. As a result, the ingestion frequency for these SaaS sources cannot be modified.


Manually Triggering Data Ingestion

Other than the scheduled ingestion run, you can explicitly trigger the ingestion via the Run Now option. To run the ingestion for the entire Pipeline, click Run Now in the Pipeline Actions menu. To start the ingestion for a particular object, click Run Now in the Actions menu for the object.

Trigger Data Ingestion


Creating a Custom Ingestion Schedule

Some Sources allow you to customize the data ingestion schedule. Read the Data Replication section of the respective Source to know more.

You can schedule the ingestion to run:

  • At a fixed interval: You can ingest data every n minutes or n hours, where n is an integer value. For example, you may want to ingest data from your Facebook Ads every two hours instead of the default one hour.

  • Daily: You can run the ingestion task as per a fixed schedule every day. For example, if you want to ingest data every two hours, after peak hours, say at 6 PM (UTC), then set the schedule to start ingestion at 6 PM (UTC), 8 PM (UTC), 10 PM (UTC), and so on.

Perform the following steps to create a custom ingestion schedule for your Pipeline:

  1. In your Pipeline’s detailed view, click the More ( More icon ) icon to open the Pipeline’s Action menu, and click Change Schedule.

    Change Schedule

  2. In the CHANGE PIPELINE SCHEDULE pop-up window, select Custom, and then one of the following options:

    Create a Custom Schedule

    • Run at fixed interval: Select this option to set the ingestion frequency to a value other than those provided by Hevo. Specify the number of hours, and click SCHEDULE to save the custom schedule.

      Run at fixed interval

      The changed schedule is applied immediately and is visible in the Pipeline Summary.

      Fixed Interval -  Pipeline Summary

    • Run Daily: Select this option to set a fixed ingestion schedule every day. To configure:

      Custom Schedule - Run daily

      1. Select a time from the drop-down. To add multiple custom time(s), click + ADD MORE. You can add up to 24 custom time values to your schedule.

      2. Click SCHEDULE to save your custom schedule.

      The changed schedule is applied immediately and is visible in the Pipeline Summary.

      Run Daily -  Pipeline Summary

Similar to ingestion, you can also define the schedule for loading data into your Destination. The schedule applies to all the Pipelines that may be configured with that Destination. Due to this, the schedule cannot be controlled at the Pipeline level. Read Scheduling Data Load for a Destination.



Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Oct-17-2022 NA Updated screenshots in the page to reflect the latest UI.
Feb-21-2022 1.82 Organized the existing content into sections:
- Changing the Default Ingestion Frequency.
- Manually Triggering Data Ingestion.
- Added section, Creating a Custom Ingestion Schedule to explain the steps to create a custom ingestion schedule.
Nov-10-2021 NA - Added information in the Overview section about how Pipeline frequency impacts the Events Quota Consumption.
- Added a See Also link to the Pipeline Frequency and Events Quota Consumption page.
Oct-04-2021 1.73 Added information about how the Change Schedule feature can be used for scheduling a Pipeline.
Sep-20-2021 NA Added the last paragraph about scheduling data loads at the Destination level.
Last updated on 28 Nov 2022

Tell us what went wrong