Scheduling Data Load for a Destination

You can define the frequency at which data must be loaded to the Destination based on when you need the data. As the loading frequency affects all Pipelines using the Destination, you cannot modify it for any individual Pipeline.

For a Destination, depending on the type, you can:

You cannot change the data loading frequency for the following Destination types:

  • Databases: In Destinations such as MySQL, Amazon Aurora MySQL, MS SQL Server, and PostgreSQL, data is loaded in near real-time.

  • Google BigQuery data warehouse with streaming enabled, as data is loaded in near real-time.

Read through the following sections for the details.


Changing the Default Loading Frequency

The ingested data is loaded to a Destination at a default loading frequency, which is set when the Destination is created. You can change this frequency to one of the values provided by Hevo. Some Destinations also allow you to set a custom loading frequency. Refer to section, Creating a Custom Data Loading Schedule.

To change the default data loading frequency for a Destination:

  1. Click DESTINATIONS in the Asset Palette.

  2. In the Destinations List View, click on the required Destination to open its Detailed View. You can see the default data loading frequency in the Summary Bar.

    Default Loading Frequency

  3. Click Change.

  4. Select the desired frequency, and click SCHEDULE.

    Change data loading frequency

    The revised frequency is displayed in the Destination’s Detailed View.

    Changed loading frequency


Creating a Custom Data Loading Schedule

Most database Destinations and a Google BigQuery Destination that loads data in streaming mode, do not allow you to set a custom loading frequency. For other Destinations, you can create a custom schedule to load the data.

Perform the following steps to create a custom loading schedule for your Destination:

  1. Click DESTINATIONS in the Asset Palette.

  2. In the Destinations List View, click on the required Destination to open its Detailed View. You can see the currently configured data loading frequency in the Summary Bar.

    Current Loading Frequency

  3. Click Change.

  4. In the CHANGE DESTINATION SYNC SCHEDULE pop-up window, click Custom, and select one of the following schedule types:

    • Run at a fixed interval: Select this option to set the data loading frequency to a value other than those provided by Hevo. For example, you may want to load data to your Amazon Redshift Destination every four hours instead of the default five minutes.

      Specify the number of hours, and click SCHEDULE to save the custom schedule.

      Custom Loading - Fixed Interval

      The changed schedule is applied immediately and is visible in the Destination’s Detailed View.

      Fixed Interval - Destination View

    • Run daily: Select this option to set a fixed, daily loading schedule. For example, you may want to load data every four hours during peak hours, and every two hours after peak hours. To configure:

      Custom Loading - Run Daily

      1. Select a time from the drop-down. To add multiple custom time(s), click + ADD MORE. You can add up to 24 custom time values to your schedule.

      2. Click SCHEDULE to save your custom schedule.

    The changed schedule is applied immediately and is visible in the Destination’s Detailed View.

    Run Daily - Destination View

Note: You can change the custom schedule type later, and also set the data loading frequency back to one of the values provided by Hevo.


Manually Triggering Data Loading

You can explicitly trigger the data loading task to load any pending Events displayed in the UI.

To do this, select Load Events Now from the actions available for the Destination. However, this loads all the pending Events for all Pipelines which use that Destination. Therefore, it is not recommended to explicitly trigger a data load.

Trigger Data Loading for Destination

To trigger a data load for a specific table, select the Load Events Now option from the Actions menu of the table.

Trigger Data Loading for Table

Similar to the data loading schedule, you can define a schedule for ingesting data from your Source in the Pipeline. Read Scheduling a Pipeline.



Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Mar-07-2022 1.83 - Added section, Creating a Custom Data Loading Schedule to explain the steps to create a custom data loading schedule.
- Reorganized the content for clarity.
Sep-20-2021 1.72 New document.
Last updated on 27 Jun 2022

Tell us what went wrong