Prioritizing a Pipeline

For every Pipeline you create, Hevo executes multiple tasks for ingesting data from the Source and loading it to the Destination. Hevo’s task manager executes these tasks on the basis of the Source configurations and Pipeline priority. However, if you want data from a Pipeline urgently or want to analyze it in near real-time, Hevo provides you with the feature of prioritizing the Pipeline. Prioritization ensures that Hevo replicates your business-critical data first while ensuring resource availability for other Pipelines.

For all high-priority Pipelines, Hevo:

• Reads data from database and SaaS Sources closer to the time of change instead of at the configured ingestion frequency. For example, suppose you create a Pipeline with a 15-minute ingestion frequency and data is added in your Source every five minutes. Then, if you increase the priority of the Pipeline, Hevo replicates the data as soon as it is added to the Source.

• Ingests data for all log-based Pipelines in real-time instead of at the configured ingestion frequency. This ensures that the data is ingested as soon as it is added to the Source.

• Processes and loads the ingested data on priority over other Pipelines.

Note: Prioritization of Pipelines and the higher ingestion frequency does not mean that the lower priority Pipelines are not executed or are executed with a delay. They continue to be run as per their ingestion and loading schedules.

Prioritization Limits

You can prioritize only up to 25% of Pipelines created with unique Source configurations. For example, suppose you create 12 Pipelines with different Sources. Then, you can prioritize three of these. Now, suppose you have 12 Pipelines and six of these have MySQL, while the other six have Google Sheets as the Source. If the MySQL Pipelines have the same database host, then for prioritization purposes, they are considered as one Pipeline. Similarly, if the Google Sheets Pipelines have the same authorized user, they are considered as one Pipeline. Therefore, you can prioritize these 12 Pipelines without exceeding the prioritization limit. For each Source, configuration uniqueness may be derived based on different parameters.

Changing the Priority of a Pipeline

Note: You must be assigned the Team Administrator, Team Collaborator, Pipeline Administrator, or Pipeline Collaborator role in Hevo, to change the priority of a Pipeline.

You can change the priority of a Pipeline to High by clicking the arrow ( ) next to the Pipeline number visible at one of the following locations:

• Asset List of the Pipeline List View

  

• Summary bar of the Pipeline Detailed View

  

Note: The arrow ( ) is deactivated for streaming Pipelines and remains inactive even when the ingestion schedule is changed to frequency-based.

Your high priority Pipelines are represented with a red arrow in the Pipeline List View.

Note: The order in which the high priority Pipelines are displayed on the user interface does not define their actual order of execution; that depends on the resource availability for your account.

---

Revision History