Prioritizing a Pipeline
On This Page
Each run of a Pipeline constitutes several independent jobs, of which the two most important jobs are:
-
Fetching data from the Source
-
Upserting data to the Destination
Hevo’s job manager regulates the execution of all the jobs and ensures that they get their fair share of opportunity to run and concurrent executions don’t overwhelm the Source/Destination servers. The jobs that are submitted earlier are prioritized over the jobs submitted more recently.
High Priority Pipelines
At times, there may be a need for prioritizing the receipt of data from some Pipelines over the others. In these situations, you can elevate the priority of the Pipeline. For high priority Pipelines:
-
Log-based ingestion occurs continuously instead of the normal frequency of five minutes.
-
The data is read closer to the time of change, and consequently, is processed in almost real time.
Hevo’s job manager prioritizes the jobs corresponding to the High priority Pipelines only to the extent that other lower priority jobs are not deprived of execution for a long time.
Prioritization limits
You can classify Pipelines as High priority for up to 25% of the total number of unique Sources for a given team.
Note: The priority indication is only a hint to the job manager. The actual order of execution depends on several factors.
Changing the Priority of a Pipeline
You can increase or decrease the priority by toggling the priority arrow next to the Pipeline’s name either on the Pipeline search page or the Details page.
Priority arrow in Red indicates a High priority Pipeline.
