Prioritizing pipelines

Each run of a pipeline constitutes of several independent jobs. Two of the most important jobs are:

  • Fetching data from the source
  • Upserting data to the destination

Hevo’s job manager regulates the execution of all the jobs and ensures that the jobs get their fair share of opportunity to run and ensures that concurrent executions don’t overwhelm the source/destination servers. The execution of jobs that are submitted earlier in time is prioritized over the execution of jobs submitted more recently.

However, at times, there may be a need for prioritizing receiving data from some pipelines over the others. In these situations, the priority of the pipelines may be elevated.

Increasing the priority of a pipeline

The priority of a pipeline may be set to high by clicking on the priority arrow to the left of pipeline number either on the pipeline search page or the details page.

priority high

Decreasing the priority of a pipeline

The priority may be lowered by clicking on the priority arrow to the left of pipeline number either on the pipeline search page or the details page.

priority low

Limits

At most 20 percent of the pipelines belonging to a team may be marked with a high priority. The job manager would try to prioritize the jobs corresponding to these pipelines over the jobs belonging to other pipelines only up to a point so that other low priority jobs are not deprived of execution for a long time.

Note that the priority indication is only a hint to the job manager. The actual order of execution depends on several factors.