Pipeline Jobs

A Job is an immutable set of tasks for all objects, child objects, and tables at the Source and Destination levels that are used to replicate your data. It provides you with both a high-level and a granular view of the entire data movement lifecycle from the Source to the Destination along with any associated metadata. Each job can be seen as one complete Pipeline run, encompassing all the stages of data replication: Ingestion → Transformation → Schema Mapping → Loading.

You can view the job details to know:

• The timeliness of data replication.

• The data volume correctness.

• The Events being replicated at any stage in a Pipeline run (Job).

• Errors logged in each Pipeline run.

These details can help you identify any data mismatches, latencies, and Event consumption spikes.

A job is created when you configure your Pipeline, based on the type of data that is to be ingested, for example, historical or incremental jobs. Each job is assigned a Job ID; you can use this ID to track the job details in your Audit Tables. For SaaS Sources and Pipelines with ingestion mode as Table, a job includes all the Source objects selected for ingestion and is run as per the defined Pipeline schedule. For log-based ingestion, Hevo reads the streaming logs every 5 minutes, however, the data is loaded every hour.

You can view the jobs for all the Pipelines created by your team or view the jobs for a specific Pipeline. Further, within each job, you can view the details of each object.

Note: If your Pipeline uses Python code-based or Drag-and-Drop Transformations to modify ingested Events, the displayed job details, such as status and number of objects processed, may not be accurate.

---

Accessing the Jobs List

Use the All Jobs tab on the Navigation Bar to view, track, and manage all your Pipeline jobs in one place.

Alternatively, use the Jobs tab on the Tools Bar of a Pipeline to view the list of jobs created for it.

The Jobs list displays the following summary information for a job:

The sum of loaded and failed Events in a job equals the ingested Events. If Events are created or deleted using Transformations, you may find a count mismatch.

---

Job and Object Statuses

If any object has Event failures during a job run, the job remains in In-Progress status while Hevo tries to replay the Events. If you find any object with failed Events in the job details, you can restart the object using the Restart Now or Run Now action. This creates a Historical job with just that object.

---

Viewing Job and Object Details

To view the details of a job, such as the list of objects processed by it and the information about each object:

1. Access the Jobs List.

2. Click the arrow at the right to view additional details for the job.

   

3. View the job details and the list of objects processed in it.

   

4. Optionally, click the drop-down next to an object to view the ingestion and loading details for it.

   

   The object details section displays:

   • The ingestion and loading stages of data replication.

   • The details of each stage, for example, the start time and duration for which the task ran, the number of Events read vs ingested or loaded, and the count of failed Events if any.

   • The number of failed Events for each failure reason. Errors that are transient in nature are automatically resolved by Hevo and no action is required from you for these.

---

See Also

• Familiarizing with the Pipelines UI

---

Revision History