Offset and Latency of Jobs and Objects (Edge)

Hevo maintains an offset to identify the position till where the job has picked up Events for processing. The offset, which is a string of bytes, is maintained for each Source object. In the case of log-based Pipelines, the offset is the log sequence number (LSN). This LSN is converted to a human-readable format and displayed on the user interface (UI) as a time value, comprising the date and time in UTC. The offset is an indicator of the Pipeline’s progress.

The offset value is preserved while the job is ongoing. Once ingestion is completed, it is updated with the position of the last successfully processed Event. The subsequent job in the schedule processes Events with an offset higher than this value. In case a job fails or completes with failures, the next job starts from the offset of the last successfully processed Event. If the job did not ingest any data from the Source in the run, the offset value remains unchanged.

---

Offset in Log-Based Pipelines

In log-based Pipelines, Hevo records the offset for a historical job to help monitor the Pipeline’s progress. However, if transactions are being recorded in the database log, the offset visible on the UI may be more recent, or if the position of the last transaction in the log is too far, the offset may seem to be less. The offset for an incremental job is determined from the log position and is the same for all the objects in a job. When an object is resynced from the Object Config tab, its offset at the object level is changed to the position of the last successfully processed Event. The resync operation, however, does not affect the offset of that object in the incremental job or the data loaded to the corresponding Destination table.

To view the offset of the objects in a job, do the following:

1. Log in to your Hevo account. By default, PIPELINES is selected in the Navigation Bar.

2. In the Pipelines List View, click the Edge tab and then click your Pipeline to view the list of jobs created for it.

   

3. In the list, click a job to open its details.

4. In the Objects section, hover over the value in the Offset column of any object to view the current offset. For example, in the image below, the offset for all the objects is Sep 29, 2024, 20:40 (UTC).

   

   Suppose the category object was resynced. As seen in the image below, its offset changed to Sep 29, 2024, 20:51 (UTC).

   

---

Job and Objects Latency

Latency is calculated as the difference between when the data was modified at the Source and when it was loaded to the Destination. It is also called data latency or end-to-end latency. Latency helps ascertain the freshness of data in the Destination. For example, if the latency is around an hour, then it could be that the data was ingested at least an hour ago. Hevo records the latency for a job as Max Latency, which is calculated when the entire batch of data has been loaded to the Destination. It is calculated as the maximum of latencies across all Source objects. Latency at the object level is determined as the difference between when the record was modified in the Source object and when it was loaded into the corresponding Destination table.

Latency metrics

Hevo provides certain metrics that can be used to determine the overall efficiency of your Pipeline. These metrics are applicable at the job and object levels. The metrics for an object are calculated using the values of some Hevo-generated metadata columns in the object and are persisted when the entire batch of data is loaded.

The following table lists the metrics and their definitions:

Viewing the job and object latency

To view the latency of a job:

1. Log in to your Hevo account. By default, PIPELINES is selected in the Navigation Bar.

2. In the Pipelines List View, click the Edge tab and then click your Pipeline to view the list of jobs created for it.

   

3. In the list, click any job whose Max Latency column is not empty to view its details.

4. In the Job Summary section, hover over the value in the Max Latency field to view the Min, Mean, and Max latencies of that job.

   

   As seen in the image above, the Min Latency is 10 min 55 sec, which is from object 2, and the Max Latency is 32 min 6 sec, which is from object 3.

To view the latency of an object:

1. Log in to your Hevo account. By default, PIPELINES is selected in the Navigation Bar.

2. In the Pipelines List View, click the Edge tab and then click your Pipeline to view the list of jobs created for it.

   

3. In the Objects section of the job, hover over a non-empty value in the Max Latency column. The Min, Mean, and Max latencies of the object are displayed in the hover window.

   As seen in the image below, for object 2, the Min Latency is 10 min 55 sec, the Mean Latency is 12 min 23 sec, and the Max Latency is 17 min 33 sec.