Handling Rate Limit Exceptions (Edge)

When a Source object encounters a Rate Limit Exception (RLE) during ingestion, Hevo:

• Captures the data ingested up to that point and saves the corresponding offset.

• Loads the partially ingested data to the Destination as part of the same job.

• Resumes ingestion for the rate-limited object from the saved offset in the next run, avoiding duplicate data fetches.

• Updates job and object statuses to indicate which objects encountered RLEs.

This approach ensures that Pipelines remain efficient and transparent, with no loss of progress during ingestion.

---

Handling Rate Limit Exceptions during Historical Job

When an RLE occurs during a historical job, Hevo saves the offset up to which data was ingested and attempts to load the partial data into the Destination. If the partial load is successful, both the job and the object are marked as Deferred. In the next scheduled run, another historical job resumes ingestion from the saved offset for deferred objects, while completed objects are skipped. Historical jobs will continue to run until no objects remain in a deferred state.

However, if the partially ingested data fails to load, the object is marked as Failed. In this case, the next scheduled job switches to incremental mode, restarting ingestion for the failed object rather than resuming from the saved offset.

---

Handling Rate Limit Exceptions during Incremental Job

During an incremental job, if any Source Object encounters an RLE, Hevo saves the offset up to which data was ingested, stops ingestion immediately, and sends the partially ingested data for loading.

If the data is loaded to the Destination, the object is marked as Deferred. The next job will still be an incremental job, scheduled according to the Pipeline frequency. In the subsequent run, the job resumes ingestion for the deferred object from the last saved offset, while other objects are processed as part of the incremental job as usual.

---

Handling Rate Limit Exceptions during Truncate and Load Job

Truncate and Load (TnL) jobs are one-time operations triggered when you resync an object, resync the entire Pipeline, or pause and then resume the Pipeline. Unlike historical or incremental jobs, TnL jobs do not run on a schedule.

If an RLE occurs during a TnL job, Hevo retries ingestion for the rate-limited object up to five times, with a one-hour interval between each retry to avoid incomplete data ingestion. If you want to increase the number of retries, contact Hevo Support.

During each retry, Hevo resumes ingestion for the rate-limited object, starting from the last saved offset. If the object is successfully ingested and loaded during any retry, it is marked as Completed. If all five retries are exhausted without completing ingestion, the object is marked as Deferred and picked up for ingestion in the next scheduled job.

Note: If an incremental job is scheduled while an object is being retried, Hevo skips the entire incremental job. Once the retry attempts are exhausted or the object is successfully ingested, the incremental job is triggered as scheduled.