Rate Limit Exceptions (Edge)

SaaS Sources impose restrictions on how frequently data can be accessed within a defined time period. These restrictions are known as rate limits. These limits help prevent system overload and ensure fair usage across all users. For example, a Source may allow only 1000 requests per day.

A Rate Limit Exception (RLE) occurs when Hevo tries to fetch data from a Source but exceeds its rate limit. When this happens, the Source temporarily blocks any further requests, and Hevo is unable to continue fetching data until the limit resets.

RLEs can occur during data ingestion for several reasons, such as:

• Too many requests to the Source within a short period.

• Multiple Pipelines are making requests to the same Source simultaneously.

• Dynamic rate limits are imposed by the Source, which vary based on system traffic or other conditions.

Job Status

When a job encounters an RLE, Hevo marks it with one of the following statuses:

• Deferred: The job is completed, but one or more objects encountered an RLE during ingestion. Partial data for those objects is ingested and loaded to the Destination.

  

• Deferred with Failures: The job has completed, but some objects encountered an RLE during ingestion, and one or more objects have failed Events.

  

Object Status

In addition to visibility at the job level, Hevo marks individual Source objects with one of the following statuses when they are rate-limited:

• Deferred: Ingestion of the object was interrupted by an RLE. Offsets were saved, and the partially ingested data was loaded to the Destination.

  

• Deferred with Failures: The object encountered an RLE during ingestion, and one or more Events failed during ingestion or loading.

  

Hevo automatically handles RLEs to ensure the reliability of your Pipelines and prevent data loss, removing the need for manual action. To understand how Hevo handles RLEs, read Handling Rate Limit Exceptions.