Hevo restricts the number of events that can be ingested in a given time window. If the number of events ingested in a given time window exceeds the designated limit, ingestion is stalled for the remaining duration of the time window. Post which the ingestion continues on its own.
The limits on ingestion for a given time window are defined in terms of the following parameters:
- Time window: A fixed sized time interval in which the events are counted. The window is typically 30 minutes long.
- Total limit: The maximum events, if ingested, that lead to temporary stalling of further ingestion of events in a given time window. The limit is applied on the account across all the pipelines.
- Source/Replay ingestion limits: The total limit is applied cumulatively to events directly polled from the source and replayed from the replay queue. Limits are further applied individually to source ingestion and replayed events so that one does not completely throttle the other.
Why does it exist?
Limits are placed primarily to protect users from ingesting a large number of events unknowingly, usually due to bad configuration. Limits allow users time to fix their configuration without oversubscribing their events quota.
What do you need to do?
The events will start getting ingested again after the interval. However, you can pause source ingestions to prioritize events ingested via the replay queue and vice versa.
In case you see a need to temporarily increase the rate limits for your pipelines, please contact support.