What do I do when there is delay in my Pipeline?
ON THIS PAGE
Hevo is built on real-time data ingestion architecture. That means Hevo ingests data in real-time and writes to the destination as soon as possible. Hevo’s architecture allows it to scale horizontally whenever it detects a higher volume of events being ingested through the Pipelines. Still, there are situations where you might see a delay in your Pipelines.
Below are a few possible reasons:
- Increase in events being ingested. There might be an increase in the number of events being ingested from one or more of your Pipelines. In such a scenario, you may want to pause the Pipelines which are showing the highest activity depending on their criticality and resume it in an off-peak hour.
- Replayed a large number of failed events. A large number of events may have been replayed in one or more of your Pipelines. Replayed events are fed back to the Pipelines. They share the same resources that are used by the Pipeline itself. In this scenario, you can stop replaying the events in the Pipeline from the Overview Section.
- Slow Destination. In cases of Destinations where data upload is not done through files, e.g. MySQL and Postgres, a slow Destination may result in the Pipeline experiencing delays. It happens because Hevo is not able to write data into the Destination as fast as it is ingesting data from the Source. If you think the slowness in the Destination is a temporary situation, you may wait out till it gets resolved, else you should upgrade your destination to a better hardware configuration that allows it to accept a higher rate of writes.
Things to Note
- Delay in one of your Pipelines may result in a delay in other Pipelines as well. All pipelines in your account share the same resources and hence, compete for the same resources amongst each other.
- Pipelines of different accounts on Hevo do not affect each other. So, delay in your Pipelines is not a result of activity on Hevo outside your account.
- Hevo maintains a cache of data for up to 6 hours until the data has been written to the destination. If the delay is beyond 6 hours, Hevo might start dropping events altogether. If the delay in your pipelines is consistently very high please reach out to our support.