Why is there a delay in my Pipeline?

Hevo is built on real-time data ingestion architecture. This means that Hevo ingests data in real-time and writes to the Destination as soon as possible. Hevo’s architecture allows it to scale horizontally whenever it detects a higher volume of Events being ingested through the Pipelines. Still, there are situations where you might see a delay in your Pipelines, depending on your Pipeline configuration such as:

• Ingestion and Loading Frequencies of your Pipeline, which affect how often the data is replicated from your Source to the Destination. If you require near real-time updates in your Destination, you should configure a high ingestion and loading frequency. However, based on the Destination, increasing the load frequency may increase the cost of your load queries as more frequent loads can trigger additional query or compute charges. For more details, refer Ingestion and Loading Frequency. Hevo’s ingestion speed is influenced by:

  • API rate limits imposed by the Source. If the Source enforces rate limits, Hevo may not be able to ingest data as quickly as needed.

  • Network throughput affects the speed of data transfer.

  • Source throughput determines how quickly data is made available for ingestion.

  • Large Source datasets or frequent updates can introduce delays, as the ingestion process may not be able to keep up with the volume or frequency of data updates.

  • Geographical location of the server hosting your Source database can impact ingestion speed. If the server is located in a different region than the Hevo Pipeline, network latency can slow down ingestion. To reduce such delays, ensure that the server and the Hevo Pipeline are located in the same region.

• Applying complex Transformations on your Source data, which can cause delays in loading the data to the Destination.

• Replay of a large number of failed Events in one or more of your Pipelines. Replayed Events are fed back to the Pipelines and share the same resources that are used by the Pipeline itself. In this scenario, you can stop replaying the Events in the Pipeline from the Pipeline Overview page.

  Note: Pipelines across different accounts in Hevo do not affect each other.

• Connection limits for the Source imposed by Hevo. To ingest data, Hevo establishes multiple connections to your Source. The number of connections created and their usage depends on the Source type and the ingestion mode.

  • For log-based Pipelines, one connection is created for each object to ingest historical data. Once the historical ingestion is complete, a single connection is maintained for incremental ingestion.

  • For Pipelines using Table or Custom SQL ingestion mode, one connection is created per object for both historical and incremental ingestion.

  • For Pipelines created with SaaS Sources, separate connections are created per object for historical ingestion, incremental ingestion, and data refresh.

  When multiple Pipelines are created using the same Source, the total number of connections to that Source can increase significantly. To manage this load and maintain consistent performance, Hevo imposes a limit on concurrent connections. This limit varies depending on the Source type. For example, in the case of database Sources, Hevo allows up to 14 concurrent connections. Any ingestion tasks beyond this limit are queued until connections become available, which can delay data ingestion, especially during historical loads.

• Destination performance and limitations.

  • In cases of Destinations, such as MySQL and PostgreSQL, where data is not staged before loading, the Pipeline may experience delays. This happens because Hevo is not able to write data into the Destination as fast as it is ingesting it from the Source. If you think the slowness in the Destination is temporary, you may wait until it gets resolved. Otherwise, you should upgrade the hardware configuration of the Destination to allow it to accept a higher rate of writes.

  • Some Destinations, such as MySQL, are not ideal for handling large query traffic and concurrent transactions. For data warehouses, Hevo writes data at a default loading frequency optimized to reduce synchronization costs.

  • Certain Destinations, such as Amazon Redshift, Google BigQuery, Snowflake, and S3, rely on batch-based loading rather than individual record inserts. Batching improves efficiency and reduces costs by limiting frequent table scans and deduplication processes. However, this introduces a 5-15 minute delay before data appears in the Destination.

• Some Sources impose query execution limits or timeouts to prevent excessive resource consumption. If a query takes too long to execute, the Source may automatically cancel it, preventing Hevo from retrieving the data. Frequent query cancellations may disrupt data ingestion, leading to delays.

• A Pipeline consists of multiple tasks that handle data ingestion, transformation, and loading. If any of these tasks fail, it can cause a delay. A task may fail due to any of the following reasons:

  • Schema changes in the Source: If a table structure changes, such as a column is added, renamed, or removed, and Auto Mapping is disabled, the Pipeline may fail until the schema is updated in Hevo.

  • Authentication or connectivity issues: If Hevo loses access to the Source or Destination due to expired credentials, incorrect permissions, or network disruptions, it may fail to process data.

Note: If you are unable to determine the exact cause of the delay, contact Hevo Support for further assistance.

Revision History