Pausing a Log-based Pipeline

Each log has a retention period associated with it which specifies the duration a log is maintained before it is automatically deleted. Logs are purged periodically to prevent them from taking up too much disk space.

Consequences of Pausing a Log-based Pipeline

Pausing a Pipeline pauses the process of ingestion. Pausing a log-based Pipeline for more than 24 hours may lead to data loss as a result of the log being deleted. The log can get deleted due to the expiry of its retention period or limited disk storage space in the case of large log files.

In case there is a loss of data after resuming a paused Pipeline, restart the historical load for all the objects to ingest the lost data. To do so, in the Pipeline Overview page:

  1. Select the Objects check box to select all the objects in the Pipeline. You can also select specific objects by selecting the check box next to their names.

    Restart historical load for the Pipeline

  2. Select the Restart option from the MORE drop-down to start the historical data ingestion.

The historical load starts immediately. The re-ingested data does not count towards your quota consumption and is not billed.

Note: You can similarly recover lost data by restarting individual jobs configured with change tracking mode in the case of an SQL Server Source.



See Also


Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Jan-24-2022 1.80 Added information about ingesting data when the Pipeline is paused for more than 24 hours.
Last updated on 22 Mar 2022