Data Ingestion Statuses

Hevo may assign the following data ingestion statuses to the objects in the Pipeline to inform you of the progress and any action that you may need to take:

• INIT: The ingestion activity has just been created.

• SCHEDULED: The ingestion has been scheduled and will run as per the next run cycle which is based on the configured schedule.

• QUEUED: The ingestion is ready to be started and will begin streaming as soon as the resources become available to fetch data from the Source.

• STREAMING: The Events are being ingested actively for the objects.

• BOOTSTRAPPING: This is the first ingestion run for the Pipeline and it is actively fetching historical data from the Source.

• PAUSED: The ingestion has been paused and will not fetch data until resumed.

• SKIPPED: The ingestion has been skipped and Events will not be streamed until the object is included again in the Pipeline.

• AWAITING MAPPING: The ingestion for this object has been skipped by Hevo because it is not mapped to a Destination table in the Schema Mapper.

• NOT INCLUDED: Events are ingested for this object till it is configured.

• HISTORICAL LOAD FINISHED: Ingestion of historical data is complete. To ingest the historical data again, restart the historical load for the object.

• DEFERRED: The ingestion has been deferred for some time due to a constraint enforced by the Source. This usually happens when we hit API limits while calling an API provided by the Source.

• FINISHED: The ingestion has reached an end state and will not run ever again.

Ingestion Statuses for File-Based Sources

For File-based Sources, GCS, S3, FTP, and Drive, ingestion of each file within a folder or object is individually tracked. When Hevo first reads a file, it is assigned the status, New. Subsequently, it is downloaded and ingested by the Pipeline. Once all the files in the folder have been read, the process is repeated. Any new and modified files that are found are assigned the status, New and processed for ingestion.

The statuses, in order of progression, include:

• NEW: The file is new or modified and will be ingested.

• DOWNLOADING: The file is being downloaded from the file server/storage.

• DOWNLOADED: The file has been downloaded and will be queued for ingestion.

• INGESTING: The File is being ingested actively.

• INGESTED: The ingestion is complete.

• FAILED: There was a failure while processing the file and Hevo will try downloading or ingesting it again, as applicable.

• SKIPPED: There was a failure while processing the file, and processing will not be attempted again. The offset is moved forward and the next file is picked for ingestion. The offset is the combination of the file name and the time it was last modified. Currently, a file may be skipped for two reasons:

  • It is an Excel file larger than 50 MB.

  • It is larger than 5 GB.

You can click on any folder or object in the Pipeline overview page to view the files contained within it and their ingestion statuses. See Pipelines Detailed View.

Revision History