On-Demand Syncing Data in Edge Pipelines

Hevo Edge Pipelines sync data ingested from your Source with the Destination as per the Pipeline sync schedule, which runs according to the configured frequency. However, there could be scenarios such as the following, when you may want the Pipeline to sync data outside of the schedule:

• You created the Pipeline and need the data in your Destination at the earliest.

• You changed the Pipeline configuration to resolve any job failures and want the Pipeline to start replicating data, but the configured frequency is 24 hours or more.

• The Source or Destination was not available for reasons such as an outage or planned maintenance, causing the jobs to fail.

• You noticed a data mismatch between the Source and the Destination and want to replicate the data immediately to identify the data records and for troubleshooting.

• You included new objects in the Pipeline and need to load their data into the Destination.

To handle such scenarios, Hevo allows you to sync or resync data for your Edge Pipeline on-demand.

---

Sync Now

The Sync Now action starts a job to sync incremental data from your Source with the Destination. To start this job for your Pipeline, do the following:

1. In the Pipelines List View, click the Edge tab and then click the Pipeline whose data you want to sync.

   

2. In the Pipeline Summary bar, click the Kebab menu icon ( ) and click Sync Now.

   

3. In the pop-up dialog, click SYNC NOW.

   

After you click SYNC NOW, Hevo performs the following actions on your Edge Pipeline:

• Check if any job is in progress and prompt to cancel the active job. You can then choose to cancel the active job and start a new sync.

• Start the incremental data sync from the last successful offset for all the objects included in the Pipeline.

• Start the historical data sync for newly included objects in the Pipeline.

---

Resync Pipeline

The Resync Pipeline action allows you to restart the historical load for all the active objects in your Pipeline. Resyncing a Pipeline drops the existing data in the Destination and replaces it with the data ingested from the Source. It is particularly useful when you want to resolve a data mismatch between the Source and Destination data or when the incremental data sync cannot be started due to a probable database log expiry.

Note: You can only resync a Pipeline that is ENABLED.

Perform the following steps to resync your Edge Pipeline:

1. In the Pipelines List View, click the Edge tab and then click the Pipeline that you want to resync.

   

2. In the Pipeline Summary bar, click the Kebab menu icon ( ) and click Resync Pipeline.

   

3. In the pop-up dialog, click RESYNC PIPELINE.

   

After you click RESYNC PIPELINE, Hevo performs the following actions on your Edge Pipeline:

• Move the Pipeline to the RESYNCING state.

• Cancel any actively running jobs in the Pipeline.

• Drop and re-create the resources in your Source system that Hevo uses while ingesting data, such as the replication slot in your PostgreSQL database.

• Refresh the Source schema.

• Start the historical data sync for all objects included in the Pipeline. After the historical data sync is successful, the incremental data sync is run on schedule as per the configured frequency.

• Move the Pipeline to the ENABLED state.

Note: If any of the tasks fail, the Pipeline moves to the FAILED state. You can try enabling the Pipeline later, and if the issue persists, contact Hevo Support.