On-Demand Syncing Data in Edge Pipelines

Hevo Edge Pipelines sync data ingested from your Source with the Destination as per the Pipeline sync schedule, which runs according to the configured frequency. However, there could be scenarios such as the following, when you may want the Pipeline to sync data outside of the schedule:

• You created the Pipeline and need the data in your Destination at the earliest.

• You changed the Pipeline configuration to resolve any job failures and want the Pipeline to start replicating data, but the configured frequency is 24 hours or more.

• The Source or Destination was not available for reasons such as an outage or planned maintenance, causing the jobs to fail.

• You noticed a data mismatch between the Source and the Destination and want to replicate the data immediately to identify the data records and for troubleshooting.

• You included new objects in the Pipeline and need to load their data into the Destination.

To handle such scenarios, Hevo allows you to sync or resync data for your Edge Pipeline on-demand.

---

Sync Now

The Sync Now action starts a job to sync incremental data from your Source with the Destination. To start this job for your Pipeline, do the following:

1. In the Pipelines List View, click the Edge tab and then click the Pipeline whose data you want to sync.

   

2. In the Pipeline Summary bar, click the Kebab menu icon ( ) and click Sync Now.

   

3. In the pop-up dialog, click SYNC NOW.

   

After you click SYNC NOW, Hevo performs the following actions on your Edge Pipeline:

• Check if any job is in progress and prompt to cancel the active job. You can then choose to cancel the active job and start a new sync.

• Start the incremental data sync from the last successful offset for all the objects included in the Pipeline.

• Start the historical data sync for newly included objects in the Pipeline.

---

Resync Pipeline

The Resync Pipeline action allows you to restart the historical load for all the active objects in your Pipeline. Resyncing a Pipeline re-ingests historical data from the Source objects. However, the data existing in the corresponding Destination tables is only updated if changes are detected in the re-ingested data. Hevo also evolves the schemas of the Destination tables if any changes are detected in the mapping. Now, if any of the objects are blocked due to incompatible schema evolution, you can choose to drop the Destination tables during the resync operation for all the active objects in your Pipeline. In this case, Hevo drops and re-creates the Destination tables for all the active objects. Data from the Source objects is re-ingested and loaded into the newly created Destination tables.

Note: You can only resync a Pipeline that is ENABLED.

Perform the following steps to resync your Edge Pipeline:

1. In the Pipelines List View, click the Edge tab and then click the Pipeline that you want to resync.

   

2. In the Pipeline Summary bar, click the Kebab menu icon () and click Resync Pipeline.

   

3. In the Resync Pipeline pop-up window, do the following:

   

   • (Optional) Select the Enable Drop and Load check box if you want Hevo to re-create the Destination tables for all the active objects in your Pipeline during the resync operation.

   • Click RESYNC PIPELINE to confirm the action.

After you click RESYNC PIPELINE, Hevo performs the following actions on your Edge Pipeline:

• Move the Pipeline to the RESYNCING state.

• Cancel any actively running jobs in the Pipeline.

• Drop and re-create the resources in your Source system that Hevo uses while ingesting data, such as the replication slot in your PostgreSQL database.

• Refresh the Source and Destination schemas.

• Compare the refreshed schemas and evolve the schema of the Destination tables based on the detected changes.

• Optionally, drop and re-create the Destination tables if the drop-and-load option is selected.

• Start the historical data sync for all objects included in the Pipeline. After the historical data sync is successful, the incremental data sync is run on schedule as per the configured frequency.

• Move the Pipeline to the ENABLED state.

Note: If any of the tasks fail, the Pipeline moves to the FAILED state. You can try enabling the Pipeline later, and if the issue persists, contact Hevo Support.