Upgrading Pipeline from Standard to Edge

A Standard Pipeline can be upgraded to an Edge Pipeline. Hevo assesses Pipelines and contacts customers to start the upgrade. The entire process is managed by Hevo and requires no action on your part.

Edge Pipelines run on Hevo’s latest architecture, providing:

• Higher reliability and better handling of schema changes

• Faster and more efficient data replication

• Improved monitoring and visibility across the data flow

The diagram below outlines the upgrade workflow. Each stage is described in the following sections.

---

Pre-Upgrade Validation

Hevo requests temporary access to your Source and Destination systems to validate your Standard Pipeline for upgrade. To begin the validation:

• A snapshot of Destination tables is created as a reference for validation.

• A temporary Edge Pipeline with the same configuration as your Standard Pipeline is created and runs in parallel.

  • The schema evolution policy for the Edge Pipeline depends on your Standard Pipeline configuration:

    • If Auto Mapping is disabled, the policy is Block all changes.

    • If Auto Mapping is enabled and:

      • Include New Tables in the Pipeline option is enabled, the policy is Allow all changes.

      • Include New Tables in the Pipeline option is disabled, the policy is Allow column-level changes only.

  • The Edge Pipeline’s sync frequency is determined by your Standard Pipeline’s ingestion and loading frequency. Between these two, Hevo sets the sync frequency to whichever is higher. For instance, if your Standard Pipeline ingests data every 15 minutes but loads every 30 minutes, the Edge Pipeline syncs data every 30 minutes.

During the pre-upgrade validation process:

• The Edge Pipeline ingests data from your Source using the last saved offset of the Standard Pipeline and loads it into the snapshot of the corresponding Destination tables. Meanwhile, your Standard Pipeline continues ingesting and loading data into the live Destination tables, ensuring that your production data flow remains uninterrupted.

• Hevo runs the Edge Pipeline for at least six hours, observing two to three ingestion and loading cycles.

  • If the Standard Pipeline loads more frequently than every six hours, data comparison between the snapshot and the live Destination can begin soon after the first six hours.

  • If the Standard Pipeline loads less frequently, for example, every 12 hours, Hevo extends the observation to 24–36 hours.

After observation, Hevo disables the Edge Pipeline and compares data between the snapshot and the live Destination.

• If data matches, the snapshot is deleted, and the process moves to the upgrade process.

• If mismatches occur, the upgrade is paused, the Edge Pipeline and snapshot are deleted, and the process shifts to the issue-fixing phase.

Fixing issues to validate your Standard Pipeline for Upgrade

If any issues are detected during pre-upgrade validation, Hevo identifies the root cause and updates the Edge Pipeline so it meets the upgrade requirements.

For instance, if your Standard Pipeline uses a datetime format not supported in Edge Pipelines, Hevo adjusts the Edge Pipeline configuration to handle the format correctly. Once the issues are resolved, the Pre-Upgrade Validation step is repeated to confirm that the Standard Pipeline is ready for upgrade.

---

Upgrading the Pipeline from Standard to Edge

Once validation is successful, Hevo initiates the upgrade. The Edge Pipeline created for validation is deleted, and a new Edge Pipeline is created with the same configuration as your Standard Pipeline. The schema evolution policy and the sync frequency of your Edge Pipeline remain the same as during validation.

At the start of the upgrade, Hevo creates a snapshot of your Destination tables to capture their exact state. This snapshot can be used later to restore data if a rollback to the Standard Pipeline is required.

Note: The upgrade setup can take 15 to 60 minutes. If your Standard Pipeline’s loading frequency is less than 60 minutes, there may be a short delay before the Edge Pipeline begins loading data into the Destination tables.

---

Monitoring the Edge Pipeline for 30 Days After Upgrade

After the upgrade, Hevo manages data replication as follows:

• All data previously replicated by the Standard Pipeline remains in the Destination tables.

• The Standard Pipeline continues to ingest data, but stops loading it to the Destination. Ingested data is securely staged within Hevo, ensuring recovery without data loss if any issue arises in the Edge Pipeline.

• The Edge Pipeline begins ingesting data from your Source using the last saved offset of the Standard Pipeline and loads the data into the Destination tables. It processes both historical and incremental jobs, applying Edge naming conventions to any new fields or objects included for replication.

• Both Pipelines run in parallel for up to 30 days, giving you time to verify data replication and monitor Pipeline stability.

After the Standard Pipeline is upgraded, you need to monitor the Edge Pipeline for 30 days to confirm that the schema and Events are replicated accurately to your Destination tables. If you notice any issues, such as schema mismatches, data inconsistencies, or ingestion failures, contact Hevo Support. Once the monitoring period ends without issues, the upgrade proceeds to the Cleaning Up Upgrade Resources step, and you can continue using the Edge Pipeline for data replication.

Fixing issues in the Edge Pipeline after Upgrade

If a data mismatch or Pipeline failure occurs after the upgrade, Hevo identifies the root cause and resolves the issue to ensure accurate data replication in your Edge Pipeline. Once resolved, the Edge Pipeline resumes data replication.

If the issue persists, Hevo rolls back to the Standard Pipeline to maintain data consistency.

Rolling Back to the Standard Pipeline

If issues are detected in your Edge Pipeline during the 30-day monitoring period that cannot be resolved immediately, Hevo initiates a rollback to the Standard Pipeline. This ensures data consistency and uninterrupted data replication.

During rollback, Hevo disables the Edge Pipeline and uses the snapshot of your Destination tables, along with the data staged internally during the monitoring period, to restore the Standard Pipeline. The rollback process ensures that no data is lost and maintains data consistency, allowing your Standard Pipeline to continue replicating data as before.

Note: As the Edge Pipeline handled loading during the upgrade, the Standard Pipeline may take some time to synchronize after rollback.

After the rollback is complete, Hevo deletes the snapshot, and the upgrade process moves back to the issue-fixing phase. Once the issues are resolved, the upgrade restarts from the Pre-Upgrade Validation step.

---

Cleaning Up Upgrade Resources

After 30 days of stable data replication, the upgrade is considered successful. At this stage, Hevo performs the following actions:

• Deletes the snapshot and staged data.

• Pauses the Standard Pipeline.

• Cleans up any resources created for the upgrade.

Your Edge Pipeline is now fully active and handles all ingestion and loading tasks.

Read Working with Edge Pipelines to learn more about how your Edge Pipeline functions.