How can I migrate a Pipeline created with one Database Source variant to another?

Hevo provides dedicated connectors for each variant of your database Source. If you migrate your database to a different variant, you must create a new Pipeline using the connector for the new variant.

Perform the following steps to migrate your Pipeline:

1. Prepare for Migration

2. Set up a new Pipeline

Prepare for Migration

To prevent data loss during migration, perform the following steps:

1. Verify that all data from your existing database Source was ingested by the Pipeline.

   • For log-based Pipelines, check the latest position in the Pipeline Activity section of the Pipeline Detailed View. This shows the offset up to which data was successfully ingested.

     

   • For Pipelines created with Table or Custom SQL ingestion modes, check the ingestion position for all the active objects listed in the Objects section of the Pipeline Detailed View.

     

2. Ensure that no write operations, such as inserts, updates, or deletes, are performed on the Source database until the migration is complete and you have created a Pipeline with the new database variant.

3. Ensure that all ingested data is loaded into the Destination.

   • For data warehouse Destinations, manually trigger the loading of Events.

   • For other Destinations, wait for 30 minutes to 1 hour for the data to be loaded into the Destination.

   • In case there are any failed Events, ensure that they are successfully retried and loaded into the Destination. Read Troubleshooting Failed Events in a Pipeline.

4. Pause the Pipeline created with the existing Source database.

5. Migrate your database to the new variant.

6. Ensure that no write operations are performed on the new database until the new Pipeline is created with it.

Set up a new Pipeline

Perform the following steps to create a Pipeline that continues ingesting data into the same Destination tables:

1. Refer to the Source documentation and complete the prerequisites to configure the new Source type in your Hevo Pipeline.

2. Create a Pipeline using the configuration details of your new database. As the Destination already contains the historical data for the Source objects, you can do one of the following:

   • Migrate your Pipeline with historical data replication. To do this, during Source configuration, ensure that the Load Historical Data option is enabled under the Advanced Settings section.

     

   • Migrate your Pipeline without historical data replication. To do this, disable the Load Historical Data option during Source configuration. This ensures that the Pipeline ingests only incremental data generated after the Pipeline is created.

3. Use the same Destination setup as the paused Pipeline, including the table prefix. This ensures that data continues to load into the same Destination tables.

   Note:

   • If the Source tables were mapped manually in the old Pipeline, apply those mappings in the new Pipeline to continue loading data into the same Destination tables.

   • If any Transformations were applied in the old Pipeline, copy them to the new Pipeline to retain the data processing logic.

4. Once the Pipeline is created, resume write operations on the new database.

5. When the new Pipeline starts ingesting data, delete the old Pipeline.

Revision History