Loading Data in a Database Destination

Last updated on Nov 15, 2023

Hevo replicates the data from your Source system to the database Destination based on the primary keys defined in the Destination table. As part of loading the data, deduplication is done to ensure that only unique records are replicated and duplicates are dropped. This deduplication is done using the value of the primary key.

Deduplicating Data Loaded to the Database

The key steps involved in the deduplication process are:

  1. Load data into a temporary table.

  2. Adding metadata fields to the Destination schema.

  3. Apply the data from the temporary table to the Destination table in one of the following ways:

    • Update existing rows with ingested Events if the primary key already exists.

    • Append the ingested Events as new rows if primary key does not exist.

    Note: Deletion of an Event is handled as an update, by setting the value of __hevo__marked_deleted field to True.

  4. Delete the temporary table.

The Data Loading Process is illustrated below.

Note: The CPU time and storage space of the Destination is consumed for the duration of the data loading process.

Data Loading Process

The following diagram illustrates the process of loading data to a Database Destination:

Loading Data to a database

Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Apr-11-2022 NA Reorganized content.
Feb-22-2021 NA Updated the data loading process diagram to reflect the current process for handling updates and deletes.

Tell us what went wrong