How can I make sure that each record is loaded only once?

Last updated on Jan 31, 2024

To ensure that each record is loaded only once, you can take some actions prior to and post-Pipeline creation:

  • Pre-Pipeline creation:
    • Create the Pipeline with Auto Mapping disabled. Manually map the Event Type (Source object) and define one of the fields as the primary key. This field must have unique and non-null values. Finally, enable Auto Mapping for the Event Type.

    • Disable the Append Rows on Update option for the table in the Destination Overview page.

      Disable Append Rows on Update

      Read How do I enable or disable the deduplication of records in my Destination tables?

      Note: This feature is available only for Amazon Redshift, Google BigQuery, and Snowflake Destinations.

Post-Pipeline creation

  1. Disable Auto Mapping for the Event Type from the Schema Mapper.

    Disable Auto Mapping

  2. Click the Kebab menu icon next to the Destination Table name and click Drop Table.

    Drop present Destination table

  3. Click CREATE TABLE & MAP to create a new Destination table.

    Create a new Destination table

  4. Set a field as the primary key by selecting the Primary Key check box for it.

    Select Primary key fields

  5. Specify the Destination Table Name and click CREATE TABLE & MAP.

    The data is replicated to the new Destination table as per the Pipeline schedule, using the defined primary keys to ensure no duplicate Events are created.


Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Nov-07-2022 NA Created as a new document.

Tell us what went wrong