How do I enable or disable the deduplication of records in my Destination tables?

Last updated on Nov 23, 2023

Use the Append Rows on Update option within a Destination table to indicate whether the ingested Events must be directly appended as new rows, or must these be checked for duplicates. You can specify this setting for each table.

Note: This feature is available only for Amazon Redshift, Google BigQuery, and Snowflake data warehouse Destinations. For RDBMS Destinations such as Aurora MySQL, MySQL, Postgres, and SQL Server, deduplication is always done.

In the Destination Detailed View page:

  1. Click the (More) icon next to the table name in the Destination Tables List.
  2. Update the Append rows on update option, as required:

    Option setting Description
    Enabled Events are appended as new rows without deduplication
    Disabled Events are de-duplicated

    Modify Append rows on update setting for a table

  3. Click OK, GOT IT in the confirmation dialog to apply this setting.

Note: If you disable this feature after having previously enabled it, uniqueness is ensured only for future records in case of Google BigQuery and Snowflake. Therefore, both old and new versions of the same record may exist. In case of Amazon Redshift, however, uniqueness can be achieved for the entire data upon disabling the feature.

Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Nov-07-2022 NA Created as a new document.

Tell us what went wrong