How do I enable or disable the deduplication of records in my Destination tables?

Use the Append Rows on Update option within a Destination table to indicate whether the ingested Events must be directly appended as new rows, or must these be checked for duplicates. You can specify this setting for each table.

Note: This feature is available only for Amazon Redshift, Google BigQuery, and Snowflake data warehouse Destinations. For RDBMS Destinations such as Aurora MySQL, MySQL, Postgres, and SQL Server, deduplication is always done.

In the Destination Detailed View page:

1. Click the () icon next to the table name in the Destination Tables List.

2. Update the Append rows on update option, as required:

   Option setting	Description
   Enabled	Events are appended as new rows without deduplication
   Disabled	Events are de-duplicated

   

3. Click OK, GOT IT in the confirmation dialog to apply this setting.

Note: If you disable this feature after having previously enabled it, uniqueness is ensured only for future records in case of Google BigQuery and Snowflake. Therefore, both old and new versions of the same record may exist. In case of Amazon Redshift, however, uniqueness can be achieved for the entire data upon disabling the feature.

Revision History