Optimizing Data Loading for a Destination Warehouse

While loading data to a Snowflake or Google BigQuery Destination, Hevo uses primary keys and metadata columns to deduplicate the data. Read Loading Data to a Data Warehouse. This deduplication process involves a full scan of the warehouse table every time data is loaded into it, which incurs a cost. In cases where the table is receiving only new data and no existing rows are being updated, this process leads to unnecessary Destination costs.

Hevo keeps track of the number of duplicates encountered for every table during deduplication over a period of time. If the number of duplicates found for a table is close to zero, Hevo displays a Destination Cost Advisor alert on your UI recommending Append-only mode for that table. With Append-Only mode, the rows are directly added to the table without any deduplication steps. Read Enabling Append-Only Mode for Suggested Tables section below.

Note: This feature is currently available for Snowflake and Google BigQuery only.

Enabling Append-Only Mode for Suggested Tables

You can enable the Append-only mode via the Destination Cost Advisor alert. Hevo displays this advisor in the Pipelines Detailed View and the Destinations Detailed View, to recommend ways for optimizing the costs.

Follow these steps to enable Append-only mode:

1. Click VIEW RECOMMENDATIONS in the Destination Cost Advisor tooltip. Hevo displays the list of tables where the number of duplicates found during deduplication is almost zero.

   

2. Select the tables you for which you want to enable Append-only mode, or click Select All to enable this for every table in the list.

3. Click ENABLE APPEND-ONLY MODE.

   

4. In the pop-up dialog that appears, click CONFIRM.

   

Note: You can disable Append-only mode for a table at any time. Read Deduplicating Data in a Data Warehouse Destination.

See Also

• Data Loading

Revision History