Optimizing Query Modes for Objects

You can change the query mode for one or more objects in a Pipeline created for an RDBMS Source in Table or Log-based mode, through the Pipeline Overview page. In case of log-based Pipelines, the query mode applies only to the historical load ingestion as the incremental data is fetched using logs. Read Factors Affecting Event Usage - Query Modes to understand how the query mode impacts the consumption of your Events quota.

The ingestion for the object restarts once you change its query mode. Please note that the entire data is ingested again from the beginning. The historical data is not billed during re-ingestion.

Read Query Modes for Ingesting Data for more information.

To change the query mode for a single object:

1. In the Pipeline List View, click on the Pipeline you want to modify, to open it in the Pipeline Overview page.

2. Select the Edit Config option from the Action menu of the object in the Pipeline Objects list.

   

3. Select the required Query Mode from the drop-down.

   

4. Depending on the query mode, specify the following parameters:

   Query Mode	Parameter
   Full Load	None
   Delta-Timestamp	Timestamp Column (name) and Timestamp Column Delay
   Change Data Capture	Incrementing Column (name), Timestamp Column (name), and Timestamp Column Delay
   Unique Incrementing Append Only	Unique Column (name)
   XMIN (available only for PostgreSQL Sources)	XMIN Column

   For example, for the Delta-Timestamp query mode, you must specify the parameters as shown below:

   

5. Click SAVE CHANGES.

To select objects in bulk and update the query mode:

1. In the Pipeline List View, click on the Pipeline you want to modify, to open it in the Pipeline Overview page.

2. Select each object whose query mode you want to edit, or select the Objects check box to edit the query mode for all the objects.

   

3. Click the MORE drop down and select Update Query Mode.

   

4. Update the query mode for each selected object and click SAVE CHANGES. You can also update only some of the objects at this time.

   

Note: These method works only for the Pipelines having a relational database (such as Amazon Aurora MySQL, Amazon Redshift, SQL Server, MySQL, or PostgreSQL as the Source.

Revision History