Optimizing Query Modes for Objects

Last updated on Feb 20, 2023

You can change the query mode for one or more objects in a Pipeline created for an RDBMS Source in Table or Log-based mode, through the Pipeline Overview page. In case of log-based Pipelines, the query mode applies only to the historical load ingestion as the incremental data is fetched using logs. Read Factors Affecting Event Usage - Query Modes to understand how the query mode impacts the consumption of your Events quota.

The ingestion for the object restarts once you change its query mode. Please note that the entire data is ingested again from the beginning. The historical data is not billed during re-ingestion.

Read Query Modes for Ingesting Data for more information.

To change the query mode for a single object:

  1. In the Pipeline List View, click on the Pipeline you want to modify, to open it in the Pipeline Overview page.

  2. Select the Edit Config option from the Action menu of the object in the Pipeline Objects list.

    Edit config option

  3. Select the required Query Mode from the drop-down.

    Select query mode for an object

  4. Depending on the query mode, specify the following parameters:

    Query Mode Parameter
    Full Load None
    Delta-Timestamp Timestamp Column (name) and Timestamp Column Delay
    Change Data Capture Incrementing Column (name), Timestamp Column (name), and Timestamp Column Delay
    Unique Incrementing Append Only Unique Column (name)
    XMIN (available only for PostgreSQL Sources) XMIN Column

    For example, for the Delta-Timestamp query mode, you must specify the parameters as shown below:

    Settings for Delta Timestamp query mode

  5. Click SAVE CHANGES.

To select objects in bulk and update the query mode:

  1. In the Pipeline List View, click on the Pipeline you want to modify, to open it in the Pipeline Overview page.

  2. Select each object whose query mode you want to edit, or select the Objects check box to edit the query mode for all the objects.

    Select multiple objects

  3. Click the MORE drop down and select Update Query Mode.

    update query mode option for multiple objects

  4. Update the query mode for each selected object and click SAVE CHANGES. You can also update only some of the objects at this time.

    save bulk changes

Note: These method works only for the Pipelines having a relational database (such as Amazon Aurora MySQL, Amazon Redshift, SQL Server, MySQL, or PostgreSQL as the Source.


Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Feb-10-2023 NA Created as a new document from the FAQ, How do I change the query mode for my Pipeline? Added details of changing the query mode for multiple objects at once.

Tell us what went wrong