Pipeline Objects

Objects represent the entity that Hevo ingests from your Source system. The entity can be a table, collection, report, or Event Type, based on the type of the Source. You can select or deselect the objects to ingest data from while creating the Pipeline and after. Both parent and child objects are displayed and the objects received from webhook Events are also present in the Object list.

The SELECT SOURCE OBJECTS page displays the objects in your Source once you complete the Source configuration settings.

All the objects are selected by default. You can deselect the objects you do not want to ingest data for; these are marked as Skipped in the Pipeline.

Post object selection, you are directed to the Configure Objects page in case any of the objects require the data query mode to be configured. In such cases, Hevo recommends the optimal query mode for each object, and provides a list of columns to choose from when ingesting data using that query mode. However, you can also select a different query mode for an object, and then specify the column required for that query mode.

You can also provide the configuration later.

For example, Hevo recommends the Unique Incrementing Append Only query mode for the employees.departments object and allows you to select either the dept_no or the dept_name column in order to ingest data using the query mode.

An object might get its data in more than one way. For example, a log-based Pipeline can ingest historical data as well as the data from the log file. Read Log-based Pipelines.

---

Post-Pipeline Query Mode Optimization

Hevo allows you to optimize query mode for all the objects being ingested in your Pipeline after you have created your Pipeline successfully.

Note: This feature is currently available for relational database Sources, such as MS SQL, MySQL, and PostgreSQL Sources only.

If the objects ingested in Full Load are higher than 10% of the billable events, an OPTIMIZE QUERY MODE button is provided in the Smart Assist widget at the top of the Pipeline Overview page to configure the query mode of the objects in your Pipeline, to avoid higher ingestion and load costs. In the image below, we are selecting Generic PostgresSQL as a Source database as an example.

Read Query modes to understand how Hevo fetches your data.

Note: You can click the Close () to remove the Smart Assist widget from the Pipeline Overview page. Once the banner is closed, it will not be displayed for the next 30 days. However, you can change the query mode at any point in time.

Perform the following steps to optimize the query modes for the objects in your Pipelines:

1. Do one of the following:

   • Use the Smart Assist widget:

     In the Smart Assist widget, click OPTIMIZE QUERY MODE, to configure the query mode for the objects in your database.

     

   • Use the Pipeline Objects section of the Pipeline Overview page:

     1. Select the check boxes for the required objects individually or select the check box above the objects list to select all the objects.

        

     2. Click More, and then click UPDATE QUERY MODE from the drop-down.

        

2. In the Configure Objects window, click UPDATE TO XMIN to change the query mode for the selected objects to XMIN or select the check boxes corresponding to the required objects, and click Query Mode to configure the query mode individually.

   Note: UPDATE TO XMIN and UPDATE TO CHANGE TRACKING options are specific to PostgreSQL and MS SQL variants, respectively.

   

3. Click SAVE CHANGES to apply the changes.

---

Revision History