Managing Objects in Pipelines

Last updated on Mar 10, 2025

Selecting and Deselecting Objects and Fields

During Pipeline creation, you can select and deselect the objects that you want Hevo to include or skip, respectively. For example, you may not want to load sensitive information such as your users’ personal details and financial records. Or, you may want to omit large text fields like descriptions, summaries, and comments. When you include an object, the historical data ingestion for it is started, followed by incremental data ingestion.

By default, Hevo ingests all fields for the included objects. However, you can change this selection. Unmapped fields are not loaded to the Destination tables.

To include an object for replication:

In the Object Configuration section of the Pipeline creation flow, select the check box corresponding to the object. For Merge load mode, if no primary key is defined, Hevo invites you to select one at this time.

All the fields of the object are selected by default.
Optionally, click the Edit () icon next to the object to modify its field selections.
On the Field Selection page, select or deselect the check box next to the fields that you want to include or exclude, respectively. You cannot deselect or change the primary key field if it is already defined.

To skip an object for replication:

Deselect the check box corresponding to the object on the Object Configurations tab. Hevo will no longer fetch data for the object. However, any existing data for the object in the Destination is not affected.

Defining the Primary Key for an Object

If a primary key is not already defined for an object, you can select the field(s) that Hevo must use as the primary key for deduplicating data in the Destination. This option is available and applicable only for the Merge load mode.

You can set the primary key only for the objects that do not have a pre-defined key. You cannot change a pre-defined primary key during or post-Pipeline creation.

To set a field as the primary key:

During Pipeline creation:
1. In the Object Configuration section, Keys column, click Define corresponding to the object for which you want to set the primary key.
2. In the Field Selection page, Primary Key column, select the check box corresponding to the field that you want to use as the primary key.
3. Click APPLY CHANGES.
Post-Pipeline creation:
1. Click EDIT in the Pipeline Detailed View.
2. On the Object Configuration page of the Edit Pipeline flow, in the Keys column, click Define corresponding to the object for which you want to set the primary key.
3. In the Field Selection page, Primary Key column, select the check box corresponding to the field that you want to use as the primary key.
4. Click APPLY CHANGES.

Change the load mode for an object

During Pipeline creation, you can select a load mode on the Pipeline Configuration page, which is applied to all objects by default. However, you can change the load mode for individual objects on the Object Configuration page.

To change the load mode for an object:

During Pipeline creation:
1. On the Object Configuration page, in the Actions column, click the Edit ( ) icon corresponding to the object for which you want to change the load mode.
2. At the top right corner of the Field Selection slide-in page, from the drop-down, select the load mode for your object.
  
  Note: For changing the load mode for an object from Append to Merge, ensure that the primary key fields are selected for the object.
3. Click APPLY CHANGES.
Post-Pipeline creation:
1. Click EDIT in the Pipeline Detailed View.
2. On the Pipeline Configuration page, click Next.
3. On the Object Configuration page, in the Actions column, click the Edit ( ) icon corresponding to the object for which you want to change the load mode.
4. You can do one of the following:
  - To change the load mode from Merge to Append, select Append from the drop-down in the top right corner of the Field Selection slide-in page.
  - To change the load mode from Append to Merge:
    1. Ensure that the Primary Key fields are selected for the object. If not, do the following:
      1. Select the check box next to the primary key fields, and then click Apply Changes.
      2. Click the Edit ( ) icon corresponding to the object for which you included the primary key fields.
    2. In the top right corner of the Field Selection slide-in page, from the drop-down, select Merge.
5. Click APPLY CHANGES.
Note: When you change the load mode of an object from Append to Merge, a banner appears at the bottom of the Summary section on the Review page. It notifies that the updated Destination tables will be dropped, recreated, and then loaded with the ingested data from the Source into the newly created table.

Resyncing an Object

You can resync an object if you want to replace the existing data for it in the Destination. This action is also useful when you want to resolve failed Events in an object and want to reload it to the Destination. With the Append load mode, this can result in duplicates if some of the Events had been loaded in a previous run.

On resyncing an object, Hevo drops the existing data from the Destination, and ingests and loads it again.

To resync an object:

Navigate to the Object Configuration tab of your Pipeline.
Click the More () icon next to the required object and click Resync Object.
In the confirmation dialog, click RESYNC.

Refreshing Schema

You can manually refresh the schema to update your Pipeline with changes from the Source schema before the next scheduled run. This allows newly added objects and fields to appear in the Object Configuration tab, so you can include them for ingestion and loading immediately if required. Read Pipeline Schema Management to know how Hevo handles any changes that occur in the Source data after the Pipeline is created.

To refresh the schema:

Click EDIT in the Pipeline Detailed View.
On the Object Configuration page of the Edit Pipeline flow, click the Refresh schema icon.
In the confirmation dialog, click REFRESH.

After a successful schema refresh, changes such as newly added, deleted, or renamed objects and fields will be reflected in the Object Configuration tab.