Managing Objects in Pipelines

The list of objects available in your Source database is displayed in the Object Configuration tab of your Pipeline. The objects that you include for replication are shown as selected. The objects that Hevo is not able to access are marked as Inaccessible.

You can use the Object Configuration tab or the edit Object Configuration page to do the following:

• Select and deselect objects and fields for replication

• Set the primary key for an object during Pipeline creation

• Resync an object post-Pipeline creation

• Refresh schema post-Pipeline creation

Refer to the respective sections for the steps to manage the objects as per your requirements.

---

Selecting and Deselecting Objects and Fields

During Pipeline creation, you can select and deselect the objects that you want Hevo to include or skip, respectively. For example, you may not want to load sensitive information such as your users’ personal details and financial records. Or, you may want to omit large text fields like descriptions, summaries, and comments. When you include an object, the historical data ingestion for it is started, followed by incremental data ingestion.

By default, Hevo ingests all fields for the included objects. However, you can change this selection. Unmapped fields are not loaded to the Destination tables.

To include an object for replication:

1. In the Object Configuration section of the Pipeline creation flow, select the check box corresponding to the object. For Merge load mode, if no primary key is defined, Hevo invites you to select one at this time.

   

   All the fields of the object are selected by default.

2. Optionally, click the Edit () icon next to the object to modify its field selections.

   

3. On the Field Selection page, select or deselect the check box next to the fields that you want to include or exclude, respectively. You cannot deselect or change the primary key field if it is already defined.

   

To skip an object for replication:

• Deselect the check box corresponding to the object on the Object Configurations tab. Hevo will no longer fetch data for the object. However, any existing data for the object in the Destination is not affected.

---

Defining the Primary Key for an Object

If a primary key is not already defined for an object, you can select the field(s) that Hevo must use as the primary key for deduplicating data in the Destination. This option is available and applicable only for the Merge load mode.

You can set the primary key only for the objects that do not have a pre-defined key. You cannot change a pre-defined primary key during or post-Pipeline creation.

To set a field as the primary key:

• During Pipeline creation:

  1. In the Object Configuration section, Keys column, click Define corresponding to the object for which you want to set the primary key.

     

  2. In the Field Selection page, Primary Key column, select the check box corresponding to the field that you want to use as the primary key.

     

  3. Click APPLY CHANGES.

• Post-Pipeline creation:

  1. Click EDIT in the Pipeline Detailed View.

     

  2. On the Object Configuration page of the Edit Pipeline flow, in the Keys column, click Define corresponding to the object for which you want to set the primary key.

     

  3. In the Field Selection page, Primary Key column, select the check box corresponding to the field that you want to use as the primary key.

     

  4. Click APPLY CHANGES.

---

Resyncing an Object

You can resync an object if you want to replace the existing data for it in the Destination. This action is also useful when you want to resolve failed Events in an object and want to reload it to the Destination. With the Append load mode, this can result in duplicates if some of the Events had been loaded in a previous run.

On resyncing an object, Hevo drops the existing data from the Destination, and ingests and loads it again.

To resync an object:

1. Navigate to the Object Configuration tab of your Pipeline.

   

2. Click the More () icon next to the required object and click Resync Object.

   

3. In the confirmation dialog, click RESYNC.

   

---

Refreshing Schema

You can manually refresh the schema to update your Pipeline with changes from the Source schema before the next scheduled run. This allows newly added objects and fields to appear in the Object Configuration tab, so you can include them for ingestion and loading immediately if required. Read Pipeline Schema Management to know how Hevo handles any changes that occur in the Source data after the Pipeline is created.

To refresh the schema:

1. Click EDIT in the Pipeline Detailed View.

   

2. On the Object Configuration page of the Edit Pipeline flow, click the Refresh schema icon.

   

3. In the confirmation dialog, click REFRESH.

   

After a successful schema refresh, changes such as newly added, deleted, or renamed objects and fields will be reflected in the Object Configuration tab.