Managing Objects in Pipelines

Objects represent the entity that Hevo ingests from your Source system. The entity can be a table, collection, report, or an Event Type, based on the type of the Source.

Once you complete the Source configuration while creating the Pipeline, the SELECT SOURCE OBJECTS page displays the objects in your Source. Only parent-level objects are displayed for selection. For example, let us say, there are the objects, Order and Order_items. Then, only the Order object is available for selection as Hevo understands Order_items to be a child of this object.

By default, all the objects are selected for ingestion. At this time, you can deselect the objects that you do not want to ingest data from; these are marked as SKIPPED in the Pipeline. You can also change your selection post-Pipeline creation. In the case of a Google Sheets Source, while each tab of a worksheet is listed and can be selected, it is ingested as a separate object.

In the case of RDBMS Source objects in Pipelines created with Log-based or Table ingestion modes, Hevo tries to find the optimal query mode for ingesting the data. If it is unable to do so, you are required to configure the query mode for that object yourself. In that case, you are directed to the Configure Objects page, in which Hevo displays the objects pending query mode identification. You can set up the query mode at this time and select the required column for it. If you skip this step, then the objects are queried in Full Load mode.

You can see the objects that you selected in the Objects section in the Pipeline Detailed View. The information displayed for each object is based on the ingestion mode and the Source configuration you specified while creating the Pipeline. For example, the following image shows the object list for a Source where the historical data ingestion option was selected for some objects and some objects were skipped.

NOTE: Objects are not listed for Sources which depend on webhooks to receive data.

The following information is typically provided for a Pipeline object:

• Tha name of the object

• The number of Events ingested

• A trend-line graph showing the activity in the last 24 hours

• Hover text on the trend-line graph to display the billable and historical Events count

• The object ingestion status, such as Active, Paused, or Skipped

• The time when the Events for this object were last ingested

• The More () icon that leads to the Actions menu for the object

Pipelines created with database Sources may display some additional details such as the log position till which data has been ingested and the historical load status, if the Pipeline is log-based and the Load Historical Data option is selected, respectively, as shown in the image below.

In the image above, OpLog information indicates the record position till which the incremental data has been ingested from a MongoDB Source. The historical data ingestion status is displayed for the Pipeline as a whole and for each object.

Click the More () icon to view the Actions menu that lists the actions available for an object. You can use these actions to control the data that you ingest for each object. Read SaaS Objects and Actions and Database Objects and Actions for more information.

---

Revision History