Managing Objects in Pipelines

You can see the objects you select to ingest data for in the Pipeline Detailed View. The information displayed is based on the Pipeline mode and the Source configuration you select. You can control the data ingestion for individual objects using the options provided for it.

NOTE: Objects are not listed for Sources which depend on webhooks to receive data.

The following information is typically provided for each object:

• Name of the object

• Number of Events ingested

• A trend-line graph showing the activity in the last 24 hours

• Hover text on the graph to display billable and historical Events count

• Object ingestion state

• Time when the Events were last ingested

Bulk Actions

Select multiple objects in the Pipeline Overview page to apply a bulk action to them. Using bulk actions, you can:

• PAUSE and RESUME objects.

• INCLUDE and SKIP objects.

• More button:

  • Run Now: Initiate ingestion for the selected objects.

  • Restart: Reingest the objects.

  • Replay Failed Events: Ingest the failed Events.

  • Skip Failed Events: Skip the failed Events.

  • Update Query Mode: Modify the query mode of objects in the Pipeline.

• Pipeline Search Bar: Search objects in the Pipeline.

• Filter: Filter the objects in the Pipeline.

  • Click Filter () to see the status of each Pipeline object.

    • ALL: View all the objects in your Pipeline based on the query mode selected. For example, click Full Load to view all the objects in Full Load.

    • Active: View all the objects which are currently ingesting data into the Destination.

    • Pause: View all the currently paused objects or not ingesting data into the Destination.

    • Failed: View all the objects facing some issue to ingest the data from Source Database.

      

---

SaaS Source Objects

Data for SaaS Sources is ingested based on the respective APIs exposed by the Source.

Available actions

The following standard actions are available for each object:

• Change Position: Ingest data from a different offset than the value determined by Hevo. You can shift the offset within the limits set by the Source. For example, Facebook Ads allows fetching data up to the past three years. This option is provided for all SaaS-based Pipelines. Changing the position also initiates the data ingestion activity.

• Run Now: Queues the jobs related to the Source object. This action works on an active object and Pipeline.

• Sample Events: View a sample of the ingested Events.

• Pause or Resume: Pause or resume the data ingestion for the object.

• Restart Object: Restart the incremental data ingestion for the object. The data is ingested again from the beginning. Some Sources may restrict how far back Hevo can go to re-ingest the data. Hevo stores the position of each ingestion run at its end. On every consecutive run, it looks for data beyond the previously stored position.

  NOTE: Not all Sources support the Restart action.

• Skip or Include Object: Skip and Include are complementary actions. You can skip and include active or paused Source objects.

• Pause or Resume Historical Load: Pause ingestion of historical Events, where applicable. You can resume the ingestion later from the last position.

• Restart Historical Load: Ingest historical data again from the start. Hevo calculates the Event from where ingestion must start in the first run of the Pipeline itself, based on the historical sync duration, if applicable. Subsequently, it is a moving window based on when you restart the load.

• Skip or Include Historical Load: Skip ingestion of the historical Events and do it at a later point in time, if historical loads are applicable.

---

RDBMS Source Objects

Pipelines for RDBMS Sources display similar information as SaaS Sources, along with the log-based ingestion and historical load status, if Pipeline is log-based and the Load Historical Data option is selected, respectively, as shown in the image below.

In the image above, BinLog information indicates the record position till which the data has been ingested. This information is also displayed and controllable at the object-level. Click the More icon and then, Change Position to start ingesting from a different position. In log-based Pipelines, the position-change option works on historical load if it exists.

---

Available actions

The following standard actions are available for each object:

• Change Position: Ingest data from a different offset than the value determined by Hevo. This option is provided for all Pipelines with mode as Table. All log-based Pipelines, except for WAL mode in PostgreSQL and Streams in Amazon DynamoDB, too support change of position. Changing the position also initiates the data ingestion activity.

• Sample Events: View a sample of the ingested Events.

• Skip or Include Object: Skip and Include are complementary actions.

  • You can skip and include active or paused Source objects.

  • For log-based Pipelines, skipping or including the Source object means dropping or including the related Events from the log-based ingestion.

  Read Including Skipped Objects Post-Pipeline Creation to know how you can include and ingest the skipped objects (if any).

• Pause or Resume Historical Load: Pause ingestion of historical Events. You can resume the ingestion later from the last position. This option is available in Pipelines created from Hevo Release 1.67 onwards.

• Restart Historical Load: Ingest data since the beginning (the beginning is defined per Source type).

  • Log-based ingestion cannot be restarted. For such Pipelines, the historical load has to be restarted.

  • This action works on Active state of the object and Pipeline.

• Skip or Include Historical Load: Skip ingestion of the historical Events and do it at a later point in time. This option is available in Pipelines created from Hevo Release 1.67 onwards.

• Edit Config: Modify the query mode for individual objects ingesting in the Pipeline.

---

Revision History