Editing Edge Pipelines
On This Page
Edge Pipeline is currently available under Early Access. You can request access to evaluate and test its features.
Hevo Edge allows you to modify your Pipeline’s configuration after it is created. You can:
-
Change some of the Pipeline’s settings, such as the Pipeline name and the load mode. Read Modifying the Pipeline Configuration.
-
Change the configuration of the objects included in the Pipeline. For more details, read Managing Objects in the Pipeline.
-
Modify the Source configuration. Read Modifying the Source Configuration.
-
Modify the Destination configuration. Read Modifying the Destination Configuration.
Accessing Edit Pipeline
-
Log in to your Hevo account. By default, PIPELINES is selected in the Navigation Bar.
-
In the Pipelines List View, click the Edge tab, and then click the required Pipeline to open its detailed view.
-
To access the edit Pipeline screen, do one of the following:
-
Click EDIT in your Pipeline’s Summary bar.
-
Click Pipeline Setup in your Pipeline’s Toolbar, and then click EDIT in the Pipeline Configuration section on the displayed screen.
-
-
Follow the instructions in the next section to modify your Pipeline Configuration.
Modifying the Pipeline Configuration
Perform the following steps to change your Edge Pipeline’s settings:
-
Access the Edit Pipeline page.
-
On your <Pipeline Name> editing page, update the settings in the desired sections.
-
Click NEXT. This action takes you to the Object Configuration tab.
-
In the Object Configuration tab, click NEXT.
-
On the Summary page, scroll to the Pipeline Configuration section and verify all the changed settings in the NEW VALUE column.
-
If applicable, click the acknowledgment check box at the bottom of the page.
-
Click SAVE & EXIT to save the changed configuration.
Note: If you click CANCEL at any point before saving, the changes made to your Pipeline and or Object configuration are lost.
Hevo applies the changed configuration to your Pipeline in the following sync schedule.
Update the Pipeline settings
You can change the values of any of the fields in the following sections:
Note:
-
The sections that cannot be altered are grayed out.
-
If you click CANCEL at any point, the changes made to the Pipeline and or Object configuration are lost.
Pipeline Setup
In this section, you can modify the:
- Pipeline Name: Renaming the Pipeline does not affect your data. The modified name should not exceed 255 characters.
Select a Load Mode
In this section, select the desired Load Mode. You can change the mode in which your Pipeline loads data into the Destination tables from Append to Merge and vice versa.
When you change your load mode from:
-
Append to Merge: Hevo adds new records only if it does not find a match in the existing data; otherwise, it updates the matched records. When you switch to the Merge mode, Hevo deduplicates your data using primary keys. Hence, you need to ensure that primary keys are defined for all the selected objects in your Pipeline. If any of the selected objects do not have a primary key, you can define one for it in the Object Configuration screen.
Once you have saved your configuration changes to ensure data integrity between your Source and Destination data, Hevo performs the following actions:
-
Delete all existing data from your Destination table.
-
Re-ingest data from your Source table and load it to the Destination table.
-
-
Merge to Append: Hevo appends new records to the end of the dataset in your Destination table. As a result, you may see duplicates in your Destination table.
Failure Handling Policy
In this section, select the desired option from the Object Level Settings drop-down. Based on the current selection, you can change the policy to one of the following:
-
Don’t load data for Objects with failed Events: In this mode, Hevo loads data from only those Source objects that were successfully processed by a job.
-
Load data for Objects with less than 10K failed Events: In this mode, data from the Source objects processed by a job is not loaded to the Destination if the number of failed Events across all objects exceeds 10K.
Note: Selecting this option may lead to a mismatch between the Source and Destination data.
Schema Evolution Policy
In this section, select the desired Schema Evolution Policy to propagate or block all Source schema changes to your Destination schema.
When you change your Schema Evolution Policy from:
-
Allow all changes to Block all changes: Hevo does not sync any updates made in the Source schema, columns, and tables with the Destination. For example, if a schema is added to the Source after the policy is changed, Hevo does not automatically add the schema to your Destination.
-
Block all changes to Allow all changes: Any changes made to the Source schema, columns, and tables are synced with the Destination schema.
Sync Frequency
In this section, you can change your Pipeline’s schedule to sync data at a higher or lower frequency, based on the amount of data being ingested and or loaded.
Note: The sync frequency can only be changed at the Pipeline level, not for individual objects.
You can do one of the following:
-
Select Every ˋnˋ minutes to choose one of the values provided by Hevo. For example, 30 minutes.
-
Select Hourly and specify the number of hours in the range of 1-48. For example, 2.
The changed frequency is applied immediately, and the next sync job runs at the new frequency in your current time zone. For example, if you change the sync frequency from one hour to two hours and your last sync job ended at 11 AM, the next sync job will run at 1 PM instead of noon.
Pipeline Latency
In this section, you can enable or disable the Send alerts for Pipeline Latency option to receive notifications for latency in your Pipeline. When enabled, you must specify a value equal to or greater than 5 minutes in the Pipeline Latency (in minutes) field.
Note: The Pipeline Latency value cannot exceed 2880 minutes (2 days).