Creating an Edge Pipeline

You can create a Pipeline in Hevo to synchronize data from your Source with a Destination. To get started with creating a Pipeline, you need:

• An active Hevo account. You can start with a 14-day full feature, free trial account.

• Access to the Source system (integration) where your data resides.

• Access to a Destination system in which the Source data is replicated.

  Note: At this time, Hevo supports creating Edge Destinations only with Snowflake.

---

Perform the following steps to create an Edge Pipeline:

1. Log in to your Hevo account. By default, PIPELINES is selected in the Navigation Bar.

2. (Optional) In the User Information bar at the top of the page, select the region in which you want to create your Edge Pipeline, if this is different from the default region of your workspace.

   

3. On the Pipelines page, click the Edge tab, and then + CREATE EDGE PIPELINE.

   

4. On the Create Pipeline page, do the following:

   

   1. Set up your Source from which you want to ingest data.

   2. Configure your Destination into which you want to replicate your Source data.

   3. Provide the Pipeline-specific settings.

   4. Select the Source objects from which you want to ingest data.

5. In the Summary screen, review all the configurations defined for the Pipeline.

6. Click the acknowledgment check box.

7. Click SAVE PIPELINE. This button is enabled after you have performed the step above.

You are automatically redirected to the Job History tab, which displays the jobs running in your Pipeline. You can click on a job to view its details.

---

Select and Set up the Source

Perform the following steps to set up your Source in the Edge Pipeline:

1. On the Create Pipeline page, under Source Configuration, do the following:

   1. In the Selection screen, select your Source as PostgreSQL.

   2. In the PostgreSQL screen, specify the following:

      

      • Source Name: A unique name for your Source, not exceeding 255 characters. For example, PostgreSQL Source.

      • In the Connect to your PostgreSQL section:

        Note: In Edge Pipelines, Hevo replicates data using the database logs.

        • Database Host: The PostgreSQL host’s IP address or DNS. For example, 10.123.10.001 or postgresql.westeros.inc.

          Note: For URL-based hostnames, exclude the http:// or https:// part. For example, if the hostname URL is https://postgres.database.azure.com, enter postgres.database.azure.com.

        • Database Port: The port on which your PostgreSQL server listens for connections. Default value: 5432.

        • Database User: The user who has permission only to read tables in your database. For example, hevouser.

        • Database Password: The password of your database user.

        • Database Name: The database from where you want to replicate data. For example, dvdrental.

        • Publication Key: The name of the publication in your PostgreSQL Source database added to track the changes in your database tables. Read Publications to understand them and how to create them.

      • Log Monitoring: Enable this option if you want Hevo to disable your Pipeline when the size of the WAL being monitored reaches the set maximum value. Specify the following:

        • Max WAL Size (in GB): The maximum allowable size of the Write-Ahead Logs that you want Hevo to monitor. Specify a number greater than 1.

        • Alert Threshold (%): The percentage limit for the WAL, whose size Hevo is monitoring. An alert is sent when this threshold is reached. Specify a value between 50 to 80. For example, if you set the Alert Threshold to 80, Hevo sends a notification when the WAL size is at 80% of the Max WAL Size specified above.

        • Send Email: Enable this option to send an email when the WAL size has reached the specified Alert Threshold percentage.

          If this option is turned off, Hevo does not send an email alert.

        Note: If you need to change the values specified for Max WAL Size and Alert Threshold after the Pipeline is created, contact Hevo Support.

      • Additional Settings

        • Connect through SSH: Enable this option to connect to Hevo using an SSH tunnel instead of directly connecting your PostgreSQL database host to Hevo. This provides an additional level of security to your database by not exposing your PostgreSQL setup to the public.

          If this option is turned off, you must configure your Source to accept connections from Hevo’s IP addresses. Refer to the content of your PostgreSQL variant for steps to do this.

        • Use SSL: Enable this option to use an SSL-encrypted connection. Specify the following:

          • CA File: The file containing the SSL server certificate authority (CA).

          • Client Certificate: The client’s public key certificate file.

          • Client Key: The client’s private key file.

   3. Click TEST & CONTINUE to test the connection to your PostgreSQL Source. Once the test is successful, you can set up your Destination.

---

Select and Configure your Destination

You must configure your Destination after you have set up the Source. To do this:

1. On the Create Pipeline page, in the Selection screen under Destination Configuration, do one of the following:

   

   • Click Select Existing Destination, and from the list of displayed Destinations, click the name of the Destination that you want to use.

   • Click Create New Destination to configure a new Destination, and then follow these steps:

     Note: You must select Create New Destination if you are creating your first Edge Pipeline or have not created any Edge Destinations.

     1. Select your Destination as Snowflake.

        Note: At this time, Hevo supports creating Edge Destinations only with Snowflake.

     2. In the Snowflake screen, specify the following:

        

        • Destination Name: A unique name for your Destination, not exceeding 255 characters. For example, Snowflake Destination.

        • In the Connect your Snowflake section:

          • Account URL: The URL for connecting to the Snowflake data warehouse. For example, https://xy12345.ap-southeast-1.snowflakecomputing.com.

          • Database User: The user with a non-administrative role in the Snowflake database that you created for Hevo. For example, HEVOUSER.

          • Database Password: The password of the database user.

          • Warehouse: The name of the Snowflake warehouse where the SQL queries and DML operations are performed for data replication. For example, SNOWFLAKE20.

          • Database Name: The name of the database in the Destination warehouse where the data is to be loaded. For example, HEVO_20.

          • Database Schema: The name of the schema in the Destination database where the Destination tables are created. For example, HDOCS.

          Note: All the field values are case-sensitive.

2. Click TEST & CONTINUE to test the connection to your Snowflake Destination. Once the test is successful, you can provide your Pipeline-specific settings.

---