Databricks (Edge)

Note: If you already have a Databricks workspace that you want to connect to Hevo, skip to the Create a SQL Warehouse or an All-purpose Cluster section.

A workspace is a Databricks deployment in your cloud service account. This is the environment where you and your team access your data, run queries, and manage the resources that Databricks provides. Databricks assigns a unique URL to each workspace, and you use this URL to log in and to connect applications such as Hevo.

Perform the following steps to create a workspace:

1. Log in to your Databricks account console.

2. Create the workspace for your cloud provider. Read the AWS, Azure, or GCP documentation for the steps.

   You are assigned the workspace administrator role on the workspace that you create. This role is required to create a SQL warehouse or a cluster, grant privileges on a catalog, and create an IP access list.

3. Add the team members who need access to the workspace, and assign them the privileges to create and manage a SQL warehouse or a cluster. Read Manage users for the steps.

Use this workspace to create the SQL warehouse or cluster that Hevo connects to.

1. Log in to your Databricks workspace.

2. In the left navigation pane, under SQL, click SQL Warehouses.

   

3. On the Compute page, click Create SQL warehouse.

   

4. In the New SQL warehouse dialog, specify the following:

   

   • Name: A unique name for your warehouse. For example, hevo_docs_wh.

   • Cluster size: The size of the warehouse. A 2X-Small warehouse is suitable for most Hevo workloads. The dialog displays the cost of the selected size in Databricks Units per hour (DBU/h).

   • Auto stop: The idle time after which the warehouse stops.

   • Type: The warehouse type. Serverless warehouses start in a few seconds, while Pro and Classic warehouses take a few minutes.

5. Click Create.

   Databricks creates the warehouse, starts it, and displays the Manage permissions dialog. You are assigned as the owner of the warehouse that you create.

   

1. Log in to your Databricks workspace.

2. In the left navigation pane, click Compute, and then click the All-purpose compute tab. Databricks labels an all-purpose cluster as all-purpose compute in its user interface.

   

3. On the Compute page, click Create compute.

   

4. On the Create new compute page, specify the following:

   

   • Compute name: A unique name for your cluster. For example, hevo_docs_cluster.

   • Policy: The set of rules that limits the configuration settings available to you. Default value: Unrestricted.

   • Machine learning: Select this check box to use a Databricks runtime that includes machine learning libraries. Hevo does not use these libraries.

   • Databricks runtime: The runtime version for the cluster. Select a Long Term Support (LTS) version.

   • Photon acceleration: Select this check box to use Photon, the Databricks query engine that accelerates SQL workloads.

   • Preferred worker type: The type of worker node, which determines the memory and the number of cores available to the cluster.

   • Min and Max: The lowest and the highest number of worker nodes that Databricks runs. These fields apply only when Enable autoscaling is selected.

   • Single node: Select this check box to create a cluster with a driver node and no worker nodes. This is the smallest configuration, and it is suitable for the SQL queries that Hevo runs.

   • Enable autoscaling: Select this check box to allow Databricks to add and remove worker nodes based on the load.

   • Terminate after: The idle time after which the cluster stops.

5. Click Create.

   Databricks creates the cluster and starts it. You are assigned as the owner of the cluster that you create.

1. Log in to your Databricks workspace.

2. In the left navigation pane, under SQL, click SQL Warehouses, and then click the name of your warehouse.

   

3. Click the Connection details tab.

   

4. Copy the Server hostname and the HTTP path values, and save them securely like any other password.

   

   • The workspace URL is the server hostname prefixed with https://. For example, if the server hostname is dbc-xxxxxxxx-xxxx.cloud.databricks.com, the workspace URL is https://dbc-xxxxxxxx-xxxx.cloud.databricks.com.

   • The HTTP path of a SQL warehouse is in the format /sql/1.0/warehouses/<warehouse ID>. Hevo also accepts the older /sql/1.0/endpoints/<warehouse ID> form.

1. Log in to your Databricks workspace.

2. In the left navigation pane, click Compute, and then click the name of your cluster.

   

3. In the Configuration tab, expand the Advanced section, and click JDBC/ODBC.

   

4. Copy the Server hostname and the HTTP path values, and save them securely like any other password.

   • The workspace URL is the server hostname prefixed with https://. For example, if the server hostname is dbc-xxxxxxxx-xxxx.cloud.databricks.com, the workspace URL is https://dbc-xxxxxxxx-xxxx.cloud.databricks.com.

   • The HTTP path of a cluster is in the format sql/protocolv1/o/<workspace ID>/<cluster ID>. Databricks displays this value without a leading slash.

Perform the following steps to grant the Can use permission:

1. Log in to your Databricks workspace.

2. In the left navigation pane, under SQL, click SQL Warehouses, and then click the name of your warehouse.

   

3. Click Permissions.

   

4. In the Manage permissions dialog, click the Type to add multiple users or groups field, and select the Databricks user that Hevo connects as.

   

5. Select Can use in the permission list, and then click Add.

Your Databricks user can now run queries on this SQL warehouse.

Perform the following steps to grant the Can Restart permission:

1. Log in to your Databricks workspace.

2. In the left navigation pane, click Compute, and then click the name of your cluster.

   

3. Click the More icon in the top right corner, and then click Permissions.

   

4. In the Permission Settings dialog, click the Select user, group or service principal field, and select the Databricks user that Hevo connects as.

   

5. Select Can Restart in the permission list, click Add, and then click Save.

   Note: The Can Restart permission allows the user to start, restart, and terminate the compute, and also includes the Can Attach To permission.

Your Databricks user can now start this cluster and run queries on it.

Note: This section is applicable only if your workspace uses Unity Catalog.

Perform the following steps to grant the privileges on the catalog:

1. Log in to your Databricks workspace as a user with the MANAGE privilege on the catalog.

   Note: A workspace administrator holds this privilege. If you do not have it, ask your administrator to run the script for you.

2. In the left navigation pane, under SQL, click SQL Editor.

   

3. Copy the following script into the editor, and replace the sample values with your own:

   -- Replace "main" with the name of the catalog that Hevo loads data into
   -- Replace "hevo_user@example.com" with the Databricks user that owns the token
   
   GRANT USE CATALOG, CREATE SCHEMA ON CATALOG `main` TO `hevo_user@example.com`;
   
   -- Grant access to the schemas that Hevo creates and loads data into
   GRANT USE SCHEMA, CREATE TABLE, MODIFY, SELECT ON CATALOG `main` TO `hevo_user@example.com`;
   

   Note: All the privileges are granted on the catalog. In Unity Catalog, a privilege granted on a catalog applies to all the current and future schemas and tables in it. Hence, you do not need to grant them again for the schemas and tables that Hevo creates later.

4. Select your SQL warehouse or cluster in the compute list at the top of the editor.

5. Click Run all to run the script.

   

Your Databricks user can now create the schemas and tables that Hevo loads your data into.

Databricks lets you restrict the addresses that can reach your workspace with the IP access list feature. If this feature is enabled for your workspace, Hevo cannot connect until you add the Hevo IP addresses of your region to an allow list.

Note:

• If the IP access list feature is not enabled for your workspace, skip to the Configure Databricks as a Destination section.

• You must have the workspace administrator role to create an IP access list.

Databricks provides a command-line interface and a REST API for configuring the IP access lists of a workspace. To add the Hevo IP addresses, call the Create access list API with the POST method from any API client, such as Postman or a Terminal window. Use the personal access token that you created in the Create a Databricks Personal Access Token section as the Bearer token for making the API call. Specify the following in the JSON request body:

• label: A string value to identify the access list. For example, HEVO.

• list_type: A string value to identify the type of list created. This parameter can take one of the following values:

  • ALLOW: Add the specified IP addresses to the access list.

  • BLOCK: Remove the specified IP addresses from the access list, or block connections from them.

• ip_addresses: A JSON array of IP addresses and CIDR ranges, given as string values. Use the Hevo IP addresses of your region and their CIDR ranges in this parameter.

The base path for the API endpoint is https://<deployment name>.cloud.databricks.com/api/2.0. For example, if the deployment name is dbc-westeros, the URL to call the Create access list API is https://dbc-westeros.cloud.databricks.com/api/2.0/ip-access-lists.

The following example adds an access list to allow the Hevo IP addresses for the Asia region:

curl -X POST -n \
    -H "Authorization: Bearer <your personal access token>"
    -H "Content-Type: application/json"
    -H "Accept: application/json"
        https://<deployment-name>.cloud.databricks.com/api/2.0/ip-access-lists
        -d '{
          "label": "HEVO",
          "list_type": "ALLOW",
          "ip_addresses": [
              "13.228.214.171/32",
              "52.77.50.136/32"
            ]
          }'

Note: Replace the placeholder values in the command above with your own. For example, <deployment-name> with dbc-westeros.