Configuring a dbt™ Project

Last updated on Jan 02, 2023

At this time, the dbt Core™ on Hevo is in BETA. Please reach out to Hevo Support or your account executive to enable it for your team.

You can create dbt projects in Hevo and load the transformed data to tables and views in your Destination. These targets are created the first time you run the dbt project. In subsequent runs, the structure of your targets is retained but the existing data in them is deleted and new data loaded.

You can run the dbt project as per a defined schedule or use the Run Now option to run it on-demand. Read Scheduling a dbt Project for the details.


Prerequisites

  • An account in any Git provider such as GitHub, GitLab, or Bitbucket.

  • Read-only access to the Git repository containing your dbt project.

  • One of the following for Hevo to connect to your Git repository:

    • For all Git providers:

    • For GitHub only:

      • The repo OAuth scope is granted to Hevo in GitHub.
  • The path to your dbt project configuration file in the Git repository.

  • The details to add a Destination or an existing Destination from the following types:

  • The tables on which the dbt project must run exist in your Destination.


Obtaining a Personal Access Token

Hevo requires a Personal Access Token (PAT) or an App password to connect to your Git provider with the HTTPS URL.

To create a PAT or an App password:

Creating a personal access token in GitHub

  1. Log in to your GitHub account.

  2. In the upper-right corner of the page, click your profile photo, and click Settings.

    Click Settings

  3. In the left navigation pane of Your Profile page, scroll to the bottom, and click Developer settings.

    Developer Settings

  4. In the Developer settings page, under Personal access tokens, click Tokens (classic).

    Note: At this time, Hevo only supports classic PATs.

    Classic Tokens

  5. In the Personal access tokens (classic) page, click Generate new token, and select Generate new token (classic) from the drop-down.

    Generate new token

  6. In the New personal access token (classic) page:

    New classic token

    • Describe your token in the Note field.

    • Set the Expiration to one of the predefined values or select Custom in the drop-down, to set a date using the calendar. Default value: 30 days.

    • Select the scopes of your token. Hevo requires the repo scope to be able to clone a private repository.

  7. Scroll to the bottom of the page and click Generate token.

  8. Copy the displayed token and save it securely, like any other password.

    New generated token

You can provide the PAT while connecting to your Git repository using HTTPS in the Configure your dbt™ Model page.


Configuring your dbt Project

  1. In the Asset Palette, click MODELS.

  2. If this is the first Model you are creating in Hevo, perform the actions listed in this step, else, skip to Step 3.

    1. Click + CREATE MODEL.

      Create Model

    2. In the Select Destination page, do one of the following:

      • Select an existing Destination.

        Select Destination

      • Click + ADD DESTINATION to configure a new Destination.

    3. In the Select your Model variant page, select dbt™ Models and click CONTINUE.

      Select dbt Models

    4. Skip to Step 4.

  3. To create your first dbt project:

    1. Select the dbt Core™ tab in the Models List View, and click CREATE dbt™ MODEL.

      Select dbt tab

    2. In the Select Destination page, do one of the following:

      • Select an existing Destination.

        Select Destination

      • Click + ADD DESTINATION to configure a new Destination.

  4. In the Configure your dbt™ Model page, specify the following:

    Configure your dbt Model

    • dbt™ Model Name: A unique name for your dbt project.

    • dbt™ Repository Settings: The Git repository containing your dbt projects.

      Select one of the following ways to connect to the repository:

      • Using GitHub OAuth: Select this method to connect to your GitHub repository.

        1. Click CONNECT GITHUB ACCOUNT.

        2. Sign in to your GitHub account, and click Authorize to grant Hevo access to your repository.

          Authorize Hevo

        3. Select your Git repository details from the drop-down in each field:

          GitHub repository details

          • Authorized Account: The GitHub account to which you granted access in the previous step.

          • Select Repository: The Git repository linked with your dbt project.

          • Branch: The branch in your Git repository that contains your dbt project.

          • dbt™ Core Version: The version in which you created your dbt project. Default value: 1.3.

      • Connect using HTTPS: Select this method to connect to your Git repository in any Git provider, such as GitHub, GitLab, and Bitbucket. Specify the following:

        Connect using HTTPS

        • Git HTTPS: The HTTPS URL of the repository linked with your dbt project. For example, https://gitlab.example.com/namespace/mydbtproject.git is the HTTPS URL of a GitLab repository called mydbtproject.

        • Git Branch: The branch in your Git repository that contains your dbt project. For example, main.

        • Personal Access Token: A secret value to authenticate Hevo with your Git provider. Refer to your Git provider’s documentation for the steps to create one.

          Note: At this time for a GitHub repository, only classic personal access tokens are supported.

        • dbt™ Core Version: Select the version in which you created your dbt project from the drop-down. Default value: 1.3.

    • Advanced Settings:

      • Project Path: The location of your dbt project configuration file, dbt_project.yml in your Git repository. For example, /dbt_sample_project.

      • Threads: The number of independent models in your dbt project that dbt can simultaneously work on. For example, if this value is set to 3, dbt can work on 3 independent models. Default value: 15. You can set a value from 1-32. However, increasing the number of threads affects the load on your Destination and may cause other processes in your database or data warehouse Destination to slow down. Read Understanding threads to see how threads affect the dbt project and your data warehouse cost and resource utilization.

  5. Click TEST CONNECTION if you want to check your configuration.

  6. Click TEST & CONTINUE.

Your dbt project is created. To run the project and the models in it, you need to create a schedule. If you want, you can also run your project manually.



Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Jan-02-2023 NA Changed from PAT to App password for Bitbucket.
Nov-25-2022 NA New document.

Tell us what went wrong