Working with Workflows

You can create a Workflow by linking multiple Models and Activations together. You can add conditions on the data in the Destination tables to define how the Models and Activations are run, and transform your data according to your requirements. You can schedule a Workflow to run at a pre-defined frequency. However, you can also use the Run Now option available under the More ( ) icon, to run the Workflow on-demand.

Prerequisites

At least one SQL or dbt™ Model is created in your account.

---

Example

Let us suppose, you have created three Models as follows:

• An SQL Model named CreatingCountries.

• Two dbt™ Models named CountriesByRegion and Film Rentals.

Then, the following image illustrates a sample Workflow created with these Models and a data load condition:

The Workflow execution takes place as follows:

1. The SQL Model CreatingCountries is run first.

2. The data load condition Quantity is evaluated.

3. One of the following happens depending on the outcome of the condition:

   • TRUE: All models in the Film Rentals dbt project are executed.

   • FALSE: All models in the CountriesByRegion dbt project are executed.

4. The transformed data is loaded to the Destination tables by the dbt project that was run.

---

Creating a Workflow

Perform the following steps to create a Workflow:

1. In the Asset Palette, click WORKFLOWS.

2. If this is the first Workflow that you are creating, click + CREATE WORKFLOW, else, click + CREATE in the Workflows List View.

   

   A Workflow is created in the DRAFT state.

3. Replace Untitled Workflow 1 with a meaningful name for the Workflow, and click SAVE. The Workflows Detailed View is displayed.

   

4. In your Workflow canvas, click:

   • ADD MODEL to add an SQL Model or a dbt project

   • ADD ACTIVATION to add an Activation

   • ADD CONDITION to add data load conditions

5. Click on a Model or Activation block and perform the following steps:

   

   1. Specify the following settings:

      • Select Model or Activation: Select an SQL Model, a dbt project, or an Activation from the drop-down list that is displayed. When you select a dbt project, all the models within it are added to the Workflow. Optionally, you can add an individual model from the dbt project.

        For example, the image below displays the models in the PGDVDRentals dbt project. You can add one of these models, say country_sales_model instead of the entire dbt project to your Workflow.

        

      • Stop on Failure: Enable this option to stop the Workflow in case the Model or Activation fails due to some reason.

      • Run Outside Workflow: Enable this option to allow the selected Model, dbt project, or Activation to also run independent of the Workflow.

        For example, in the image below, the SQL Model, FilterbySalaries is configured to run on an independent schedule as well.

        

   2. Click SAVE CONFIGURATION. Repeat Step 6 for each Model and Activation that you want to add.

6. (Optional) Click ADD CONDITION to specify a condition for linking the Models and Activations and specify the following settings:

   

   • Name of the Condition: A unique name for the data load condition.

   • Conditions: The condition(s) that you want to apply to the data loaded into the Destination table(s) since the last run of the Workflow. You can add multiple conditions and combine them with the And or Or operators.

     • Number of rows loaded in: Select one of the Destination tables that your team has access to, on which you want to apply the condition.

     • Select one of the Greater Than Equal To, Lesser Than Equal To, and Greater than Zero conditions. These are used to compare the number of rows loaded in the table since the last run of the Workflow and the value that you provide for comparison.

     • Value: The number to be used for comparison in the condition selected above. For example, 10.

7. Hover the mouse over a block to display arrows on its four sides. These arrows enable you to add other blocks, as required, to create the Workflow.

8. Click the arrow symbol of the first block in your Workflow, hover over the second block, and click on one of the hollow dots displayed on the four sides of the second block to connect the two blocks.

9. Optionally, use the Undo, Redo, and CLEAR ALL options to modify your Workflow.

   Note: The CLEAR ALL action cannot be undone.

   

Note: You can only use Models and Activations that are in Active state to create a Workflow.

---

Scheduling a Workflow

After successfully creating a Workflow, you are required to set a schedule as per which the Workflow must run.

There are two ways in which you can do this:

• Runs Every: Schedule the Workflow to run at a fixed frequency. The highest frequency is 1 Hour.

• Custom: Choose Custom in Runs Every option to define a specific time for the Workflow run. The Run On drop-down provides you three options in this:

  • Fixed Interval of: Provide a time interval in hours, at which the Workflow must run. Let us suppose you specify the interval as 4 hours, and as per that, the Workflow is scheduled to run at noon, 4 PM (UTC), 8 PM (UTC), and so on. If you manually run the Workflow between these times, say at 5 PM (UTC), then the next run happens 4 hours after that, at 9 PM (UTC) and not 8 PM (UTC).

  • Daily: Schedule the Workflow to run daily at the specified time.

  • Weekly: Select the days of the week, and the time when you want the Workflow to run.

Click SCHEDULE to set up the schedule. In the Summary Bar, click Change to change the schedule for an existing Workflow.

---

Modifying a Workflow

You can modify a Workflow after creating it. To do so:

1. Select your respective Workflow from the Workflow List View.

2. Click EDIT WORKFLOW.

   

3. Make your changes, and once you are done, click SAVE CHANGES.

   

---

Revision History