How to Use Fivetran's Sample dbt deployment.yml File

NOTE: This article applies to legacy transformations with dbt Core.

Use Case

You want to use the deployment.yml file to perform advanced orchestrations with your dbt Git repository. You want to do the following:

• Run specific models
• Run specific tasks
• Run other dbt commands

Environment

dbt Git repository and your local source code editor

Recommendation

Fivetran provides a sample deployment.yml file that enables us to run your dbt commands on a schedule. You can edit the jobs in the deployment.yml file to reflect how and when you want Fivetran to run your dbt commands.

The deployment.yml file allows you to define multiple jobs, each of which translates to a transformation in your Fivetran dashboard.

Each job include three main components:

• Name: Enter a unique job name. The name is displayed in your Fivetran dashboard once your jobs are imported.
• Schedule: Define when this job should run using the Cron format. It is possible to define a list of Cron schedules using a YAML array. Both the single line and multiline formats are supported.

  schedule: ["0 12 * * *", "0 5 * * *"] # Quotes needed to shield cron expression

  schedule:
    - 0 12 * * *
    - 0 5 * * * 

• Steps: A job performs multiple steps sequentially, so define each step here:

  • Name: Give each step in your job a name. This enables you to track the steps in the logs.
  • Command: Enter the dbt command that should run in this step.

Optionally, you can also specify if the dbt job should be retried in case of failure. By default, the failed job is retried up to 4 times. You can change this by setting the value of the retryOnFailure parameter to false in the job configuration:

- name: daily
  schedule: 0 12 * * *
  retryOnFailure: false

Common workflows

To ensure data integrity, use two steps (run the models and test the models) within a job:

- name: daily
  schedule: 0 12 * * * # This example will run every day at 12:00pm
  steps:
     - name: run models
       command: dbt run 
     - name: test models
       command: dbt test

You can also run the models in a specific order by including steps that only run certain models:

- name: daily
  schedule: 0 12 * * * # This example will run every day at 12:00pm
  steps:
     - name: run first sales model
       command: dbt run --models sales_model_1 
     - name: run second sales model
       command: dbt run --models sales_model_2