Transformations for dbt Core Setup Guide

Follow our setup guide to set up Transformations for dbt Core*.

Prerequisites

• A destination that supports Transformations for dbt Core

• An account with a Git provider of your choice (such as GitHub or BitBucket) with permissions to add deploy keys at the repository level

• A dbt project in your Git repository using a Fivetran-supported version of dbt Core. If you don't have a dbt project yet, follow our Create a dbt project guide to create one.

  Your dbt project cannot exceed 1 GB in size or have symbolic links.

Setup instructions

Get started

1. In your Fivetran dashboard, click Transformations in the left navigation column.

2. Click Add transformation.

3. Select your destination.

4. On the Select project type page, hover over dbt Core, then click Connect project.

   If you have already connected your dbt Core project and want to add a new transformation job, hover over dbt Core, then click Create job.

   You will be redirected to the Fivetran setup form.

   • Your dbt project must have a dbt_project.yml file. Learn more in dbt Labs' dbt_project.yml documentation.
   • By default, Fivetran expects your dbt project to be at the root of your Git repo. If your dbt project is elsewhere, you must specify its file path (for example, project/models) in your Fivetran setup form in the Finish Fivetran Configuration step.

Grant Fivetran access to Git repository

1. In the Fivetran setup form, find the public key and click the blue clipboard icon to copy it. You will need it to grant Fivetran SSH access to your Git repository.

2. Grant Fivetran SSH access to your repository in your Git provider. For help, see the instructions for your Git provider below.

   Expand to see how to grant SSH access instructions by Git provider

   • In GitHub: Follow GitHub's Deploy key setup instructions. In your SSH key's Key field, paste the Fivetran public key you found in the Grant Fivetran Access to Git Repository step of this section.

   • In GitLab: Follow GitLab's Deploy key instructions. In your SSH key's Key text box, paste the Fivetran public key you found in the Grant Fivetran Access to Git Repository step of this section.

   • In BitBucket: Follow these BitBucket SSH key setup instructions. In your SSH key's Key field, paste the Fivetran public key you found in the Grant Fivetran Access to Git Repository step of this section.

   • In Azure Repos: Follow step 2 of Microsoft's Azure Repos SSH key setup instructions. In your SSH key's Public Key Data text box, paste the Fivetran public key you found in the Grant Fivetran Access to Git Repository step of this section.

   • In AWS CodeCommit:

     i. Open your Amazon IAM console.

     ii. Click Add User to create a new AWS IAM user for Fivetran. This new user must have Programmatic Access and clone access to your repository.

     The easiest way to set up clone access is to give the user AWSCodeCommitReadOnly permission.

     iii. Open the newly created Fivetran user.

     iv. Go to the Security Credentials tab.

     v. Click Upload SSH public key.

     vi. In the SSH public key field, paste the Fivetran public key you found in the Grant Fivetran Access to Git Repository step of this section.

Finish Fivetran configuration

1. Find the URL of your Git repository. The URL must be in the format git@host.xz:path/to/repo.git (except for AWS CodeCommit).

   Find URL instructions by Git provider

   • In GitHub:

     i. Go to your repository page and click Code.

     ii. In the drop-down window, select SSH, then copy the repository URL that appears in the text box below.

   • In GitLab:

     i. Go to your repository page and find the HTTPS/SSH drop-down.

     ii. Select SSH, then copy the repository URL that appears in the text box on the right.

   • In BitBucket:

     i. Go to your repository's Source page.

     ii. Click Clone.

     iii. In the pop-up window, select SSH, then copy the repository URL that appears in the text box below (be sure to remove the git clone prefix). Click Close.

   • In Azure Repos:

     i. Go to your repository page and click Clone.

     ii. In the pop-up window, select SSH, then copy the repository URL that appears in the text box below.

     iii. Close the window.

   • In AWS CodeCommit, go to your repository page and copy the SSH URL.

     The SSH URL must be in the following format ssh://Your-SSH-Key-ID@git-codecommit.us-east-2.amazonaws.com/v1/repos/MyDemoRepo. To find your SSH Key ID, go to your Amazon IAM console > Users > IAM user you created for Fivetran > Security Credentials > SSH keys.

2. In the Fivetran setup form, enter your Git repository's URL.

3. Choose the Connection Method. There are two ways to connect to your git repository:

   • Directly
   • Using Proxy Agent

   The Fivetran Proxy Agent is installed in your network and creates an outbound network connection to the Fivetran-managed SaaS. This allows for secure communication between Fivetran processes and your repository without opening an inbound port in your firewall and/or other access control systems.

   Refer to our Connection Options documentation to learn how to generate Proxy Agent settings and install a Proxy Agent.

4. Enter your chosen Default Schema Name. This production schema will contain your transformed data. Learn more about schema names in dbt's Custom schemas documentation.

5. Choose the dbt Core Version to run your project.

6. (Optional) Add Environment Variables (optional) in the NAME=VALUE format.

   The variable name must start with the DBT_ prefix and contain the following characters only: A-Z, 0-9, dash, underscore, or dot. Variable names must use capital letters. For example, DBT_TESTING_VARIABLE=HELLO is correct, but DBT_testing_variable=hello is not correct.

7. By default, Fivetran does the following:

   • Works with the main branch of your repository
   • Searches for a dbt project in the root folder
   • Accesses your destination using the same credentials that you configured in the Destination section of your Fivetran dashboard
   • Uses the same values as defined for your destination for other additional settings, such as, for example, the number of threads for working on your dbt commands.

   If you want to override these parameters:

   i. Set the Show Advanced Options toggle to ON.

   ii. (Optional) Enter the Git branch.

   iii. (Optional) Enter the Project path (optional).

   iv. Manage the targets.

   Changing the target name used with an existing data model will update the target name for that model.

8. Click Save & Test. When your changes have been saved and tests have been passed, you will see a Done button.

9. Click Done.

You will be redirected to your main Transformations page. Fivetran will sync your project in a few minutes. Your dbt models are synced directly from your git repository.

Add a job

Add a new dbt job, either in the deployment.yml file or in the Fivetran Transformations tab. If you use the deployment.yml file, you can see the jobs appear in your Transformations tab after the next project sync.

* dbt Core is a trademark of dbt Labs, Inc. All rights therein are reserved to dbt Labs, Inc. Fivetran Transformations is not a product or service of or endorsed by dbt Labs, Inc.