How Does the New Transformation Flow Built Around dbt Jobs Differ From the Former Flow Built Around Output Models?

This section provides answers to several questions about the differences and benefits of the new transformation workflow built around dbt jobs in comparison to the old workflow built around output models.

Environment

Transformations for dbt Core* and Quickstart transformations

What’s new?

Fivetran Transformations now provide:

• User-defined jobs for dbt Core
• Quickstart jobs with simplified output-model toggling
• Cron scheduling for dbt Core
• Integrated scheduling support in deployment.yml

Do I need to do anything to switch to the new transformation workflow?

No action is required on your part. Your current jobs will continue running as usual. These updates simply give you more flexibility and control if you choose to use the new features.

What are user-defined dbt jobs?

You can now configure the dbt commands you want to schedule directly in your Fivetran dashboard. If you’re familiar with the deployment.yml file, this essentially brings a similar functionality into the Fivetran dashboard. You can set up multiple commands for a job and customize their execution order.

Here’s an example of what it looks like:

How does this new workflow built around dbt jobs compare to the previous workflow built around output models?

Previously, scheduling was limited to output-models in the dashboard. This often led to cluttered dashboards with numerous scheduled models, making debugging and schedule adjustments more challenging. While we introduced a bulk operations tool to mitigate this, we decided on a more fundamental re-design.

Have Integrated schedules changed?

Integrated schedules now let you trigger transformations based on specific connections. Previously, we automatically inferred associated connections from output models, which was restrictive for some use cases. Now, you have the flexibility to select which connections drive a job.

What’s the impact of the change on Quickstart transformations?

For Quickstart transformations, scheduling and toggling output models have been streamlined. You’ll schedule transformations at the package level and manage individual output-models or reports on the Data Models tab.

Why am I seeing fewer jobs?

We’ve combined jobs where their details such as their schedules or integrated connections match, to make sure they are efficient and have fewer model runs for intermediate models, saving you CPU costs on the warehouse.

If you see a number appended to the job name (for example, RUN_MODELS/prod/dbt/kpi_report+5 ), this means that we’ve added that number of other output models with matching parameters to the original output model and combined them to a single job. You can see details of these job commands on the Job commands tab of your transformation details page.

Why are my job names different?

We have included multiple details in the names of migrated jobs, ensuring they are clearly defined and easily distinguishable.

Here are some examples for Quickstart and dbt Core:

• Quickstart transformations:
  • Naming convention: The old naming pattern PackageDefinition - [displayName] has been replaced with the new naming pattern Integration/name
  • Example:
    • Before: Fivetran Log Data Model - [fivetran_log]
    • After: Fivetran Log/fivetran_log
• Transformations for dbt Core:
  • Naming convention: The old naming pattern output_model_name has been replaced with the new naming pattern run_type/target_name/output_schema_name/output_model_name, with +5 denoting the number of combined jobs.
    • Before: kpi_report
    • After: RUN_MODELS/prod/dbt/kpi_report+5

Has anything changed in the deployment.yml file?

Scheduled in Code transformations remain the same, but we’ve added Integrated scheduling to deployment.yml. You can now specify connection IDs directly in the job schedule for sync-triggered execution.

Here’s an example:

- name: integrated-job
  schedule:
    type: integrated
    integrations:
      - abject_normative
      - prodigy_vaso

How the new transformation flow affects different types of transformations

Below is a summary of the before and after differences across different types of transformations:

Scheduled in Fivetran transformations for dbt Core

The former output model-centered flow:

• You can only select models.
• You cannot select upstream connectors, they are inferred by the selected model.

The current dbt job model-centered flow:

• You define the target.
• You specify the job's name.
• You add one or more dbt job commands. They are run in the order in which you added them.
• If you choose to use Integrated or Custom with Smart Syncing schedule, you can add upstream connectors.
• In addition to Integrated and Custom scheduling, Cron scheduling is available.

Scheduled in Code transformations for dbt Core

The former output model-centered flow:

• deployment.yml file only supports cron scheduling.

The current dbt job model-centered flow:

• deployment.yml file supports both Integrated and Cron scheduling.
• For Integrated scheduling, you select the upstream connections.

Quickstart transformations

The former output model-centered flow:

• Quickstart transformations are displayed in an expandable, accordion-style view.
• Each connector package model in the transformation has its own schedule.
• You can only have a single Quickstart transformation per connection.
• You need to remove every single model.

The current dbt job model-centered flow:

• The transformation has a single schedule that is common for the entire connector package model.
• You can include or exclude each connector package model.
• If you choose to use Integrated or Custom with Smart Syncing schedule, you can add upstream connections, not necessarily of the same type as the one you used when creating the transformation.
• In addition to Integrated and Custom scheduling, Cron scheduling is available.

* dbt Core is a trademark of dbt Labs, Inc. All rights therein are reserved to dbt Labs, Inc. Fivetran Transformations is not a product or service of or endorsed by dbt Labs, Inc.