At Fivetran, we are maniacally focused on creating automated connectors that make data as easily accessible and reliable as electricity — and easy to query in minutes.

But no matter how much work we put into moving data, businesses will always need to transform their data for their unique use cases and business logic. That’s why we released Fivetran Transformations for dbt Core* last year. It was designed to take advantage of the powerful compute that is provided by data destinations today in a language all analysts understand: SQL.

As customers started to use our new transformation service, we knew we were onto something. Javid Igani, Manager of Internal Solutions at Skuid, described his experience this way:

Pretty much anything you can write in SQL you can put through a transformation and benefit from it because it runs in the same way it would if you were to take that code and run it in your database client. We don’t have to worry about how we’re going to get the data anymore and we can focus on delivering the types of reports that executives want. It is a really big game-changer.

While this was a great start, our customers needed more as their analytics strategy matured. We knew we had to keep improving and we started planning for what was going to come next!

Listening to Our Customers

As we started engaging with our customers, it became clear they had a few key requirements:

• Transformations need to be in SQL. It is a common language that analysts and engineers could both work in.
  
• Version controlling and peer reviews were needed so that teams could collaborate more easily and ensure top-quality data models.
  
• Customers wanted the ability to run tests on their modeled data and generate documentation so other teammates could easily understand the models.
  

Customers like Ritual were struggling to manage data models across growing teams:

Code for the retention table was a mix of different languages spread across Python, SQL and LookML, making it difficult to understand what transformations were being used to get the final number and to update the data models.

When we started exploring how to solve these problems, it quickly became clear that we were going to end up rebuilding an open-source tool hundreds of our customers were already taking advantage of today: dbt Core by dbt Labs.

So instead of reinventing the wheel, we reached out to the team over at Fishtown and started exploring a deeper partnership.

dbt Core by dbt Labs

dbt Core, by dbt Labs is one of the most popular open-source transformations frameworks in the data analyst community. Analysts model their data as SQL select statements and create relationships and dependencies between models, and materialize models as tables and views in their warehouse. As of this writing, over 2,500 companies and 10,000+ analysts use dbt Lab products as the heart of their analytics engineering workflow.

dbt Core and Fivetran

At this point, it should come as no surprise that we are big fans of what the team at dbt Labs, the company behind dbt Core, has been building. That’s why we decided to integrate with the open-source software dbt Core and announce, starting today, that Fivetran Transformations for dbt Core is available in beta.

By doing this, Fivetran customers can take advantage of a best-in-class automated cloud data integration experience in a single environment. When customers use Fivetran for data integration they no longer have to worry about data plumbing between 150+ data sources for SaaS applications, databases, events and other cloud services and cloud data platforms. Now, with Fivetran Transformations, customers can orchestrate the cleaning, testing, transformation, modeling, and documentation of data sets.

‍

Fivetran dbt Transformations Overview - YouTube

Analytics and Data Engineering Working Together in a Single Tool

Customers who want to get started with the integration can find that it’s just as easy as connecting a git repository with an existing dbt Core project in it. (Haven’t worked with dbt Core before? Get started here.) After Fivetran successfully connects with the git repository, we begin orchestrating the models and tests in the data destination that Fivetran already has access to.

This means that all the code generated by the analytics team is now centralized in a single source of truth repository and can be reviewed and tested before being pushed live, ensuring quality control of trusted data sets. Here's how it works at Ritual:

With dbt Core, the business no longer needs to search through random Python scripts and layered LookML files. dbt Core is the single source of truth for analytics code: In-warehouse transformations reduce complexity and ensure everything is in one place.

Easily Manage Your Transformation Schedules

With Fivetran, we made it simple to define which jobs to run. All you need to do is edit the deployment file to reflect how often and when you want Fivetran to run your dbt Core models.

‍

The file contains everything you need to get running and simplifies your data stack by eliminating the need for any third-party scheduling tooling.

Accelerate time to insights

dbt packages are libraries of reusable dbt models, similar to packages in other programming languages like Python or R. Fivetran delivers open-source dbt packages that take advantage of the normalized schemas you are already getting with Fivetran. These packages create analytics-ready aggregated data and canonical schemas in your destination.

The source packages expose and document the underlying Fivetran schemas being created in the destination. The model packages reproduce commonly-recreated models for each source.

Customers can use Fivetran-developed and maintained dbt packages immediately to get a head start on an analytics project. For example, you can connect Fivetran to Zendesk, install Fivetran dbt packages for Zendesk, and have analytics-ready tables, including aggregations, that enable you to better understand the performance of your support team all within one day versus weeks or months.

Spend less time data engineering, data modeling and monitoring

Setting up logging and alerting systems is not everyone’s favorite job. That’s why Fivetran incorporates the logs from dbt Core into our UI so you don’t have to manage them yourself.

We make it easy to:

• View the status and success history of every job
  
• Dive into the details of each run
  
• Receive alerts in the UI and via email if a job fails

‍

Fivetran Transformations ensures that alerts generated by dbt Core will immediately show in the Fivetran Alerts tab. These alerts can be mapped to email notifications that you can choose to receive or keep only as a dashboard alert when you’re logged in to Fivetran.

Get started today!

Free 14 Day Trial – Experience data integration in minutes, not days!

CREATE YOUR ACCOUNT TODAY

Fivetran Transformations is now available to all customers while it is in Beta. Get started today by checking out our setup guide.

*dbt Core is a trademark of dbt Labs, Inc. All rights therein are reserved to dbt Labs, Inc. Fivetran Transformations is not a product or service of or endorsed by dbt Labs, Inc.