Use our native integration with dbt to run your transformation models directly from our application.
At Fivetran, we are maniacally focused on creating automated connectors that make data as easily accessible and reliable as electricity — and easy to query in minutes.
But no matter how much work we put into moving data, businesses will always need to transform their data for their unique use cases and business logic. That’s why we released Fivetran Transformations last year. It was designed to take advantage of the powerful compute that is provided by data destinations today in a language all analysts understand: SQL.
As customers started to use our new transformation service, we knew we were onto something. Javid Igani, Manager of Internal Solutions at Skuid, described his experience this way:
Pretty much anything you can write in SQL you can put through a transformation and benefit from it because it runs in the same way it would if you were to take that code and run it in your database client. We don’t have to worry about how we’re going to get the data anymore and we can focus on delivering the types of reports that executives want. It is a really big game-changer.
While this was a great start, our customers needed more as their analytics strategy matured. We knew we had to keep improving and we started planning for what was going to come next!
As we started engaging with our customers, it became clear they had a few key requirements:
Transformations need to be in SQL. It is a common language that analysts and engineers could both work in.
Version controlling and peer reviews were needed so that teams could collaborate more easily and ensure top-quality data models.
Customers wanted the ability to run tests on their modeled data and generate documentation so other teammates could easily understand the models.
Customers like Ritual were struggling to manage data models across growing teams:
Code for the retention table was a mix of different languages spread across Python, SQL and LookML, making it difficult to understand what transformations were being used to get the final number and to update the data models.
When we started exploring how to solve these problems, it quickly became clear that we were going to end up rebuilding an open-source tool hundreds of our customers were already taking advantage of today: dbt by Fishtown Analytics.
So instead of reinventing the wheel, we reached out to the team over at Fishtown and started exploring a deeper partnership.
dbt is a data transformation and modeling tool that applies software engineering best practices — modularity, automated testing, documentation, and CI/CD (continuous-integration/continuous-delivery) — to build trusted, ready-to-query data sets. Analysts model their data as SQL select statements and create relationships and dependencies between models, and materialize models as tables and views in their warehouse. As of this writing, over 2,500 companies and 10,000+ analysts use dbt as the heart of their analytics engineering workflow.
At this point, it should come as no surprise that we are big fans of what the team at Fishtown Analytics, the company behind dbt, has been building. That’s why we decided to integrate with the open-source software dbt Core and announce, starting today, that Fivetran dbt Transformations is available in beta.
By doing this, Fivetran customers can take advantage of a best-in-class automated cloud data integration experience in a single environment. When customers use Fivetran for data integration they no longer have to worry about data plumbing between 150+ data sources for SaaS applications, databases, events and other cloud services and cloud data platforms. Now, with Fivetran dbt Transformations, Fivetran customers can orchestrate the cleaning, testing, transformation, modeling, and documentation of data sets.
Customers who want to get started with the integration can find that it’s just as easy as connecting a git repository with an existing dbt project in it. (Haven’t worked with dbt before? Get started here.) After Fivetran successfully connects with the git repository, we begin orchestrating the models and tests in the data destination that Fivetran already has access to.
This means that all the code generated by the analytics team is now centralized in a single source of truth repository and can be reviewed and tested before being pushed live, ensuring quality control of trusted data sets. Here's how it works at Ritual:
With dbt, the business no longer needs to search through random Python scripts and layered LookML files. dbt is the single source of truth for analytics code: In-warehouse transformations reduce complexity and ensure everything is in one place.
With Fivetran, we made it simple to define which jobs to run. All you need to do is edit the deployment file to reflect how often and when you want Fivetran to run your dbt models.
The file contains everything you need to get running and simplifies your data stack by eliminating the need for any third-party scheduling tooling.
dbt packages are libraries of reusable dbt models, similar to packages in other programming languages like Python or R. Fivetran delivers open-source dbt packages that take advantage of the normalized schemas you are already getting with Fivetran. These packages create analytics-ready aggregated data and canonical schemas in your destination.
The source packages expose and document the underlying Fivetran schemas being created in the destination. The model packages reproduce commonly-recreated models for each source.
Customers can use Fivetran dbt packages immediately to get a head start on an analytics project. For example, you can connect Fivetran to Zendesk, install Fivetran dbt packages for Zendesk, and have analytics-ready tables, including aggregations, that enable you to better understand the performance of your support team all within one day versus weeks or months.
Setting up logging and alerting systems is not everyone’s favorite job. That’s why Fivetran incorporates the logs from dbt into our UI so you don’t have to manage them yourself.
We make it easy to:
View the status and success history of every job
Dive into the details of each run
Receive alerts in the UI and via email if a job fails.
Alerts generated by dbt Transformations will immediately show in the Alerts tab. These alerts can be mapped to email notifications that you can choose to receive or keep only as a dashboard alert when you’re logged in to Fivetran.