One of the arguments against Fivetran is that, once adopted, it will eliminate the need for data engineers. We get where this comes from, but Fivetran has really saved businesses from having to hire additional engineers. Without Fivetran, DiscoverOrg would need to hire two to three additional engineers, Moroch Agency would need an additional engineer and DBA, Pleo would need three additional engineers, and Strava would need an additional full-time engineer to support just the marketing team’s data needs.

We’re here to argue that data engineering jobs are safe. Despite the labor-saving nature of Fivetran, the industry still has a shortage of data engineering talent and the demand for the role continues to grow.

Eliminating manual ETL to open opportunity for data engineers

Businesses need data engineers, but not for ETL. This is actually pretty great news because data engineers don’t actually want to do ETL.

Jeff Magnusson, VP of Data Platforms at Stitch Fix, wrote a blog post back in 2016 that really hit this message home: “A common fear of engineers in the data space is that, regardless of the job description or recruiting hype you produce, you are secretly searching for an ETL engineer. In case you did not realize it, nobody enjoys writing and maintaining data pipelines or ETL. It’s the industry’s ultimate hot potato. It really shouldn’t come as a surprise, then, that ETL engineering roles are the archetypal breeding ground of mediocrity.”

What it comes down to is that data engineers are extremely talented, valuable resources that should be dedicated to developing proprietary products and niche solutions to help your business excel. Why would you dedicate them to the tedium of building and maintaining pipelines when a solution like Fivetran can handle that process for you? And further, do you think your engineers want to be working on problems that they could easily solve with an existing tool?

The role of a modern data engineer

So, what does a data engineer do? With data pipelines automated, the role of the data engineer is transformed. Three years after Magnusson shared his thoughts, Tristan Handy of Fishtown Analytics shared what he believed modern data engineers should focus on, which he explains in this article:

• managing and optimizing core data infrastructure
• building and maintaining custom ingestion pipelines
• supporting data team resources with design and performance optimization
• building non-SQL transformation pipelines

Five companies where data engineers make a difference

We’ve heard plenty of stories from data engineering teams and their colleagues that emphasize how critical the role is. Eliminating ETL from the data engineer job description enables engineers to focus on innovation, customization and optimization. We’d say data engineers are enjoying a renaissance.  

1. Sendbird

The Head of Operations and BI at Sendbird admits that getting buy-in for Fivetran from the data engineering team was difficult at first. They weren’t convinced that a solution could automatically and reliably deliver the data to the warehouse without maintenance. Once they saw the tool in action, their opinion changed. With daily manual tasks alleviated and nearly 20 hours per month per engineer saved, they were able to dedicate their time to building a data lake. Besides enabling their engineering projects, the freed up hours have enabled the team to understand their broader business impact. They’re looking at how product features and engineering impact sales and retention.

2. The Ignition Group

The Chief Technology & Innovation Officer of The Ignition Group found himself in a similar situation. When he introduced Fivetran to the Data Warehouse team, they pushed back. Now, after seeing it in action, the team is the biggest fan of Fivetran. With its time back, the team can partner with other business units, gather feedback and learn, truly supporting the entire business rather than awkward hardware configurations. With Fivetran in place, the team has been able to focus on helping the company’s analysts by writing dimension views for its Snowflake warehouse.

3. Square

At Square, engineers spend less time worrying about rebuilding infrastructure within multiple legacy systems and instead build the infrastructure for new products. With Fivetran helping to scale the pipelines engineers can focus on building out 1st-party solutions for business needs.

4. Fountain

With reliable data automatically integrated into its warehouse, Fountain is able to pursue machine learning rather than spending time building and maintaining data pipelines.

5. Billie

With pipelines taken care of, German fintech startup Billie is able to improve documentation, data cataloguing for internal stakeholders such as the data science and analytics teams, and cost-optimization. Focus has shifted from pipelines to transforming the data to make it more useful for users. VP of Data Igor Chtivelband is excited about the future for data engineers: “Data engineers shouldn’t be afraid that they’re going to be out of a job. Every new technology creates new jobs and new opportunities. From the elimination of manual, mind-numbing tasks, comes the opportunity for more sophistication.”

What projects would you be able to work on if you didn’t have to do ETL? Tweet us and let us know!‍