HomeBlog
The Evolution of Data Engineering
Fivetran
HomeBlog
The Evolution of Data Engineering

The Evolution of Data Engineering

As cloud-based tools increasingly handle the grunt work of ETL, data engineers can start thinking bigger.

By Jason Harris, March 26, 2021

When fashion startup Rent the Runway (RTR) hit the ecommerce catwalk 11 years ago, it reinvented the way shoppers interact with clothing. Its platform connecting eager fashionistas with an endless “closet in the cloud” ushered in a major disruption to the retail fashion industry, and netted the company a $750 million valuation.

But powering that platform and the company’s physical warehouses involves a lot of logistics — and even more data. Automated data tools enabled the company’s growth, and adopting them has changed the role of the RTR data engineering team. At Data Engineer Appreciation Day, RTR’s Danielle Dalton, Data Engineer III, shared how her team is staying current.

Embrace Change and “Control Your Destiny”

At RTR, Dalton and her team are thinking bigger than ETL. “We try to control our own destiny,” she said. “We try to do that on the data engineering team by leveling up our skills and supporting the general infrastructure overall and not just the data ingestion itself.”

The factors driving that shift aren’t unique to fashion or ecommerce. With more teams adopting automated cloud-based tools, the modern data stack, is freeing up data engineering teams across industries, enabling them to focus on more strategic initiatives, instead of spending time on manual coding and fixing broken pipelines.

Dalton’s team, for example, has dedicated time over the past year to setting up generalized frameworks that can be used to easily onboard new vendors or data sources. Another big push at RTR is setting up a self-service data model that can empower data users throughout the company. While the upfront work is substantial, she expects it will pay dividends down the road.

“This past year we laid the foundational blocks that will help us speed up,” said Dalton.

Establish Yourself as a Strategic Advisor 

With reliable data ingestion a given, Dalton and her team are now focusing on how they can provide the best guidance — not just to their direct clients inside the business, but also to the end users of the data. “It’s very much trying to serve the business overall, and just doing it in the context of data,” she said.

That kind of expertise positions data teams like hers to become strategic advisors, and move away from being order-takers. For example, her data engineering team’s direct clients are composed of five other specialized data engineering teams, each embedded in a different functional area of the business. Every time a new project request comes in, her team partners with the client to truly understand the larger business context for the ask, and then guides them through the cost and resource tradeoffs of different solutions.

“We try to get requirements up front to help gauge those decisions,” she said. “What are the resource constraints? What’s the best way we can serve you? We really take it as a partnership.”

Regular communication is also crucial. The data teams meet every week to share at a high-level what they’re working on, so large projects don’t take anyone by surprise.

“We don’t expect to get it right from the very beginning” of each new project, she said. “But definitely having more infrastructure allows us to iterate and meet their needs as we go along.”

Instill Trust in the Data

As part of RTR’s self-service push, analysts are rolling out Looker as a business intelligence tool for end users of the data, and Dalton’s team is supporting them in that effort. As her team continues to evolve their skills and provide strategic guidance, they’re laser-focused on instilling confidence in the data.

“You have to have really good underlying data and if you don’t have that, it’s not going to work,” she said of the shift to bigger-picture projects. “You have to put that work in up front. It’s not something a shiny new tool is going to solve.”

“It’s better to start a little bit slower and roll things out model by model and build up that trust,”  she said. “It’s hard to earn back trust that’s been eroded in bad data.”

The data engineer’s role is shifting, and for the better. This in-demand career will refocus on innovation, customization and optimization projects that ultimately bring even more value to the company.

Watch all the Data Engineer Appreciation Day Conference sessions

View the Sessions Now

Start analyzing your data in minutes, not months

Launch any Fivetran connector instantly.