Listening to the market

In the spring of 2014, a year out of Y Combinator, Fivetran built its first data connector. The technology was incidental, an extension for the new company’s “real” product: a pivot table for big data built on Amazon Redshift. Redshift happened to be the fastest-growing product in Amazon’s history, and customers were looking to move data into Redshift from sources like Salesforce and Zendesk. So the Fivetran team — co-founders Taylor Brown and George Fraser and VP of Engineering Meel Velliste — built a few connectors to the pivot table’s Redshift backend. Then they got back to work.

But the genie had escaped the bottle. Word about the connectors spread, and an early customer contacted the Fivetran team and offered to pay for a Salesforce-Redshift data connector. Would they build it?

“In a lot of ways, Fivetran happened at that moment,” George recalled at a recent company retreat in Colorado. “We already had a lot of customers saying, ‘Redshift’s working well for me, and my BI tool’s working well for me, but man, there are just no good data pipelines out there, and you guys say you have this data pipeline under the hood — I want to hear about that.’”

When the market speaks to you this clearly, you should probably listen. George and Taylor, now Fivetran CEO and COO, respectively, agreed to build the Salesforce connector. Taylor closed the deal in his “conference room” — an attached shed he shared with a hot water heater. (Somehow, in those meager quarters, his laptop perched on a cardboard box, he drove a preposterously hard bargain.) They promptly redesigned their website, erasing all traces of the pivot table in favor of “Data Pipelines for Redshift.”

Completing the modern data stack

The Fivetran team hadn’t just achieved product-market fit; they had committed themselves to solving an incredibly complex engineering problem. Data replication was the single largest obstacle for companies seeking to turn data into insight — and arguably the major data engineering challenge of the day.

As they discovered, there were no fully automated, fully managed data pipelines for professional data teams. Most companies relied on traditional ETL methodology. That meant devoting a huge amount of engineering resources to constructing bespoke pipelines that tended to break and leak — and building additional pipelines just to ask additional questions of the data.

To develop automated connectors for every common source type and deliver standardized, ready-to-query data, Fivetran would have to solve a host of engineering problems. How could you build data pipelines that automatically detected and adjusted to new cloud API features and database schema changes? What was the best way to account for attribution windows? How could you create optimal ERDs for semi-structured data?

Five years after building its first data connector, Fivetran has answered all of these questions, and many more. In doing so, we’ve proven that, from the perspective of user experience, the complexity and pain of data engineering are not inevitable — they’re not inherent attributes. Complexity can always be extinguished with smart, inventive engineering.

Perfecting the technology

“Because of the automated nature of our system,” George said at the Colorado retreat, “we have a unique opportunity to be the first data pipeline that can actually be used as a true piece of infrastructure, something customers can build vertical-specific tools on top of. In a couple years, we want to be in a place where you can stand in a room of analysts and say, ‘You probably use Fivetran data even if you don’t know it, because you’re using some other tool that was powered by Fivetran.’”

One prerequisite for under-the-hood invisibility, of course, is reliability.

In Colorado, VP of Engineering Meel Velliste made it clear that, going forward, Fivetran would focus relentlessly on reliability. “In our product context, quality and reliability are the same thing, and we still have a lot of road to go in terms of reliability,” he said. “Cloud service providers typically want to have 99.999% reliability, and we have to aim for the same standard. That may sound a little bit boring, but engineers thinking about what it actually takes to achieve those five 9s know what it means. It means that, as engineers, we have to pull off miracles. And that’s what makes it so exciting for engineers to work at Fivetran — you get to work miracles.”

If our origin story and engineering mission resonate with you, please check out our openings — we’re hiring!‍