Fully managed ELT allows data integration to be outsourced and automated, saving precious engineering time.
ELT stands for Extract, Load, and Transform. It is commonly contrasted with ETL, which is fundamentally the same process in a different sequence. ETL was invented in the 1970s to address data integration needs and is still commonly used today.
Analysts depend on transformations to produce data models to support dashboards and reporting. Transformations include all operations that alter or create values, such as cleaning, summarizing, pivoting, and combining data. These operations can be complicated, and usually obscure or alter the raw values from the source.
The critical weakness of ETL is that data is transformed before it is loaded. This means not only that original raw values are obscured, but also that the entire pipeline must be rebuilt every time data models change either at the source or destination.
By contrast, the new ELT process extracts and loads all of your raw data with no assumptions about how the data will be used. It also shifts transformation from an engineering-centric activity baked into data pipelines to an analyst-centric activity performed in a data warehouse.
This means that changes to downstream business and analytics requirements, i.e. data models for dashboards and reporting, don’t affect the pipeline. With the proper use of outsourcing and automation, your data pipelines can also continuously adapt to changes at the source so that your analysts will always have fresh data to work with.
As ELT is a novel approach to data integration, we will answer some common questions about its capabilities.
Fully managed ELT tools allow organizations to outsource data integration. This automates many typical pipeline maintenance tasks including, but not limited to:
Modifying data extraction scripts to accommodate frequent API changes
Normalizing data source extracts into a queryable schema
Keeping that schema updated with new data or new structures as they are created in the data source
The benefit of choosing a provider with fully managed ELT is that many of these technical barriers are eliminated so that your analysts reliably and rapidly access their data.
With fully managed ELT, you have all of your data from your various sources centralized in one place and continuously updated without human intervention. This persistent, up-to-date data repository gives your teams the bandwidth to tackle new questions as they emerge, accelerate project timelines, and create new insights from existing datasets. Fully managed ELT takes this another step further by updating incrementally, resulting in faster turnaround and less impact to production source systems.
We have previously discussed the differences between ETL and ELT in more detail.
Providers of purpose-built, fully managed ELT have taken pains to understand every idiosyncrasy of the data sources they connect to, and regularly pressure-test their software against a variety of use cases. The design considerations involved in developing a robust ELT solution are not trivial, and the cost in time, money, labor, and morale is substantial.
Outsourcing your data integration allows you to rapidly narrow time-to-insight and saves you valuable engineering resources both for the initial build and ongoing maintenance. Automated ELT is accessible at any level of technical ability, empowering your analysts to work with minimal delay.
With Fivetran handling ELT, the company's engineers can focus on data science and machine learning projects.Learn More
No! Although ELT is a cloud-native approach to data integration, organizations with on-premise tools still enjoy the benefits of a system that continually keeps data updated, enables their monitoring dashboards and helps their data teams keep up with changing business processes.
All it takes to set up integrations is to provide or obtain a user account(s) with the appropriate access levels. This is typically a one step process. From there, automated data connectors will do a full historical load of all the data available in the data source, and will keep that dataset updated as changes are read from the source. Integrations can be set up in minutes and you’ll never have to check back on the pipeline as your data source evolves.
If your organization is ready to consider fully managed ELT, Fivetran is happy to help.