Automate data lake management with Fivetran

Our approach of moving and landing data in a data lake uniquely provides customers with data management features reminiscent of data warehouses. Automated conversion to Apache Iceberg and Delta Lake table formats revolutionizes data lakes, turning them from chaotic repositories into structured, well-managed data stores. This transformation streamlines operations, enhances governance, and increases usability, marking a significant step forward in leveraging data effectively for business insights and innovation. Learn more about Fivetran Managed Data Lake Service.

Data lakes explicitly separate the storage layer of the data warehouse into a separate, vendor-neutral format in object storage to create a universal storage layer so that you only move your data once but can query as needed from the best in breed services for your workload — this means multiple, specialized execution engines can interact with the same data lake, mediated by a catalog that provides transactions and fine-grained permissions. Learn more about how modern data lakes lower TCO.

Yes! Customers on the Business Critical plan have the option to connect to their data lake on a private network path that keeps traffic off the public internet. Object storage on each cloud is accessed via public endpoints, but routing of traffic is secured on the cloud provider’s internal backbone. The bucket policies restrict access to only the right processing region. We offer support for the default private networking mechanisms from each major cloud provider: AWS Gateway Endpoints, Azure Virtual Network Service Endpoints, and Google Cloud’s Private Google Access. Learn more about private networking for data lakes.

Yes! Fivetran Managed Data Lake Service includes a self-hosted Iceberg REST catalog (Apache Polaris) by default to ensure a seamless and fully functional out-of-the-box experience for managing Apache Iceberg tables, which require a catalog to manage ACID transactions. You also have the option to bring your own external catalog like AWS Glue, Databricks Unity Catalog, BigQuery Metastore, or OneLake, and use the Iceberg REST catalog as a fallback and for advanced metadata management scenarios. Learn more about catalog support for data lakes.

Fivetran Managed Data Lake Service converts source data into Parquet files in the pipeline and stores them in designated tables in your data lake. It’s best to use the table metadata to read or query your data — which is available in both Iceberg and Delta Lake table standards in your catalog. Learn how to query data from your data lake.