Metadata Catalogs
Managed Data Lake Service lets you access your data lake tables through multiple catalog options. Fivetran always creates Apache Iceberg™ metadata for every table and provides a pre-configured Fivetran Catalog by default. You can optionally enable Delta Lake metadata creation and integrate additional catalogs to make your tables accessible through services that may already exist in your environment.
You can integrate the following catalogs with your data lake:
- AWS Glue for Iceberg tables in AWS data lakes
- BigLake metastore for Iceberg tables in GCS data lakes
- Databricks Unity Catalog for tables with Delta Lake metadata in AWS, Azure, and GCS data lakes
- OneLake for tables with Delta Lake metadata in Azure
These optional catalog integrations publish Fivetran-managed table metadata through familiar catalog services and metastores.
Fivetran manages all tables in your data lake. Do not modify these tables manually or by using any external catalog, as doing so may lead to data integrity issues.
Fivetran Catalog
Fivetran Catalog serves as the default catalog for the Iceberg metadata that Managed Data Lake Service creates for every table. Each Managed Data Lake Service destination has its own dedicated Fivetran Catalog, which we configure based on your destination details. We use Apache Polaris™ to implement Fivetran Catalog. The catalog is read-only for query engines, and we update it during every sync so that it reflects the latest table metadata. If an external catalog becomes inconsistent, Fivetran detects the issue during the next sync and republishes the latest Iceberg metadata from Fivetran Catalog. Fivetran Catalog serves as the definitive source of Iceberg metadata.
We use the Iceberg OpenAPI Specification to build the catalog, ensuring compatibility with any Iceberg REST catalog–compliant query engine for reading data from your data lake. For more information about integrating query engines with Fivetran Catalog, see our Integration Guide.
AWS Glue
You can integrate AWS Glue with an AWS data lake to publish the Iceberg metadata that Fivetran creates for your tables to AWS Glue Data Catalog. This integration enables access through AWS services such as Amazon EMR, Athena, and Redshift.
AWS Glue is an optional catalog integration and does not replace the pre-configured Fivetran Catalog. To configure the integration, you must create an IAM policy for AWS Glue Data Catalog. For more information, see our Setup Guide.
BigLake metastore
With Managed Data Lake Service, you can integrate BigLake metastore with your GCS data lake to publish the Iceberg metadata that Fivetran creates for your tables and query the tables using BigQuery. This integration is optional and does not replace Fivetran Catalog. To integrate BigLake metastore with your data lake, see our Setup Guide.
Databricks Unity Catalog
Managed Data Lake Service enables you to integrate Databricks Unity Catalog with your AWS, Azure, or GCS data lake. When Delta Lake metadata creation and Unity Catalog integration are enabled, Fivetran publishes the Delta Lake metadata to Unity Catalog, allowing you to organize, manage, and query the tables from the Databricks ecosystem.
To integrate Unity Catalog with your data lake, create an external location in Databricks and specify the storage credentials required to access the data lake. The external location must reference the same storage location that Managed Data Lake Service uses for the underlying table data. After you configure the external location, you can query the tables through Unity Catalog. For more information about integrating the catalog with your data lake, see our Setup Guide.
When filtering timestamp values from Unity Catalog, use the string(<col_name>) clause to get the accurate values.
OneLake
When Delta Lake metadata creation is enabled, you can integrate OneLake with your Azure data lake and query the tables using a SQL analytics endpoint. Fivetran publishes the Delta Lake metadata for the tables to OneLake. To integrate OneLake with your data lake, see our Setup Guide.
Legal notices
Apache Iceberg and Apache Polaris are trademarks of the Apache Software Foundation.