Metadata Catalogs
Managed Data Lake Service lets you access your data lake tables through multiple catalog options. Fivetran always creates Apache Iceberg™ metadata for every table and provides a pre-configured Fivetran Catalog by default. You can optionally enable Delta Lake metadata creation and integrate additional catalogs to make your tables accessible through services that may already exist in your environment.
You can integrate the following catalogs with your data lake:
- AWS Glue for Iceberg tables in AWS data lakes
- BigLake metastore for Iceberg tables in GCS data lakes
- Databricks Unity Catalog for tables with Delta Lake metadata in AWS, Azure, and GCS data lakes
- OneLake for tables with Delta Lake metadata in Azure
These optional catalog integrations publish Fivetran-managed table metadata through familiar catalog services and metastores.
Fivetran manages all tables in your data lake. Do not modify these tables manually or by using any external catalog, as doing so may lead to data integrity issues.
Fivetran Catalog
Fivetran Catalog is the default catalog for the Apache Iceberg™ metadata that Managed Data Lake Service creates for every table. We create a dedicated Fivetran Catalog for each Managed Data Lake Service destination and configure it using your destination details. Fivetran Catalog stores and serves the Iceberg metadata that query engines use to discover and query your Fivetran-managed tables.
We implement Fivetran Catalog by using Apache Polaris™ and update it during every sync to reflect the latest table metadata. Fivetran Catalog implements the Iceberg OpenAPI Specification, allowing any compatible Iceberg REST client to connect to it.
If an optional external catalog becomes inconsistent with Fivetran Catalog, we detect the inconsistency during the next sync and republish the latest Iceberg metadata to the external catalog.
You can connect supported engines to Fivetran Catalog by using one of the following credential types:
- Read-only credentials let you query data without modifying catalog metadata or table data.
- Write credentials let supported execution engines create, update, and delete schemas and tables, and modify table data.
By default, Catalog Integration > Base configuration on your destination details page provides read-only credentials for query engines. If you are an Account Administrator or Destination Administrator, you can access write credentials from the Write Credentials tab.
With write credentials, you can:
- Create, update, and delete schemas and tables
- Read and modify table data by using insert, update, delete, and merge operations
To connect a supported query engine to Fivetran Catalog, see our Fivetran Catalog Integration Guide.
Use write credentials carefully. Modifying Fivetran-delivered tables directly can corrupt the tables and disrupt downstream workflows.
For more information about using write credentials and understanding which operations are safe for Fivetran-delivered tables, see our Write Credentials documentation.
AWS Glue
You can integrate AWS Glue with an AWS data lake to publish the Iceberg metadata that Fivetran creates for your tables to AWS Glue Data Catalog. This integration enables access through AWS services such as Amazon EMR, Athena, and Redshift.
AWS Glue is an optional catalog integration and does not replace the pre-configured Fivetran Catalog. To configure the integration, you must create an IAM policy for AWS Glue Data Catalog. For more information, see our Setup Guide.
BigLake metastore
With Managed Data Lake Service, you can integrate BigLake metastore with your GCS data lake to publish the Iceberg metadata that Fivetran creates for your tables and query the tables using BigQuery. This integration is optional and does not replace Fivetran Catalog. To integrate BigLake metastore with your data lake, see our Setup Guide.
Databricks Unity Catalog
Managed Data Lake Service enables you to integrate Databricks Unity Catalog with your AWS, Azure, or GCS data lake. When Delta Lake metadata creation and Unity Catalog integration are enabled, Fivetran publishes the Delta Lake metadata to Unity Catalog, allowing you to organize, manage, and query the tables from the Databricks ecosystem.
To integrate Unity Catalog with your data lake, create an external location in Databricks and specify the storage credentials required to access the data lake. The external location must reference the same storage location that Managed Data Lake Service uses for the underlying table data. After you configure the external location, you can query the tables through Unity Catalog. For more information about integrating the catalog with your data lake, see our Setup Guide.
When filtering timestamp values from Unity Catalog, use the string(<col_name>) clause to get the accurate values.
OneLake
When Delta Lake metadata creation is enabled, you can integrate OneLake with your Azure data lake and query the tables using a SQL analytics endpoint. Fivetran publishes the Delta Lake metadata for the tables to OneLake. To integrate OneLake with your data lake, see our Setup Guide.
Legal notices
Apache Iceberg and Apache Polaris are trademarks of the Apache Software Foundation.