Managed Data Lakes Service Setup Guide

Prerequisites

To connect your AWS data lake to Fivetran, you need the following:

• An AWS account with a single resource group per AWS Region.
• An Amazon S3 bucket in one of the supported AWS Regions. For optimal performance, we recommend creating the bucket in the same Region as your data lake's data processing location to improve upload and download speeds. For more information about creating an Amazon S3 bucket, see AWS documentation.

Find External ID

1. Log in to your Fivetran account.
2. Go to the Destinations page and click Add destination.
3. Enter a Destination name of your choice and then click Add.
4. Select Managed Data Lakes Service as the destination type.
5. Make a note of the value in the External ID field. This is an automatically-generated ID that is unique to your Fivetran account and remains unchanged, even if you close and re-open the setup form. You will need the ID to create an IAM role for Fivetran.

NOTE: To streamline the setup process, we recommend keeping the browser tab open while configuring the destination.

Create IAM policy for S3 bucket

1. Log in to the Amazon IAM console.

2. Go to Policies and click Create Policy.

3. Go to the JSON tab.

4. Copy the following policy and paste it in the JSON editor.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
           "Sid": "AllowListBucketOfASpecificPrefix",
           "Effect": "Allow",
           "Action": [
               "s3:ListBucket"
           ],
           "Resource": [
               "arn:aws:s3:::{your-bucket-name}"
           ],
           "Condition": {
               "StringLike": {
                   "s3:prefix": [
                       "{prefix_path}/*"
                    ]
                }
             }
           },
           {
           "Sid": "AllowAllObjectActionsInSpecificPrefix",
           "Effect": "Allow",
           "Action": [
               "s3:DeleteObjectTagging",
               "s3:ReplicateObject",
               "s3:PutObject",
               "s3:GetObjectAcl",
               "s3:GetObject",
               "s3:DeleteObjectVersion",
               "s3:PutObjectTagging",
               "s3:DeleteObject",
               "s3:PutObjectAcl"
           ],
           "Resource": [
               "arn:aws:s3:::{your-bucket-name}/{prefix_path}/*"
           ]
           }
       ]
   }
   
   

5. In the policy you pasted in the JSON editor, do the following:

   • Replace all instances of {your-bucket-name} with the name of your S3 bucket
   • Replace all instances of {prefix_path} with the prefix path where you want to load your data

     IMPORTANT:

     • If you do not specify a prefix path, the policy will grant Fivetran access to the entire S3 bucket instead of limiting our access to the objects of a specific prefix path.
     • Setting the s3:prefix condition to ["*"] grants Fivetran access to all prefixes within the specified bucket. To restrict access to a specific prefix path, set it to ["{prefix_path}/*"].

6. Click Next.

7. Enter the Policy name.

8. Click Create policy.

(Optional) Set up catalogs

IMPORTANT: Perform this step only if you want to integrate AWS Glue or Databricks Unity Catalog to manage your Iceberg and Delta Lake tables, respectively. By default, we use Fivetran Iceberg REST Catalog to manage your Iceberg tables. For more information about catalogs, see our Managed Data Lakes Service overview documentation.

CREATE TABLE <catalog>.<schema>.<table>
USING delta
OPTIONS (
path 's3://<bucketName>/<path-to-table>'
)

Create IAM role

1. Go to Roles and click Create role.

2. Choose AWS account as the Trusted entity type and select Another AWS account.

   

3. In the Account ID field, enter Fivetran's account ID, 834469178297.

4. Select the Require external ID checkbox, and then enter the External ID you found.

5. Click Next.

6. Select the checkbox for the IAM policy you created for your S3 bucket.

7. If you configured AWS Glue Data Catalog for your Iceberg tables, select the IAM policy you created for the Glue catalog.

8. Click Next.

9. Enter the Role name and click Create role.

10. In the Roles page, select the role you created.

11. Make a note of the ARN. You will need it to configure Fivetran.

(Optional) Configure IAM policy for AWS Lake Formation

IMPORTANT: This step is mandatory if AWS Lake Formation is enabled for your S3 bucket. You can skip this step if AWS Lake Formation is not enabled for your bucket.

1. Go to the AWS Lake Formation console.

2. On the navigation menu, go to Permissions > Data locations.

3. Click Grant.

4. Choose My account.

   

5. In the IAM users and roles drop-down menu, select the IAM role you created for your bucket.

6. In the Storage locations field, enter the prefix of your S3 bucket where you want to store your data.

7. Click Grant.

(Optional) Configure AWS PrivateLink

IMPORTANT: You must have a Business Critical plan to use AWS PrivateLink.

AWS PrivateLink allows VPCs and AWS-hosted or on-premises services to communicate with one another without exposing traffic to the public internet. PrivateLink is the most secure connection method. Learn more in AWS PrivateLink documentation.

Follow our AWS PrivateLink setup instructions to configure PrivateLink for your S3 bucket.

Complete Fivetran configuration

1. In the Fivetran dashboard, go to the Managed Data Lakes Service setup form.

2. In the Storage Provider drop-down menu, select AWS.

3. Enter your S3 Bucket name.

   NOTE: You cannot change the bucket name after you save the setup form.

4. In the Fivetran Role ARN field, enter the ARN you found.

5. (Optional) Enter the S3 Prefix Path of your bucket. The prefix path must not start or end with a forward slash (/).

   NOTE: You cannot change the prefix path after you save the setup form.

6. Enter your S3 Bucket Region.

7. If you want Fivetran to always use AWS PrivateLink to connect to your S3 bucket, set the Require PrivateLink toggle to ON.

   NOTE: By default, Fivetran uses AWS PrivateLink if your S3 bucket's AWS Region matches your data processing location. However, if you set this toggle to OFF and and your S3 bucket is in a different region than your data processing location, Fivetran will not use PrivateLink and will skip the PrivateLink setup test.

8. (Optional) To integrate AWS Glue with your data lake, set the Update AWS Glue Catalog toggle to ON.

9. (Optional) To enable automatic schema migration of Delta Lake tables to Unity Catalog, set the Update Databricks Unity Catalog toggle to ON and do the following:

   i. Choose the Databricks Connection Method.

   ii. Enter the following details of your Databricks account:

   • Catalog name
   • Server Hostname
   • Port number
   • HTTP Path

   iii. Select the Authentication Type you configured.

   iv. If you selected PERSONAL ACCESS TOKEN as the Authentication Type, enter the Personal Access Token you created.

   v. If you selected OAUTH 2.0 as the Authentication Type, enter the OAuth 2.0 Client ID and OAuth 2.0 Secret you created.

10. In the Snapshot Retention Period drop-down menu, select how long you want us to retain your table snapshots.

    NOTE: We perform regular table maintenance operations to delete the table snapshots that are older than the retention period you select in this field. You can select Retain All Snapshots to disable the deletion of table snapshots.

11. Choose your Data processing location.

12. Choose your Cloud service provider and its region as described in our Destinations documentation.

    NOTE: For faster uploads, downloads, and optimal load times, we recommend selecting AWS as the Cloud Service Provider and choosing the AWS Region where your S3 bucket is located.

13. Choose your Time zone.

14. Click Save & Test.

Fivetran tests and validates the data lake connection. On successful completion of the setup tests, you can sync your data using Fivetran connectors to your AWS data lake.

In addition, Fivetran automatically configures a Fivetran Platform Connector to transfer the connection logs and account metadata to a schema in this data lake. The Fivetran Platform Connector enables you to monitor your connections, track your usage, and audit changes. The Fivetran Platform Connector sends all these details at the destination level.

IMPORTANT: If you are an Account Administrator, you can manually add the Fivetran Platform Connector on an account level so that it syncs all the metadata and logs for all the destinations in your account to a single destination. If an account-level Fivetran Platform Connector is already configured in a destination in your Fivetran account, then we don't add destination-level Fivetran Platform Connectors to the new destinations you create.

Setup tests

Fivetran performs the following AWS data lake connection tests:

• The S3 Read and Write Access test checks access to your S3 bucket and validates the resources specified in your IAM policy.
• The Glue Access test checks access to AWS Glue Data Catalog and validates the IAM policy resources. This test runs only if you set the Update AWS Glue Catalog toggle to ON.
• The PrivateLink test verifies that your S3 bucket is in the same AWS Region as Fivetran. This test runs only if you set the Require PrivateLink toggle to ON.
• The Bucket Region test Ensures that the specified S3 Bucket Region is valid.
• The Validate Permissions test checks if the Databricks credentials have the necessary permissions to CREATE, ALTER, or DROP tables in the database. This test runs only if you set the Require PrivateLink toggle is ON.

NOTE: The tests may take a couple of minutes to complete.

Prerequisites

To connect ADLS to Fivetran, you need the following:

• An ADLS Gen2 account with Administrator permissions
• An ADLS Gen2 container
• Permissions to create an Azure service principal

Create storage account

1. Log in to the Azure portal.

2. Create a storage account by following the instructions in Microsoft documentation.

   NOTE:

   • When creating the storage account, ensure that the Enable hierarchical namespace is selected in the Advanced tab of the Create storage account page.
   • If you have a firewall enabled and your Fivetran instance is not in the same region as your Azure storage account, create a firewall rule to allow access to Fivetran's IPs.
   • If you have a firewall enabled and your Fivetran instance is in the same region as your Azure storage account, configure virtual network rules and add Fivetran’s internal virtual private network subnets to the list of allowed virtual networks. For more information, see Microsoft documentation.
   • Since the subnet in the virtual network belongs to another Microsoft Entra tenant (Fivetran), you must use a fully qualified subnet ID. For more information, see Microsoft documentation
   • To obtain the list of region-specific fully qualified subnet IDs, contact our support team.

Create ADLS container

1. Go to the storage account you created.

2. On the navigation menu, go to Containers and click + Container.

3. In the New container pane, enter a Name for your container and make a note of it. You will need it to configure Fivetran.

4. In the Public access level drop-down menu, select an access level for the container.

5. Click Create.

   

Register an application and add a service principal

1. On the navigation menu, select Microsoft Entra ID (formerly Azure Active Directory).

2. Go to App registrations and click + New registration.

   

3. Enter a Name for the application.

   

4. In the Supported account types section, select Accounts in this organizational directory only and click Register.

5. Make a note of the Application (client) ID and Directory (tenant) ID. You will need them to configure Fivetran.

   

Create client secret

1. Select the application you registered.

2. On the navigation menu, go to Certificates & secrets and click + New client secret.

   

3. Enter a Description for your client secret.

4. In the Expires drop-down menu, select an expiry period for the client secret.

5. Click Add.

6. Make a note of the client secret. You will need it to configure Fivetran.

   

Assign role to container

1. Go to the container you created and select Access Control (IAM).

   

2. Click Add and then select Add role assignments.

   

3. In the Role tab, select Storage Blob Data Contributor and click Next.

4. In the Member tab, select User, group, or service principal.

5. Click + Select members.

6. In the Select members pane, select the service principal you added and then click Select.

7. Click Review + assign.

(Optional) Set up Unity Catalog

IMPORTANT: Perform this step only if you want to integrate Databricks Unity Catalog to manage your Delta Lake tables. By default, we use Fivetran Iceberg REST Catalog to manage your Iceberg tables. For more information about catalogs, see our Managed Data Lakes Service overview documentation.

To create and manage your Delta Lake tables in Unity Catalog, you have two options:

• Manual table management: You create and manage the tables manually in Databricks
• Automatic schema migration by Fivetran (recommended): Fivetran automatically creates and maintains your tables in Databricks without affecting sync performance

Expand one of the following sections based on how you want to manage your tables in Databricks:

CREATE TABLE <catalog>.<schema>.<table>
USING delta
OPTIONS (
path 'abfss://<containerName>@<storageAccountName>.dfs.core.windows.net/<path-to-table>'
)

(Optional) Connect using Azure Private Link

IMPORTANT: You must have a Business Critical plan to use Azure Private Link. If any private endpoints exist in the Storage Account > Networking section of your Azure portal, you must use Azure Private Link to connect Fivetran to the container. We strongly recommend that you create your private endpoints by following our Configure Azure Private Link instructions.

Azure Private Link allows Virtual Networks (VNets) and Azure-hosted or on-premises services to communicate with one another without exposing traffic to the public internet. Learn more in Microsoft's Azure Private Link documentation.

Prerequisites

To set up Azure Private Link, you need a Fivetran instance configured to run in Azure.

Configure Azure Private Link

1. Go to the storage account you created.

2. On the navigation menu, click Overview.

3. Make a note of the storage account's name, resource group, and subscription ID and provide these details to your Fivetran account manager.

   NOTE: We set up a Private Link connection for both the blob and dfs endpoints since Azure storage uses different private endpoints for different operations. For more information, see Microsoft's documentation to understand how Azure Storage uses private endpoints.

   

4. Once your account manager confirms that the setup was successful, verify and approve both the endpoint connection requests from Fivetran. Fivetran then completes the Private Link setup for your data lake.

   

Complete Fivetran configuration

1. In the Fivetran dashboard, go to the Managed Data Lakes Service setup form.

2. In the Storage Provider drop-down menu, select Azure Data Lake Storage.

3. Enter the Storage Account Name.

   NOTE: You cannot change the storage account name after you save the setup form.

4. Enter the Container Name you found.

   NOTE: You cannot change the container name after you save the setup form.

5. (Optional) Enter the Prefix Path of your ADLS container.

   NOTE: You cannot change the prefix path after you save the setup form.

6. Enter the Tenant ID and Client ID you found.

7. In the Secret Value field, enter the client secret you found.

8. In the Consent URL section, do the following:

   i. Right-click the hyperlink and open it in a new tab.

   

   ii. In the new browser tab, modify the URL to replace tenant_id with the ID you entered in the Tenant ID field.

   

   iii. Click Accept.

9. Choose your Connection Method:

   • Connect directly
   • Connect via Private Link

     NOTE: The Connect via Private Link option is available only for Business Critical accounts.

10. (Optional) To enable automatic schema migration of Delta Lake tables to Unity Catalog, set the Update Databricks Unity Catalog toggle to ON and do the following:

    i. Choose the Databricks Connection Method.

    ii. Enter the following details of your Databricks account:

    • Catalog name
    • Server Hostname
    • Port number
    • HTTP Path

    iii. Select the Authentication Type you configured.

    iv. If you selected PERSONAL ACCESS TOKEN as the Authentication Type, enter the Personal Access Token you created.

    v. If you selected OAUTH 2.0 as the Authentication Type, enter the OAuth 2.0 Client ID and OAuth 2.0 Secret you created.

11. In the Snapshot Retention Period drop-down menu, select how long you want us to retain your table snapshots.

    NOTE: We perform regular table maintenance operations to delete the table snapshots that are older than the retention period you select in this field. You can select Retain All Snapshots to disable the deletion of table snapshots.

12. Choose your Data processing location.

13. Choose your Cloud service provider and its region as described in our Destinations documentation.

14. Choose your Time zone.

15. Click Save & Test.

Fivetran tests and validates the data lake connection. On successful completion of the setup tests, you can sync your data using Fivetran connectors to your data lake.

In addition, Fivetran automatically configures a Fivetran Platform Connector to transfer the connection logs and account metadata to a schema in this data lake. The Fivetran Platform Connector enables you to monitor your connections, track your usage, and audit changes. The Fivetran Platform Connector sends all these details at the destination level.

IMPORTANT: If you are an Account Administrator, you can manually add the Fivetran Platform Connector on an account level so that it syncs all the metadata and logs for all the destinations in your account to a single destination. If an account-level Fivetran Platform Connector is already configured in a destination in your Fivetran account, then we don't add destination-level Fivetran Platform Connectors to the new destinations you create.

Setup tests

Fivetran performs the following ADLS connection tests:

• The Read and Write Access test verifies access to your ADLS Gen2 container and validates the ADLS credentials provided in the setup form.
• The Validate Permissions test Checks whether the Databricks credentials have the necessary READ/WRITE permissions to CREATE, ALTER, or DROP tables in the database. This test runs only if you opt to automate schema migration for Delta Lake tables in Databricks.