Azure Data Lake Storage Setup Guide

Follow our setup guide to connect Azure Data Lake Storage (ADLS) to Fivetran.

Prerequisites

To connect a Azure Data Lake Storage to Fivetran, you need the following:

• An ADLS Gen2 account with Administrator permissions
• An ADLS Gen2 container
• Permissions to create an Azure service principal

Setup instructions

Create storage account

1. Log in to the Azure portal.

2. Follow the instructions in Microsoft’s documentation to create a storage account.

   NOTE:

   • While creating the storage account, make sure you have selected the Enable hierarchical namespace checkbox in the Advanced tab of the Create storage account page.
   • If you have a firewall enabled and if your Fivetran instance is not configured to run in the same region as your Azure storage account, create a firewall rule to allow access to Fivetran's IPs.
   • If you have a firewall enabled and if your Fivetran instance is configured to run in the same region as your Azure storage account, you must configure virtual network rules and add Fivetran's internal virtual private network subnets to the list of allowed virtual networks. For more information, see Microsoft's documentation.
   • As you are adding a rule for a subnet in a virtual network that belongs to another Microsoft Entra tenant (Fivetran), you must use a fully qualified subnet ID. For more information, see Microsoft's documentation
   • To retrieve the list of region-specific fully qualified subnet IDs, contact our support team.

Create ADLS container

1. Go to the storage account you created in Step 1.

2. In the navigation menu, go to Containers and click + Container.

3. In the New container pane, enter a Name for your container and make a note of it. You will need it to configure Fivetran.

4. In the Public access level drop-down menu, select an access level for the container.

5. Click Create.

   

NOTE: If any private endpoints exist in the Storage Account > Networking section, you must use Azure Private Link to connect Fivetran to the container.

Register an application and add a service principal

1. In the navigation menu, select Microsoft Entra ID (formerly Azure Active Directory).

2. Go to App registrations and click + New registration.

   

3. Enter a Name for the application.

   

4. In the Supported account types section, select Accounts in this organizational directory only and click Register.

5. Make a note of the Application (client) ID and Directory (tenant) ID. You will need them to configure Fivetran.

   

Create client secret

1. Select the application you registered in Step 3.

2. In the navigation menu, go to Certificates & secrets and click + New client secret.

   

3. Enter a Description for your client secret.

4. In the Expires drop-down menu, select an expiry period for the client secret.

5. Click Add.

6. Make a note of the client secret. You will need it to configure Fivetran.

   

Assign role to container

1. Go to the container you created in Step 2 and select Access Control (IAM).

   

2. Click Add, then select Add role assignments.

   

3. In the Role tab, select Storage Blob Data Contributor and click Next.

4. In the Member tab, select User, group, or service principal.

5. Click + Select members.

6. In the Select members pane, select the service principal you added in Step 3 and then click Select.

7. Click Review + assign.

(Optional) Connect using Azure Private Link

IMPORTANT: You must have a Business Critical plan to use Azure Private Link. If any private endpoints exist in the Storage Account > Networking section of your Azure portal, you must use Azure Private Link to connect Fivetran to the container. We strongly recommend that you create your private endpoints by following our Configure Azure Private Link instructions.

Azure Private Link allows Virtual Networks (VNets) and Azure-hosted or on-premises services to communicate with one another without exposing traffic to the public internet. Learn more in Microsoft's Azure Private Link documentation.

Prerequisites

To set up Azure Private Link, you need a Fivetran instance configured to run in Azure.

Configure Azure Private Link

1. Go to the storage account you created in Step 1.

2. On the navigation menu, click Overview.

3. Make a note of the storage account's name, resource group, and subscription ID and provide these details to your Fivetran account manager.

   NOTE: We set up a Private Link connection for both the blob and dfs endpoints since Azure storage uses different private endpoints for different operations. For more information, see Microsoft's documentation to understand how Azure Storage uses private endpoints.

   

4. Once your account manager confirms that the setup was successful, verify and approve both the endpoint connection requests from Fivetran. Fivetran then completes the Private Link setup for your Azure Data Lake Storage destination.

   

(Optional) Set up automatic schema migration for Delta Lake tables in Databricks

Complete Fivetran configuration

1. Log in to your Fivetran account.

2. Go to the Destinations page and click Add destination.

3. Enter a Destination name of your choice and then click Add.

4. Select ADLS as the destination type.

5. In the destination setup form, enter the Storage Account Name.

   NOTE: You cannot change the Storage Account Name after you save the setup form.

6. Enter the Container Name you found in Step 2.

   NOTE: You cannot change the Container Name after you save the setup form.

7. (Optional) Enter the Prefix Path of your ADLS container.

   NOTE: You cannot change the Prefix Path after you save the setup form.

8. Enter the Tenant ID and Client ID you found in Step 3.

9. In the Secret Value field, enter the client secret you found in Step 4.

10. In the Table Format drop-down menu, select the format you want to use for your destination tables.

    NOTE: You cannot change the table format after you save the setup form.

11. In the Snapshot Retention Period drop-down menu, select how long you want us to retain your table snapshots.

    NOTE: We perform regular table maintenance operations to delete the table snapshots that are older than the retention period you select in this field. You can select Retain All Snapshots to disable the deletion of table snapshots.

12. (Optional) To automate schema migration of Delta Lake tables in Databricks, set the Maintain Delta Tables In Databricks toggle to ON and do the following:

    i. Choose the Databricks Connection Method.

    ii. Enter the following details of your Databricks account:

    • Catalog name
    • Server Hostname
    • Port number
    • HTTP Path

    iii. Select the Authentication Type you configured in Step 7.6.

    iv. If you selected PERSONAL ACCESS TOKEN as the Authentication Type, enter the Personal Access Token you created in Step 7.6.

    v. If you selected OAUTH 2.0 as the Authentication Type, enter the OAuth 2.0 Client ID and OAuth 2.0 Secret you created in Step 7.6.

13. Choose your Data processing location.

14. Choose your Connection Method:

    • Connect directly
    • Connect via Private Link

      NOTE: The Connect via Private Link option is available only for Business Critical accounts.

15. Choose your Cloud service provider and its region as described in our Destinations documentation.

16. Choose your Time zone.

17. (Optional for Business Critical accounts) To enable regional failover, set the Use Failover toggle to ON, and then select your Failover Location and Failover Region. Make a note of the IP addresses of the secondary region and safelist these addresses in your firewall.

18. Click Save & Test.

Fivetran tests and validates the Azure Data Lake Storage connection. On successful completion of the setup tests, you can sync your data using Fivetran connectors to the Azure Data Lake Storage destination.

In addition, Fivetran automatically configures a Fivetran Platform Connector to transfer the connector logs and account metadata to a schema in this destination. The Fivetran Platform Connector enables you to monitor your connectors, track your usage, and audit changes. The connector sends all these details at the destination level.

IMPORTANT: If you are an Account Administrator, you can manually add the Fivetran Platform Connector on an account level so that it syncs all the metadata and logs for all the destinations in your account to a single destination. If an account-level Fivetran Platform Connector is already configured in a destination in your Fivetran account, then we don't add destination-level Fivetran Platform Connectors to the new destinations you create.

Setup tests

• Fivetran performs the Read and Write Access test to check the accessibility of your ADLS Gen2 container and to validate the ADLS credentials you provided in the setup form.

• The Private Link test validates if you have accurately configured the Private Link or approved the private endpoint connection requests from Fivetran. We perform this test only if you have opted to connect through Private Link.

• Fivetran performs the Validate Permissions test checks on the databricks creds if we have the necessary READ/WRITE permissions to CREATE, ALTER, or DROP tables in the database. We perform this test only if you have opted to automate schema migration of delta tables in databricks.

NOTE: These tests may take a couple of minutes to complete.

Related articles

description Destination Overview

assignment Release Notes

settings API Destination Configuration

home Documentation Home