Azure Data Lake Storage Setup Guide
Follow our setup guide to connect Azure Data Lake Storage (ADLS) to Fivetran.
Prerequisites
To connect a Azure Data Lake Storage to Fivetran, you need the following:
- An ADLS Gen2 account with Administrator permissions
- An ADLS Gen2 container
- Permissions to create an Azure service principal
Setup instructions
Create storage account
Log in to the Azure portal.
Follow the instructions in Microsoft’s documentation to create a storage account.
NOTE:
- While creating the storage account, make sure you have selected the Enable hierarchical namespace checkbox in the Advanced tab of the Create storage account page.
- If you have a firewall enabled and if your Fivetran instance is not configured to run in the same region as your Azure storage account, create a firewall rule to allow access to Fivetran's IPs.
- If you have a firewall enabled and if your Fivetran instance is configured to run in the same region as your Azure storage account, you must configure virtual network rules and add Fivetran's internal virtual private network subnets to the list of allowed virtual networks. For more information, see Microsoft's documentation.
- As you are adding a rule for a subnet in a virtual network that belongs to another Microsoft Entra tenant (Fivetran), you must use a fully qualified subnet ID. For more information, see Microsoft's documentation
- To retrieve the list of region-specific fully qualified subnet IDs, contact our support team.
Create ADLS container
Go to the storage account you created in Step 1.
In the navigation menu, go to Containers and click + Container.
In the New container pane, enter a Name for your container and make a note of it. You will need it to configure Fivetran.
In the Public access level drop-down menu, select an access level for the container.
Click Create.
NOTE: If any private endpoints exist in the Storage Account > Networking section, you must use Azure Private Link to connect Fivetran to the container.
Register an application and add a service principal
In the navigation menu, select Microsoft Entra ID (formerly Azure Active Directory).
Go to App registrations and click + New registration.
Enter a Name for the application.
In the Supported account types section, select Accounts in this organizational directory only and click Register.
Make a note of the Application (client) ID and Directory (tenant) ID. You will need them to configure Fivetran.
Create client secret
Select the application you registered in Step 3.
In the navigation menu, go to Certificates & secrets and click + New client secret.
Enter a Description for your client secret.
In the Expires drop-down menu, select an expiry period for the client secret.
Click Add.
Make a note of the client secret. You will need it to configure Fivetran.
Assign role to container
Go to the container you created in Step 2 and select Access Control (IAM).
Click Add, then select Add role assignments.
In the Role tab, select Storage Blob Data Contributor and click Next.
In the Member tab, select User, group, or service principal.
Click + Select members.
In the Select members pane, select the service principal you added in Step 3 and then click Select.
Click Review + assign.
(Optional) Connect using Azure Private Link
IMPORTANT: You must have a Business Critical plan to use Azure Private Link. If any private endpoints exist in the Storage Account > Networking section of your Azure portal, you must use Azure Private Link to connect Fivetran to the container. We strongly recommend that you create your private endpoints by following our Configure Azure Private Link instructions.
Azure Private Link allows Virtual Networks (VNets) and Azure-hosted or on-premises services to communicate with one another without exposing traffic to the public internet. Learn more in Microsoft's Azure Private Link documentation.
Prerequisites
To set up Azure Private Link, you need a Fivetran instance configured to run in Azure.
Configure Azure Private Link
Go to the storage account you created in Step 1.
On the navigation menu, click Overview.
Make a note of the storage account's name, resource group, and subscription ID and provide these details to your Fivetran account manager.
NOTE: We set up a Private Link connection for both the
blobanddfsendpoints since Azure storage uses different private endpoints for different operations. For more information, see Microsoft's documentation to understand how Azure Storage uses private endpoints.
Once your account manager confirms that the setup was successful, verify and approve both the endpoint connection requests from Fivetran. Fivetran then completes the Private Link setup for your Azure Data Lake Storage destination.
Complete Fivetran configuration
Log in to your Fivetran account.
Go to the Destinations page and click Add destination.
Enter a Destination name of your choice and then click Add.
Select ADLS as the destination type.
In the destination setup form, enter the Storage Account Name.
Enter the Container Name you found in Step 2.
(Optional) Enter the Prefix Path of your ADLS container.
Enter the Tenant ID and Client ID you found in Step 3.
In the Secret Value field, enter the client secret you found in Step 4.
In the Table Format drop-down menu, select the format you want to use for your destination tables.
NOTE: You cannot change the table format after you save the setup form.
In the Snapshot Retention Period drop-down menu, select how long you want us to retain your table snapshots.
NOTE: We perform regular table maintenance operations to delete the table snapshots that are older than the retention period you select in this field. You can select Retain All Snapshots to disable the deletion of table snapshots.
Choose your Data processing location.
Choose your Connection Method:
- Connect directly
- Connect via Private Link
NOTE: The Connect via Private Link option is available only for Business Critical accounts.
Choose your Cloud service provider and its region as described in our Destinations documentation.
Choose your Time zone.
(Optional for Business Critical accounts) To enable regional failover, set the Use Failover toggle to ON, and then select your Failover Location and Failover Region. Make a note of the IP addresses of the secondary region and safelist these addresses in your firewall.
Click Save & Test.
Fivetran tests and validates the Azure Data Lake Storage connection. On successful completion of the setup tests, you can sync your data using Fivetran connectors to the Azure Data Lake Storage destination.
In addition, Fivetran automatically configures a Fivetran Platform Connector to transfer the connector logs and account metadata to a schema in this destination. The Fivetran Platform Connector enables you to monitor your connectors, track your usage, and audit changes. The connector sends all these details at the destination level.
IMPORTANT: If you are an Account Administrator, you can manually add the Fivetran Platform Connector on an account level so that it syncs all the metadata and logs for all the destinations in your account to a single destination. If an account-level Fivetran Platform Connector is already configured in a destination in your Fivetran account, then we don't add destination-level Fivetran Platform Connectors to the new destinations you create.
Setup tests
Fivetran performs the Read and Write Access test to check the accessibility of your ADLS Gen2 container and to validate the ADLS credentials you provided in the setup form.
The Private Link test validates if you have accurately configured the Private Link or approved the private endpoint connection requests from Fivetran. We perform this test only if you have opted to connect through Private Link.
NOTE: These tests may take a couple of minutes to complete.
Related articles
description Destination Overview
settings API Destination Configuration