Azure Data Lake Storage Setup Guidelink
Follow our setup guide to connect Azure Data Lake Storage (ADLS) to Fivetran.
Prerequisiteslink
To connect a Azure Data Lake Storage to Fivetran, you need the following:
- An ADLS Gen2 account with Administrator permissions
- An ADLS Gen2 container
- Permissions to create an Azure service principal
Setup instructionslink
Create storage accountlink
Log in to the Azure portal.
Follow the instructions in Microsoft’s documentation to create a storage account.
NOTE:
- While creating the storage account, make sure you have selected the Enable hierarchical namespace checkbox in the Advanced tab of the Create storage account page.
- If you have a firewall enabled, create a firewall rule to allow access to Fivetran's IPs.
- If you have a firewall enabled and your Fivetran instance is configured to run in the same region as your Azure Storage Account, you need to configure virtual network rules and add Fivetran's internal virtual network subnets to the list of allowed virtual networks. Learn more in Microsoft's Azure Blob Storage documentation.
TIP: To retrieve the list of region-specific, fully qualified subnet IDs, contact our support team.
Create ADLS containerlink
Go to the storage account you created in Step 1.
In the navigation menu, go to Containers and click + Container.
In the New container pane, enter a Name for your container and make a note of it. You will need it to configure Fivetran.
In the Public access level drop-down menu, select an access level for the container.
Click Create.
Register an application and add a service principallink
In the navigation menu, select Microsoft Entra ID (formerly Azure Active Directory).
Go to App registrations and click + New registration.
Enter a Name for the application.
In the Supported account types section, select Accounts in this organizational directory only and click Register.
Make a note of the Application (client) ID and Directory (tenant) ID. You will need them to configure Fivetran.
Create client secretlink
Select the application you registered in Step 3.
In the navigation menu, go to Certificates & secrets and click + New client secret.
Enter a Description for your client secret.
In the Expires drop-down menu, select an expiry period for the client secret.
Click Add.
Make a note of the client secret. You will need it to configure Fivetran.
Assign role to containerlink
Go to the container you created in Step 2 and select Access Control (IAM).
Click Add, then select Add role assignments.
In the Role tab, select Storage Blob Data Contributor and click Next.
In the Member tab, select User, group, or service principal.
Click + Select members.
In the Select members pane, select the service principal you added in Step 3 and then click Select.
Click Review + assign.
(Optional) Connect using Azure Private Linklink
IMPORTANT: You must have a Business Critical plan to use Azure Private Link.
Azure Private Link allows Virtual Networks (VNets) and Azure-hosted or on-premises services to communicate with one another without exposing traffic to the public internet. Learn more in Microsoft's Azure Private Link documentation.
Prerequisiteslink
To set up Azure Private Link, you need a Fivetran instance configured to run in Azure.
Configure Azure Private Linklink
Go to the storage account you created in Step 1.
On the navigation menu, click Overview.
Make a note of the storage account's name, resource group, and subscription ID and provide these details to your Fivetran account manager.
NOTE: We set up a Private Link connection for both the
blob
anddfs
endpoints since Azure storage uses different private endpoints for different operations. For more information, see Microsoft's documentation to understand how Azure Storage uses private endpoints.Once your account manager confirms that the setup was successful, verify and approve both the endpoint connection requests from Fivetran. Fivetran then completes the Private Link setup for your Azure Data Lake Storage destination.
Complete Fivetran configuration link
Log in to your Fivetran account.
Go to the Destinations page and click Add destination.
Enter a Destination name of your choice and then click Add.
Select ADLS as the destination type.
In the destination setup form, enter the Storage Account Name.
Enter the Container Name you found in Step 2.
(Optional) Enter the Prefix Path of your ADLS container.
Enter the Tenant ID and Client ID you found in Step 3.
In the Secret Value field, enter the client secret you found in Step 4.
Choose your Data processing location.
Choose your Connection Method:
- Connect directly
- Connect via Private Link
NOTE: The Connect via Private Link option is available only for Business Critical accounts.
Choose your Cloud service provider and its region as described in our Destinations documentation.
Choose your Time zone.
(Optional for Business Critical accounts) To enable regional failover, set the Use Failover toggle to ON, and then select your Failover Location and Failover Region. Make a note of the IP addresses of the secondary region and safelist these addresses in your firewall.
Click Save & Test.
Fivetran tests and validates the Azure Data Lake Storage connection. On successful completion of the setup tests, you can sync your data using Fivetran connectors to the Azure Data Lake Storage destination.
In addition, Fivetran automatically configures a Fivetran Platform Connector to transfer the connector logs and account metadata to a schema in this destination. The Fivetran Platform Connector enables you to monitor your connectors, track your usage, and audit changes. The connector sends all these details at the destination level.
IMPORTANT: If you are an Account Administrator, you can manually add the Fivetran Platform Connector on an account level so that it syncs all the metadata and logs for all the destinations in your account to a single destination. If an account-level Fivetran Platform Connector is already configured in a destination in your Fivetran account, then we don't add destination-level Fivetran Platform Connectors to the new destinations you create.
Setup testslink
Fivetran performs the Read and Write Access test to check the accessibility of your ADLS Gen2 container and to validate the ADLS credentials you provided in the setup form.
The Private Link test validates if you have accurately configured the Private Link or approved the private endpoint connection requests from Fivetran. We perform this test only if you have opted to connect through Private Link.
NOTE: These tests may take a couple of minutes to complete.
Related articleslink
description Destination Overview
settings API Destination Configuration