Databricks Workspace Can't Access Azure Data Lake Storage After Enabling VNet Injection
Issue
After enabling Azure Databricks VNet injection, Databricks can't access files or tables in your Azure Data Lake Storage (ADLS) account. Queries fail with network or access errors, or ADLS container folders don't appear in the Databricks workspace.
Fivetran syncs complete successfully because we write data directly to ADLS.
Environment
- Destination: Databricks
- Storage backend: Azure Data Lake Storage Gen2
- Networking: Azure VNet injection
Resolution
To resolve this issue, allow the Databricks-injected virtual network to access your ADLS storage account:
- Log in to the Azure portal.
- Go to the ADLS storage account used by your Databricks destination.
- In the left menu, go to Security + networking, then click Networking.
- Select the Firewalls and virtual networks tab.
- Select Enabled from selected virtual networks and IP addresses.
- In the Virtual networks section, click + Add existing virtual network.
- Select the Subscription, Virtual network, and Subnet used by your Databricks workspace for VNet injection.
- Click Add.
- Click Save.
After saving, verify that Databricks can access the ADLS storage account. You don't need to re-sync your data.
Cause
This issue occurs when you deploy Azure Databricks using VNet injection, which places the Databricks workspace in your own Azure virtual network, and the ADLS storage account has network access restrictions enabled.
Enabling VNet injection in Azure Databricks places the Databricks cluster nodes into your own Azure VNet. However, this doesn't automatically grant that VNet access to other Azure resources, such as ADLS storage accounts. If the storage account has a network firewall configured, you must explicitly grant access to each virtual network that needs to read from or write to the account.
We sync data directly to ADLS using the storage account credentials you provided during destination setup. Therefore, syncs complete successfully. The restriction only applies to traffic from the Databricks workspace itself, which is why tables appear in ADLS but are inaccessible from Databricks.
For more information about configuring virtual network rules on Azure storage accounts, see the Microsoft documentation.