Databricks Setup Guide
Follow our setup guide to connect Databricks to Fivetran.
Prerequisites
To connect Databricks to Fivetran, you need:
- A Databricks account
- A Databricks cluster (version 7.1, 7.2, or 7.3)
Setup instructions
Create new destination
-
Go to your Fivetran account page.
-
Click + Destination.
-
In the destination drop-down menu, select Databricks on AWS or Databricks on Azure to launch the setup form. Keep this tab open.
Create a Databricks Cluster
-
In a new tab, log in to your Databricks account.
-
In the Databricks console, click Clusters in the left menu.
-
Click + Create Cluster.
-
Enter a Cluster name of your choice.
-
Set the Databricks Runtime Version to 7.1 or later.
-
Under Advanced Options, select Spark.
-
Paste the following code in the Spark Config box:
spark.databricks.delta.alterTable.rename.enabledOnAWS true spark.hadoop.fs.s3a.impl shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem spark.hadoop.fs.s3n.impl shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem spark.hadoop.fs.s3n.impl.disable.cache true spark.hadoop.fs.s3.impl.disable.cache true spark.hadoop.fs.s3a.impl.disable.cache true spark.hadoop.fs.s3.impl shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem
-
Paste the following code in the Environment Variables box:
AWS_STS_REGIONAL_ENDPOINTS="us-east-1"
Get credentials
-
In the same Advanced Options window, select JDBC/ODBC.
-
Make a note of the following values. You will need them to fill in your Fivetran Databricks setup form.
- Server Hostname
- Port
- HTTP Path
Create a personal access token
Follow Databricks' token management guide to create a new personal access token.
Finish Fivetran configuration
-
Return to the tab with your Fivetran setup form.
-
Enter the Server Hostname, Port, and HTTP Path you found in Step 3.
-
Enter the Personal Access Token you created in Step 4.
-
(Optional) Set the Create Delta tables in an external location toggle to ON to create Delta tables as external tables. You can:
-
Enter the External Location you want to use. We will create the Delta tables in the
{externallocation}/{schema}/{table}
path. -
Use the default Databricks File System location registered with the cluster. Do not specify the external location. We will create the external Delta tables in the
/{schema}/{table}
path.
-
-
Click Save & Test.