Release Noteslink
October 2023link
We no longer enforce the column names to be in lower case. We now ignore the case of the column names in your destination tables. Your queries and scripts will continue to execute because Databricks is case-insensitive.
NOTE: If you have previously synced a column name with upper case letters to your destination table and if a column data type changes in the table, we will rename all the columns in the table to lower case.
March 2023link
For the service
parameter in the API request for our Databricks destination, you now need to specify databricks
as a value. The old values databricks_aws
and databricks_azure
are still valid.
August 2022link
Our Databricks destination now supports the Unity Catalog feature. For more information, see our setup instructions.
To upgrade your existing tables and schemas to Unity Catalog, see the following:
March 2022link
We now use Azure Blob Storage as the staging location for our temporary files if:
- Your Databricks cluster is hosted on Azure
- Your Databricks cluster is a SQL Endpoint cluster
- Your Databricks cluster is a general purpose cluster with DBR version 10.2 or greater
November 2021link
Our Databricks destination now supports clusters with Databricks Runtime versions 9.0 - 10.x.
June 2021link
Our Databricks destination now supports Databricks SQL endpoint connections. For more information, see our setup instructions.
We now support Databricks on Google Cloud.
April 2021link
Our Databricks destination now supports clusters with Databricks Runtime 8.0 and above.
March 2021link
Our Databricks destination is now generally available.
Read our Databricks destination documentation.
December 2020link
We have reduced the sync duration of append-only tables that have primary key columns of INT
, SMALLINT
, or BIGINT
data type. During internal performance testing, we observed a reduction in sync time for destinations with large table sizes.
Now dbt Transformations support the Databricks destination. If you use Databricks as a destination, you can just set up your dbt Transformations using our guide; no additional steps are required.
August 2020link
Our Databricks REST API endpoint now supports the creation of external tables. You can now opt to create Delta tables as external tables for your Databricks implementations.
Our Databricks destination now supports the creation of external tables. You can now opt to create Delta tables as external tables from the connector setup form.
We now support syncing the BINARY data type from your source.
July 2020link
We now use our own Amazon S3 bucket as an intermediate storage for staging temporary data during a sync. Now, when setting up Databricks as your destination, you do not have to create a S3 bucket.
We will end support for clusters with Databricks Runtime 7.0 and below, on August 15, 2020. To prevent your integrations from failing or causing data loss, upgrade your Databricks Runtime to 7.1 before August 15, 2020.
June 2020link
We now replicate empty tables in a PostgreSQL source database as empty tables in the destination.
February 2020link
We have added Databricks as one of our supported destinations. You can now use Databricks as your destination with Fivetran connectors.