Databricks Beta
Databricks is an open-source storage layer that brings reliability to data lakes. Databricks provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. Databricks runs on top of your existing data lake and is fully compatible with Apache Spark APIs.
Supported implementations
Fivetran supports connecting with two different Databricks implementations:
Setup guide
Follow our setup guide to connect Databricks to Fivetran.
Type transformation mapping
The data types in Databricks follow Fivetran's standard data type storage.
We use the following data type conversions:
| Fivetran Data Type | Destination Data Type | Notes |
|---|---|---|
| BOOLEAN | BOOLEAN | |
| SHORT | SMALLINT | |
| INT | INT | |
| LONG | BIGINT | |
| FLOAT | FLOAT | |
| DOUBLE | DOUBLE | |
| BIGDECIMAL | DECIMAL | |
| LOCALDATE | DATE | |
| INSTANT | TIMESTAMP | |
| LOCALDATETIME | TIMESTAMP | Databricks requires timezone value |
| STRING | STRING | |
| JSON | STRING | Databricks doesn't support JSON |
| BINARY | BINARY |
Table maintenance
We perform weekly maintenance operations on the Delta tables. We run the following operations during the weekend: