Databricks Beta
Databricks is an open-source storage layer that brings reliability to data lakes. Databricks provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. Databricks runs on top of your existing data lake and is fully compatible with Apache Spark APIs.
Setup guide
Fivetran supports connecting with two different Databricks implementations. Follow the setup guide for your Databricks implementation:
Type transformation mapping
The data types in Databricks follow Fivetran's standard data type storage.
We use the following data type conversions:
| Fivetran Data Type | Destination Data Type | Notes |
|---|---|---|
| BOOLEAN | BOOLEAN | |
| SHORT | SMALLINT | |
| INT | INT | |
| LONG | BIGINT | |
| FLOAT | FLOAT | |
| DOUBLE | DOUBLE | |
| BIGDECIMAL | DECIMAL | |
| LOCALDATE | DATE | |
| INSTANT | TIMESTAMP | |
| LOCALDATETIME | TIMESTAMP | Databricks requires timezone value |
| STRING | STRING | |
| JSON | STRING | Databricks doesn't support JSON |
| BINARY | Not Supported |
Table maintenance
We perform weekly maintenance operations on the Delta tables. We run the following operations during the weekend: