Databrickslink
Databricks is an open-source storage layer that brings reliability to data lakes. Databricks provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. Databricks runs on top of your existing data lake and is fully compatible with Apache Spark APIs.
Supported implementationslink
Fivetran supports connecting with two different Databricks implementations:
Setup guidelink
Follow our setup guide to connect Databricks to Fivetran.
Type transformation mappinglink
The data types in Databricks follow Fivetran's standard data type storage.
We use the following data type conversions:
| Fivetran Data Type | Destination Data Type | Notes |
|---|---|---|
| BOOLEAN | BOOLEAN | |
| SHORT | SMALLINT | |
| INT | INT | |
| LONG | BIGINT | |
| FLOAT | FLOAT | |
| DOUBLE | DOUBLE | |
| BIGDECIMAL | DECIMAL | |
| LOCALDATE | DATE | |
| INSTANT | TIMESTAMP | |
| LOCALDATETIME | TIMESTAMP | Databricks requires timezone value |
| STRING | STRING | |
| JSON | STRING | Databricks doesn't support JSON |
| BINARY | BINARY |
Table maintenancelink
We perform weekly maintenance operations on the Delta tables. We run the following operations during the weekend: