What Is the Optimal Configuration of Databricks for Fivetran?

Question

What is the optimal configuration of Databricks for Fivetran?

Environment

Destination: Databricks

Answer

For an optimal configuration of your Databricks destination, we recommend that you do the following:

1. Configure your Databricks destination according to our setup instructions.
2. Enable autoscaling for your Databricks clusters.
3. Configure the min and max workers as required.

Context

We recommend enabling autoscale because a Databricks cluster will weigh either towards Compute or Memory-Optimised, depending on how long queries take per table. This is determined by the amount of data being synced per table.