Snowflake Internal vs. External Staging

Question

Does Fivetran use internal or external staging for my Snowflake destination?

Environment

Destination: Snowflake

Answer

Fivetran always uses external staging in our staging environment. We manage the files in the staging area. Data is always encrypted, and we automatically generate temporary credentials to load the data. We delete the data once it is successfully loaded.

Context

In Snowflake's Overview of data loading documentation, they note that bulk load is the most efficient way to load large volumes of data. For the insert-only initial sync, Fivetran therefore leverages staging. We load data from files in the staging area directly into the target tables using copy.

For updates following the initial sync, we process the changes using controlled routines. We use staging to bulk load incremental changes into staging tables, followed by merge statements to apply the changes to the target tables. Since the data already resides in Snowflake, the merges are very efficient, allowing for high throughput (averaging a high number of changes per second).