Does Fivetran Support Enabling Change Data Feed (CDF) on Delta Tables?
Question
Does Fivetran support enabling change data feed (CDF) on Delta tables?
Environment
Destination: Azure Data Lake Storage
Answer
We don't support enabling CDF on Delta tables for the two following reasons:
- We use the Delta Standalone Java library to create and manage tables. This library supports writing and reading Delta tables with protocol version (writer: 2, reader: 1). However, when you enable CDF on a Delta table, Azure Data Lake Storage requires protocol version (writer: 4, reader: 1).
- When you enable CDF on a Delta table, the FileAction type
AddCDCFile
is created and recorded in the Delta log file. However, our Delta table maintenance process only supports the FileAction typesAddFile
andRemoveFile
.