Why Is Fivetran Re-Syncing a Table When New Data Should Be Blocked?
Question
My connection's Schema Change Handling settings are set to Block all new data. However, following a Data Definition Language (DDL) change, Fivetran is re-syncing a table. Why is this happening?
Environment
All MySQL connectors
Answer
Even when your Schema Change Handling settings block all new data, Fivetran re-syncs tables after certain schema changes. This happens because MySQL binary logs record data changes without column names, relying entirely on column order. Because MySQL maintains strict column ordering, we must preserve that same order when parsing binary log events.
How column order affects parsing
When we read binary logs, we use the current schema (column order and data types) from the source database to interpret each record. If the schema has changed since the event was written to the binary log, the column order may no longer match, making the data impossible to parse correctly.
Example
- Original schema:
{id: INT, col_a: INT, col_b: INT} - Column order:
{id: 0, col_a: 1, col_b: 2} - Binary log entry:
insert {101, "foo", "bar"}
We read this as:
id = 101
col_a = "foo"
col_b = "bar"
When you add a column
Suppose the previous schema was {id: INT, col_a: STRING}, and you do the following in MySQL:
- Insert a record using the current schema.
- Add a new column
col_b. - Insert another record that includes
col_b.
The binary log now contains events written under two schema versions: one without col_b and one with it. When we later read the binary log, we fetch the latest schema from the source database: {id: INT, col_a: STRING, col_b: STRING}
Because we use this updated schema to parse all binary log events, earlier inserts (which only have two columns) no longer align with the expected three-column order. As a result, we can't interpret those older events correctly.
Why we trigger a re-sync
When we can't parse events because of column order mismatches, we trigger a table re-sync to ensure accurate binary log parsing. This happens even if your Schema Change Handling settings block new data. This process lets your connection:
- Rebuilds the table using the correct, updated column order.
- Maintains consistent mappings between binary log data and schema definitions.
- Can correctly parse all future binary log events.
This re-sync ensures that Fivetran maintains the correct column ordering needed to accurately parse the MySQL binary logs.