Release Notes
January 2021
We have fixed a bug where TIME values were sometimes synced to the destination incorrectly. Previously, we added a .0
to values with no decimals and dropped the hundredths place for values that did. We now sync TIME values to the destination as they appear in the source:
Source value | Destination value (with bug) | Destination value (fixed) |
---|---|---|
23:59:59 |
23:59:59.0 |
23:59:59 |
23:59:59.00 |
23:59:59.0 |
23:59:59.00 |
23:59:59.10 |
23:59:59.1 |
23:59:59.10 |
If any values were incorrectly synced prior to this change, you will need to initiate a historical re-sync or table re-sync to correct them.
April 2020
We have increased the fractional second precision of our binary log parser from millisecond to microsecond for DATETIME, TIMESTAMP, and TIME types. To correct existing values of these types in your destination, trigger a table re-sync or a historical re-sync in the connector dashboard.
We now add a fivetran_synced column while syncing empty tables and columns from the source.
We have added Magento MySQL and Magento MySQL RDS to the list of connectors that can be created via our REST API.
October 2019
-
We are now rolling out support for BLOB, TINYBLOB, MEDIUMBLOB, LONGBLOB, and YEAR data types in version 4 of our MySQL connector.
-
We are now rolling out the ability to sync MySQL DATETIME columns to the destination as TIMESTAMP_NTZ columns for version 4 of our MySQL connector.
-
We have a fixed a bug that caused zeroed DATE and DATETIME values to sync incorrectly. MySQL allows
0000-00-00
and0000-00-00 00:00:00
as values for DATE and DATETIME primary key columns respectively. Previously, we converted these values to1970-01-01
and1970-01-01 00:00:00
. Therefore, zeroed times and the real values of1970-01-01
and1970-01-01 00:00:00
were mapped to the same value in the destination. Fivetran now converts these values as follows:- DATE: from
1970-01-01
to-999999999-01-01
- DATETIME: from
1970-01-01 00:00:00
to-999999999-01-01 00:00:00
- DATE: from
September 2019
-
If you have a logging service connected to your Fivetran account, the import progress of your tables in the event logs will be visible in your logs as an
import_progress
event. Table names will be in either theCOMPLETE
orIN_PROGRESS
state, depending on their sync status. Tables that are not reported in the event have not started their import. -
We have begun rolling out support for the following spatial types in version 4 of our MySQL connector:
- GEOMETRY
- GEOMETRYCOLLECTION
- LINESTRING
- MULTILINESTRING
- POINT
- MULTIPOINT
- POLYGON
- MULTIPOLYGON
August 2019
We have changed the MySQL version requirements for Fivetran to store data as JSON in a MySQL warehouse and create tables with primary keys.
We now convert JSON to text in versions below 5.7.8.
We now create primary keys in either of the following scenarios:
-
MySQL version earlier than 5.7.7:
- If the total number of primary keys provided by the source is less than 16, and the sum of the byte length of String primary keys is less than 191.
-
MySQL version 5.7.7 and later:
- If the total number of primary keys provided by the source is less than 16, and the sum of the byte length of String primary keys is less than 768.
Fivetran has released version 4 of our MySQL connector, which offers improved sync reliability. We will add a number of upgrades to this version in the coming months:
-
We will sync DATETIME values as TIMEZONE_NTZ values in your destination.
-
We will support parsing for TRUNCATE DDL events.
-
We will support the following new data types:
- BINARY
- VARBINARY
- YEAR
- BLOB
- SMALLBLOB
- MEDIUMBLOB
- LONGBLOB
- GEOMETRY
- GEOMETRYCOLLECTION
- LINESTRING
- MULTILINESTRING
- POINT
- MULTIPOINT
- POLYGON
- MULTIPOLYGON
This month, we have begun rolling out support for BINARY and VARBINARY types.
For more information about data types, see our MySQL documentation.
June 2019
We have added a test to the MySQL setup forms which checks if you are using RDS. We added the test because if you select the wrong variant in the setup form, your connector may not work properly.
The test fails when there is a discrepancy between your selection and what Fivetran detected. If you are not confident in your selection, you may go back to the Add Connector view and select the connector type that matches your MySQL database variant.
Fivetran no longer supports invalid Date
, Datetime
and Timestamp
values. For example, 2019-00-01 00:00:00
is invalid. This change immediately affects all new MySQL connectors, and we will roll it out gradually to existing connectors.
The supported ranges of Date
, Datetime
and Timestamp
in MySQL are:
Type | Supported Range |
---|---|
Date | '1000-01-01' to '9999-12-31' |
Datetime | '1000-01-01 00:00:00' to '9999-12-31 23:59:59' |
Timestamp | '1970-01-01 00:00:01' UTC to '2038-01-19 03:14:07' UTC |
We will load these invalid values as NULL
into the destination and display a warning on your Fivetran dashboard. The warning will indicate the schemas, tables, and columns which contain NULL
values.
March 2019
We have fixed a problem where unsigned BIGINT values larger than 9223372036854775807 were sometimes parsed incorrectly during incremental updates. If any values were incorrectly synced prior to this change, you will need to initiate a re-sync to correct them.
We now replicate empty tables in a MySQL source database as empty tables in the destination.
To correct data integrity issues, we will now trigger a re-sync for a table when we detect a reduction in the size of one of its ENUM columns. Before the re-sync begins, you may notice FIVETRAN_ERROR_ENUM_DELETED
values in these columns.
January 2019
Fivetran will create primary keys for a MySQL warehouse in either of the following scenarios:
-
MySQL version earlier than 5.7 or MariaDB version earlier than 10.2: If the source provides fewer than 16 primary keys and the sum of the byte length of string primary keys is less than 191.
-
MySQL version 5.7 and later or MariaDB version 10.2 and later: If the source provides fewer than 16 primary keys and the sum of the byte length of string primary keys is less than 768.
November 2018
We have identified and fixed an extremely rare bug that can cause data to be missed during binary log replication. The bug meant that, in some cases, our system believed that all data had been successfully synced, when the binary log replication process had actually stopped prematurely.
October 2018
For the following warehouses, Fivetran now drops the primary_key
field from existing tables in your warehouse if there is a primary_key
coming in from your data source:
- MySQL
- PostgreSQL
- SQL Server
- Redshift
- Periscope
- Panoply
September 2018
The MySQL connector is now able to handle unescaped control characters in textual data as if they had been properly escaped.
The MySQL warehouse now uses the load data infile strategy to write data. It requires the load_infile
system variable to be set to on
. For additional details, refer to our MySQL warehouse documentation.
August 2018
String values will now be created as LONGTEXT
columns in the MySQL warehouse instead of VARCHAR
fields if they are not the primary key. We made this change to better handle MySQL's maximum row size limit.
July 2018
You can now customize the replica ID used to access your database's binary logs. By default, the replica ID is a random integer greater than 1000.
May 2018
We now fully support the SET
data type and consistently sync it as a JSON array. Previously, we synced SET
as a string during the initial import and as an integer when processed from the binary logs. If any values were incorrectly synced prior to this change, you will need to initiate a re-sync to correct them.
April 2018
If your data warehouse uses a MySQL version earlier than 5.7, you can't use the timestamp
column as a primary key because MySQL incorrectly handles the decimal part of the timestamp. We now detect this scenario, skip tables with this column during our sync, and warn you to upgrade your data warehouse MySQL version. Note that this limitation only applies to MySQL destinations, not MySQL sources.