Release Notes

November 2024

We have made the following updates to the setup form:

Added a new configuration option, Preview Files to perform an intermediate test during configuration
Added support for displaying fields based on the file type
Added a new field, Quote character, to use an enclosing character
Reorganised fields to make the setup form more intuitive
Removed Infer as an option from the File Type field
Removed the Enable Advanced Options toggle

For more information, see our setup instructions and REST API configuration. We are gradually rolling out these changes to all existing connectors.

October 2024

We have added a new configuration option, Run connection test to the setup form to perform an intermediate test during configuration. We perform this test to validate the login credentials and the connection to the SFTP server. For more information, see our setup instructions.

The SFTP connector now supports the Hybrid Deployment model. For more information, see our setup instructions.

September 2024

We now restrict API users from triggering File connectors to sync if:

the append_file_option field has its value set to upsert_file_with_primary_keys
and the primary keys are not defined in the schema configuration

We also restrict the following actions for such connectors:

Creating a new connector in an un-paused state via API;
Updating the connector's paused field to false via API request;
Forcing the connector sync or re-sync via API request.

August 2024

We now display the public key in two separate fields. To support this change, we have replaced the Public Key field with the SFTP Server Public Key and SSH Tunnel Public Key fields in the setup form.

We have renamed the following fields in the connector setup form:

Host Address to SFTP Server Host Address
Port to SFTP Server Port
Username to SFTP Server Username
Password to SFTP Server Password
Tunnel Username to SSH Tunnel Username
Tunnel Host Address to SSH Tunnel Host Address
Tunnel Port to SSH Tunnel Port

For more information, see our setup instructions.

July 2024

We have added the Magic Folder Mode functionality to our SFTP connector. For connectors created after July 22, 2024, you can now opt to sync files as unique tables within a schema in your destination. You can either select Magic Folder or Merge Mode as your sync strategy, in the connector setup form. For more information, see our setup instructions and REST API configuration.

January 2024

We have added a new configuration option, Primary Key used for file process and load, in the connector setup form to support syncing files with custom primary keys. You can now select the custom primary keys during the connector configuration and use them to update or insert new rows in your destination. We process the most recently modified files to your destination. For more information, see our setup instructions.

September 2023

We now support syncing PGP (Pretty Good Privacy) signed and encrypted files. We have added a new option, Signer's Public Key, to the PGP Encryption Options configuration in the connector setup form. For more information, see our setup instructions.

July 2023

We have changed the default value of the JSON Delivery Mode feature to Packed in the connector setup form. Previously, we used Unpacked as the default value for this feature.

We now support syncing XML files to your destination. We have added a new option, xml, to the File Type drop-down menu of the connector setup form. We load your XML data into the _data column without flattening it. For more information, see our setup instructions.

June 2023

We now support multi-character delimiters for CSV files. You can specify a multi-character delimiter in the Delimiter field of the connector setup form. For more information, see our setup instructions.

We now support custom line separators for CSV files. You can specify a custom line separator in the Line Separator field of the connector setup form. For more information, see our setup instructions.

May 2023

We now support syncing PGP (Pretty Good Privacy) encrypted files. We have added a new advanced configuration, PGP Encryption Options, to the connector setup form. For more information, see our setup instructions.

Our SFTP connector can now sync packed JSON files from SSH servers. We have added a new drop-down menu, JSON Delivery Mode, to the connector setup form. The drop-down menu provides you the option to choose how Fivetran should handle your JSON data. For more information, see our setup instructions.

November 2022

You can now connect your SFTP Server to Fivetran using an SSH tunnel. For more information, see our SFTP setup guide.

June 2022

We have redesigned the Skip Header Lines and Skip Footer Lines setup form fields for our Amazon S3, Azure Blob Storage, Dropbox, FTP, Google Cloud Storage, and SFTP file connectors. To set these advanced options, enable them using the toggles and specify the number of skipped lines in the input fields.

Compare how the setup form looked before the change:

setup_wizard_before

and how it looks now:

setup_wizard_after

August 2020

We now support syncing headerless delimited format files (CSV, TSV, log) for SFTP connectors. We will create generic column names for CSV files without a header line. This feature is in beta and available to all connectors. See the configuration options in our files documentation for details.

July 2020

We have improved the way we track which files we have already synced to make sure we only pull new or changed data from the source containers. Previously, we re-synced files that were created at the same time as the last observed cursor position. That ensured that we never missed any files that were created while we were syncing your data. That also meant that we sometimes synced the same files twice. Now, in addition to tracking the timestamp, we also track the names of the files we have already synced. We store up to 1,000 file names. We sync files created at the time of the last observed cursor position only if we don’t have the file in our list of synced files for that timestamp.

March 2020

Our SFTP connector can now sync Parquet files. We support Parquet format 2.4.0. This feature is in Beta.

July 2019

We have fixed an out of memory error that occurs when the number of files in the folder is extremely large.

We now reconnect if the SFTP session is disconnected by the SFTP server.

September 2018

The No Such File Warning message is now fixed on your Fivetran dashboard.