Release Notes

October 2024

You can now configure the primary key used for file process and load using the Fivetran REST API.

September 2024

We now restrict API users from triggering File connectors to sync if:

the append_file_option field has its value set to upsert_file_with_primary_keys
and the primary keys are not defined in the schema configuration

We also restrict the following actions for such connectors:

Creating a new connector in an un-paused state via API;
Updating the connector's paused field to false via API request;
Forcing the connector sync or re-sync via API request.

The Amazon S3 connector now supports the Hybrid Deployment model. For more information, see our setup instructions.

January 2024

We have added a new toggle, Non-standard character escaping?, to the connector setup form to support the non-standard ways of escaping characters. If your CSV generator uses non-standard ways of escaping characters, set the toggle to ON and in the Character Escaping options drop-down menu, select one of the following options:

Custom Escape Character: To process CSV files that use custom escape characters to escape quotation marks.
Delimited Only: To process CSV files that don't use escape characters to escape quotation marks and process with delimiter only.

For more information, see our setup instructions and REST API config. We are gradually rolling out this change to all existing connectors.

We have added a new configuration option, Primary Key used for file process and load, in the connector setup form to support syncing files with custom primary keys. You can now select the custom primary keys during the connector configuration and use them to update or insert new rows in your destination. We process the most recently modified files to your destination. For more information see our setup instructions.

September 2023

We now support syncing PGP (Pretty Good Privacy) signed and encrypted files. We have added a new option, Signer's Public Key, to the PGP Encryption Options configuration in the connector setup form. For more information, see our setup instructions.

August 2023

We now validate that the external_id value that you specify when creating the Amazon S3 connector using the REST API is the same as your Fivetran account's group_id. Make sure to choose any of the group_id from your Fivetran account as the external_id parameter value. To find the group_id, use the List All Groups endpoint.

July 2023

We have changed the default value of the JSON Delivery Mode feature to Packed in the connector setup form. Previously, we used Unpacked as the default value for this feature.

We now support syncing XML files to your destination. We have added a new option, xml, to the File Type drop-down menu of the connector setup form. We load your XML data into the _data column without flattening it. For more information, see our setup instructions.

We now support custom line separators for CSV files. You can specify a custom line separator in the Line Separator field of the connector setup form. For more information, see our setup instructions.

You can now opt to provide Fivetran access to your S3 bucket using an access key and secret. For more information, see our setup instructions and REST API config.

June 2023

We now support multi-character delimiters for CSV files. You can specify a multi-character delimiter in the Delimiter field of the connector setup form. For more information, see our setup instructions.

April 2023

We can now sync packed JSON files from Amazon S3. We have added a new drop-down menu, JSON Delivery Mode, to the connector setup form. The drop-down menu provides you the option to choose how Fivetran should handle your JSON data. For more information, see our setup instructions.

March 2023

We now support syncing PGP (Pretty Good Privacy) encrypted files. We have added a new advanced configuration option, PGP Encryption Options, to the connector setup form. For more information, see our setup instructions.

June 2022

We have redesigned the Skip Header Lines and Skip Footer Lines setup form fields for our Amazon S3, Azure Blob Storage, Dropbox, FTP, Google Cloud Storage, and SFTP file connectors. To set these advanced options, enable them using the toggles and specify the number of skipped lines in the input fields.

Compare how the setup form looked before the change:

setup_wizard_before

and how it looks now:

setup_wizard_after

January 2022

We have added support for AWS PrivateLink connections to S3 buckets. If you're on a Business Critical plan, you can now opt to always connect through a private link if your bucket and destination are in the same AWS Region. In the connector setup form, set the Require PrivateLink toggle to ON. For more information, see our setup instructions.

November 2021

You can now use a custom external_id parameter when you create a new Amazon S3 connector with the Fivetran REST API.

NOTE: Connectors created in the Fivetran dashboard do not support custom external_id values, so they use the connector's group_id by default instead.

April 2021

We have added a new advanced configuration option List Strategy to the connector setup form. Enable the Enable Advanced Options toggle and select the listing strategy you want to use:

complete_listing: The default listing strategy where we list all the files in the bucket and filter new and modified files.
time_based_pattern_listing: An optimized listing strategy to improve sync speeds. We fetch the files in a lexicographic order starting from the last file of the previous sync. You can opt to use this strategy if your files are named based on the date or time they are added to the bucket. For example, if a time-based pattern exists in the bucket leading to every new file being appended in a lexicographically increasing order:
Filename Last Modified
2021/01/01.csv January 1, 2021
2021/01/02.csv January 2, 2021
2021/01/03.csv January 3, 2021
2021/01/04.csv January 4, 2021
NOTE: If we are unable to identify a time-based pattern in the file names, we will use the default strategy and list all the files.

Filename	Last Modified
2021/01/01.csv	January 1, 2021
2021/01/02.csv	January 2, 2021
2021/01/03.csv	January 3, 2021
2021/01/04.csv	January 4, 2021

August 2020

We now support syncing headerless delimited format files (CSV, TSV, log) for S3 connectors. We will create generic column names for CSV files without a header line. This feature is in beta and available to all connectors. See the configuration options in our files documentation for details.

June 2020

We now exclude S3 objects that have been archived to Glacier storage class from our data syncs. If you want us to sync these objects, restore them to standard storage.

April 2020

You can now connect to public S3 buckets without needing an AWS account.

March 2020

Our Amazon S3 connector can now sync Parquet files. We support Parquet format 2.4.0. This feature is in Beta.

June 2019

We now require you to configure a Role ARN when you set up an Amazon S3 connector, even if your bucket is a public bucket.

Existing Amazon S3 connectors will not be affected. However, if you reconfigure an existing connector or create a new one, you will have to configure a Role ARN even if your bucket is a public bucket.

We have added a new connection test to the Amazon S3 setup form that helps you adopt security best practices. The test checks your bucket policy settings to make sure you have created your Role ARN using the given externalID.

If you mistakenly create your Role ARN without using the give externID, the configuration will now fail the test. You will have to correct your Role ARN.

August 2018

We can now sync data from public buckets.

May 2018

Our connector setup form now correctly handles directory patterns that don't end with /.