Amazon S3

Amazon Simple Storage Service (Amazon S3), provides scalable cloud storage services. Fivetran supports syncing files from your S3 bucket to your destination.

Features

Setup guide

Follow our step-by-step Amazon S3 setup guide to connect Amazon S3 with your destination using Fivetran connectors.

Sync notes

We will search a small subset of your bucket, and, if no matches are found, will expand its search to the entire bucket. This can take up to several minutes.

We do not sync data from S3 replication (destination) buckets.

Syncing empty columns

The Amazon S3 connector doesn't support the creation of empty columns in your destination.

We create a column in the destination table only if the column in your source file is not empty.

Schema information

Upsert with custom primary keys Beta

For upserting with primary keys, use the Primary Key used for file process and load field.

You can select the custom primary keys during the connector setup and use them to update or insert new rows to the destination. We process the files in order of their modified time.

If we detect that one of the selected primary keys has a null value in your files, we display a warning on the connector dashboard but insert the record with a null value in your destination.

If the destination does not support null values, we insert the row with the default value for the column.

NOTE: You can modify the primary keys only if your initial sync fails. If your initial sync is successful, the option to modify the primary keys is not available.

Types of primary keys

The Amazon S3 connector uses the following types of primary keys:

• Primary key - a column that contains values that uniquely identify each row in your data.
• Composite primary key - a set of columns which, in combination, uniquely identifies a row in your data.
• Surrogate primary key - a column or columns that the system adds to provide a unique identifier for a row in your data.

Configuration options

Fivetran offers a number of configuration options in our setup form. Using these configuration options, you can select subsets of your folders, certain types of files, and more to sync only the files you need in your destination. Setting up multiple S3 connectors targeted at the same bucket, but with different options, can allow you to slice and dice a bucket any way you'd like. For more on configuration options, see our setup guide.

You can use a custom external_id parameter for authentication when creating a new Amazon S3 connector with the Fivetran REST API. We use the connector's group_id if you don't specify the external_id parameter value. Use the List All Groups endpoint to find the connector's group_id.

NOTE: We don't support custom external_id values for connectors created in the Fivetran dashboard. By default, we use the connector's group_id.