Amazon S3 link
Updated January 23, 2024
Amazon Simple Storage Service (Amazon S3), provides scalable cloud storage services. Fivetran supports syncing files from your S3 bucket to your destination.
|All tables and fields
|Fivetran data models
Follow our step-by-step Amazon S3 setup guide to connect Amazon S3 with your destination using Fivetran connectors.
We will search a small subset of your bucket, and, if no matches are found, will expand its search to the entire bucket. This can take up to several minutes.
We do not sync data from S3 replication (destination) buckets.
Syncing empty columnslink
The Amazon S3 connector doesn't support the creation of empty columns in your destination.
We create a column in the destination table only if the column in your source file is not empty.
Upsert with custom primary keys Betalink
For upserting with primary keys, use the Primary Key used for file process and load field.
You can select the custom primary keys during the connector setup and use them to update or insert new rows to the destination. We process the files in order of their modified time.
If we detect that one of the selected primary keys has a
null value in your files, we display a warning on the connector dashboard but insert the record with a
null value in your destination.
If the destination does not support null values, we insert the row with the default value for the column.
NOTE: You can modify the primary keys only if your initial sync fails. If your initial sync is successful, the option to modify the primary keys is not available.
Types of primary keyslink
The file connectors use the following types of primary keys:
- Primary key - a field that contains values which uniquely identify each row in your data.
- Composite primary key - a set of fields which, in combination, uniquely identifies a row in your data.
- Surrogate primary key - a column or columns that the system adds to your data in order to provide a unique identifier for a row in your data.
Fivetran offers a number of configuration options in our setup form. Using these configuration options, you can select subsets of your folders, certain types of files, and more to sync only the files you need in your destination. Setting up multiple S3 connectors targeted at the same bucket, but with different options, can allow you to slice and dice a bucket any way you'd like. For more on configuration options, see our setup guide.
You can use a custom
external_id parameter for authentication when creating a new Amazon S3 connector with the Fivetran REST API. We use the connector's
group_id if you don't specify the
external_id parameter value. Use the List All Groups endpoint to find the connector's
NOTE: We don't support custom
external_idvalues for connectors created in the Fivetran dashboard. By default, we use the connector's