SSH File Transfer Protocol (SFTP)

Fivetran supports syncing files from your SSH server to your destination using SFTP.

Features

Setup guide

Follow our step-by-step SFTP connector setup guide to start syncing files from your SSH server to your destination using SFTP.

Sync overview

The SFTP connector offers dual mode sync strategies. You can opt to:

• sync each file as a unique table in your destination (Magic Folder Mode)
• sync multiple files into a single destination table that you specify (Merge Mode)

We sync all files in the SFTP folder to your destination, even if someone without a Fivetran account uploaded them. Anyone with write access to that folder can drop files into the folder; Fivetran automatically syncs the files from your SFTP folder to your destination.

Magic Folder Mode

We retrieve all the files from the folder that you specify into your destination.

Each file is synced as a separate table in the destination. For spreadsheet files, we sync each worksheet in the spreadsheet as a unique table in your destination.

We don't sync empty worksheets. We don't sync an Excel worksheet if the entire first row of the worksheet is empty. We don't sync a column if its first row is empty.

We only sync the files inside the specified folder. We ignore nested folders.

Magic Folder Mode doesn't support incremental syncs. To detect changes in the files of your folder, we use the last modified date of the files. After the initial historical sync, we re-import only the recently modified files in every sync. For information about the sync strategy, see our Magic Folder connectors documentation.

Merge Mode

We retrieve all the files from each folder and its subfolders into a single table into your destination.

Merge Mode supports incremental syncs.

Schema information

Magic Folder Mode

If you select the Magic Folder Mode, we use the file name (without the extension) as the destination table name. For example, we sync a sample.csv file as SAMPLE table.

For spreadsheet files, we use a combination of the file name and worksheet name as the destination table name. For example, two worksheets called Sheet1 and Sheet2 in a spreadsheet My Workings are synced as MY_WORKINGS_SHEET_1 and MY_WORKINGS_SHEET_2 tables. We use the values present in the first row of the worksheet as the column names in the destination table.

For more information about the schema and table naming rule set, see our naming conventions documentation.

If we find files of the same name but with different extensions in the folder, we check the last modified dates of the files and sync only the least recently modified file.

Merge Mode

If you select the Merge Mode, we use the table name that you specify as the destination table name.

Upsert with custom primary keys Beta

For upserting with primary keys, use the Primary Key used for file process and load field in the Merge Mode.

You can select the custom primary keys during the connector setup and use them to update or insert new rows to the destination. We process the files in order of their modified time.

If we detect that one of the selected primary keys has a null value in your files, we display a warning on the connector dashboard but insert the record with a null value in your destination.

If the destination does not support null values, we insert the row with the default value for the column.

You can choose the primary keys only before the initial sync. Once you perform the initial sync, you won't be able to modify the primary keys.

NOTE: You can modify the primary keys only if your initial sync fails. If your initial sync is successful, the option to modify the primary keys is not available.

Types of primary keys

The SFTP connector uses the following types of primary keys:

• Primary key - a column that contains values that uniquely identify each row in your data.
• Composite primary key - a set of columns which, in combination, uniquely identifies a row in your data.
• Surrogate primary key - a column or columns that the system adds to provide a unique identifier for a row in your data.

Syncing empty columns

The SFTP connector doesn't support the creation of empty columns in your destination.

We create a column in the destination table only if the column in your source file is not empty.

Sync limitations

We don't sync hyperlink values from a worksheet and worksheets with pivot tables.