Local File System Private Preview
The Local File System connector enables you to sync files to your destination from file systems in your local environment, SMB/CIFS and NFS network file shares, and Network File System (NFS), NAS devices, and cloud file services such as Azure Files, Amazon FSx, and Amazon Elastic File System (EFS). It reads files from locations that are accessible to the Hybrid Deployment Agent. To sync files stored on a Windows server, share the folder over SMB and mount it on the host where the Hybrid Deployment Agent runs.
When to use Local File System
Use the Local File System connector to sync files from on-premises or network storage that the Hybrid Deployment Agent can access but Fivetran cannot access directly over the internet without setting up an SFTP server or copying them to a cloud storage. If your files are already available through SFTP, use the SFTP connector. If your files are stored in object storage, use the Amazon S3, Azure Blob Storage, or Google Cloud Storage connector.
Features
| Feature Name | Supported | Notes |
|---|---|---|
| Capture deletes | ||
| History mode | ||
| Custom data | ||
| Data blocking | ||
| Column hashing | ||
| Re-sync | ||
| Row filtering | ||
| API configurable | API configuration | |
| Priority-first sync | ||
| Fivetran data models | ||
| Private networking | ||
| Authorization via API |
Supported deployment models
We support the Hybrid Deployment model for the connector.
You must have an Enterprise or Business Critical plan to use the Hybrid Deployment model.
Supported file sources
The Local File System connector can sync files from the following sources:
- Local disks on the Hybrid Deployment Agent host
- SMB/CIFS shares, such as Windows file servers and NAS devices
- NFS exports
- Cloud file services, such as Azure Files, Amazon FSx, and Amazon EFS
Fivetran doesn't connect to the file share directly. You must mount the file share on a host where the Hybrid Deployment Agent can reach it. We then read the files as a local directory. We mount the source files as read-only, which prevents us from modifying or deleting them.
Setup guide
Follow our step-by-step Local File System connector setup guide to sync files accessible to the Hybrid Deployment Agent with your destination using Fivetran connectors.
Sync overview
The Local File System connector syncs multiple files into multiple destination tables from a file system accessible on your Hybrid Deployment agent. Fivetran supports the following methods for configuring file-to-table mappings:
- Define per table — Maps files to destination tables by matching file name patterns using regular expressions.
- Dynamically extract tables — Extracts destination table names directly from file paths using a regular expression with a named capture group.
Fivetran detects new and modified files using the last modified timestamp of each file. Once the initial sync is complete, we re-process only files modified after the last sync.
Define per table
Define per table method allows you to define one or more destination tables and associate each table with a file name pattern. We sync only the files that match a configured pattern and exclude those that do not match a pattern.
Dynamically extract tables
Dynamically extract tables allows you to specify a regular expression with a named capture group (?<table>...) to extract table names from file paths. If we find any new table names after the initial sync, we automatically add them to your destination, subject to your Schema change settings.
Schema information
Define per table
The Table name you specify when mapping a file pattern determines the destination table name.
Dynamically extract tables
We extract the destination table name from the file path using the table extraction pattern you configured. We follow Fivetran's naming conventions to convert the extracted names.