Custom Connector Development
This section covers connector development essentials, such as working with connector-development project files and configuring your connector for deployment. For testing, performance profiling, and memory management, see Testing and Performance Optimization.
See our Fivetran Connector SDK GitHub repository for connector development patterns and quickstart examples, and our Community Connectors repository for custom connector examples that we recommend you consider and try out. Follow the Connector SDK setup guide to start using the SDK.
Connector development essentials
- Project structure recommendation - Learn what's the recommended connector development project structure.
- Connector SDK CLI commands - Learn how to use Connector SDK CLI commands, and which options are available.
- Managing project dependencies - Learn how to manage Python dependencies for your connector using a
requirements.txtorpyproject.tomlfile, including PyPI and Git sources. - Configuration - Learn how to securely pass and manage configuration values — such as API keys and credentials — to your connector using a
configuration.jsonfile. Also covers some useful data type conversion patterns for configuration values. - Setup Form - Learn how to define an optional setup form to collect configuration values in the Fivetran dashboard and run custom setup tests.
- State management - Learn how to use the state object to enable incremental syncs, manage cursors locally and in production, and debug state-related issues.
- Working with environment variables - Learn how to use environment variables to speed up local development and connector deployment. Also, learn how to reference Fivetran-provided runtime environment variables in your connector code.
- Connection options - Learn how to use supported network connection options for your connector.
See our Connector SDK tutorials section for step-by-step instructions in written and video format.
Planning how to build a data connector
The following diagram shows the key areas to consider when designing your connector:
Connector SDK logs
You can access your Connector SDK connections' log events by:
- Navigating to the Connector SDK logs tab of your connection details page in the Fivetran dashboard.
- Using the Fivetran Platform Connector. Your connections' logs are available in the
CONNECTOR_SDK_LOGtable within the associated Fivetran destination. - Using external log services, previously configured for your destination.
Connector SDK logs provide in-depth event data, including timestamps, log levels, and messages. We prefix all Fivetran process logs with the platform identifier, and logs from the fivetran-connector-sdk library are prefixed with the sdk identifier.
For more details, refer to Technical details - logging.
Connector SDK code download
Download the package code for a Connector SDK connection from your Fivetran dashboard or the Download Connector SDK package endpoint.
Code download is enabled by default. The connector package owner or author can disable it in their Connector SDK project.
To download the connector code from the Fivetran dashboard:
- Open the Connector SDK connection.
- Select the Settings tab.
- Click Download Code to download a ZIP file containing the connector code files.
The downloaded ZIP file doesn't include configuration.json or state.json.
If code download is disabled, the Download Code button is disabled, and the Download Connector SDK package endpoint returns an HTTP 403 error.