Connector SDK Setup Guide

Prerequisites

To connect your Connector SDK to Fivetran, you need the following:

• Your Fivetran REST API key. You can use a scoped key or a system key. The system key must have the required permissions.
• A Python development environment. We support Python version ≥3.9 and ≤3.12.
• A target data source you need to build a connector for and a way to use Python to connect to get the data you need. Most Python libraries can be used to help you achieve this.

See our Connector SDK GitHub repository for examples of how to use our Connector SDK.

Setup instructions

1. Set up or go to your Python development environment and create a directory for the new connector project. Use the IDE you are most familiar with as our Connector SDK works in any Python IDE. We strongly recommend creating and activating a virtual environment. A Python virtual environment is an isolated environment for a Python project. It allows you to manage dependencies for your projects separately, ensuring that packages required for one project don’t interfere with packages required for another. This is particularly useful when different projects require different versions of the same package.

2. (Recommended) If you choose to use a virtual environment, do the following:

   i. To create a virtual environment in Windows, MacOS, or Linux, run the following command:

   python -m venv myenv
   

   This creates a directory called myenv, which contains your virtual environment.

   ii. To activate a virtual environment, depending on your environment, run either of the following commands:

   • In Windows, run the following command:

     myenv\Scripts\activate
     

   • In MacOS and Linux, run the following command:

     source myenv/bin/activate
     

   NOTE: When you are done or when you want to switch projects, in Windows, MacOS or Linux, run the following command to deactivate the virtual environment:

   deactivate
   

3. Install the Fivetran Connector SDK:

   pip install fivetran-connector-sdk
   

4. In our public repository of examples, select your preferred example as a template. Select one from any of the following types that closely resembles the connector you want to write:

   • The Quickstart examples show the simplest implementations of the Fivetran Connector SDK.
   • The Common patterns for connectors examples show commonly used patterns which you can copy paste for use in your code.
   • The Source examples show working examples for syncing records from real-world databases like DynamoDB, Redshift, etc.

5. Copy its code and paste it into a file in your directory called connector.py.

   NOTE: When you create your new connector project, your IDE might create a main.py file for you. If so, rename that file to connector.py so that the Connector SDK can find it.

6. The connector.py file is where you write your custom connector. The following is required for the connector implementation:

   • The file must include the operations to send data to Fivetran
   • The file must include an initialization of the Connector object.
   • You need to include the required imports at the top of your connector.py file.

   Modify the code from the example you copied to write your connector. You can create connectors that use multiple files as long as all the files you need are present in your project directory and are called from your connector.py file. Any additional Python library that can be installed with pip can also be used, and will need to be declared in your requirement.txt file.

   TIP: We encourage you to deploy and run one of our examples as a first connector to get familiar with the Connector SDK.

7. Test the connector by running it locally using the following command:

   fivetran debug
   

   If you have included the following code in your connector.py file, you can test your connector by running your file directly from your IDE:

   if __name__ == "__main__":
      connector.debug()
   

   This test creates a local warehouse.db file, which is a Duck DB representation of the data that the connector delivers to your destination. This file is located in <project_directory>/files/warehouse.db.

   You can explore our state management and configuration to control exactly what is tested with each test run.

   IMPORTANT: Use fivetran debug to test and troubleshoot your connector's behavior with your source's actual data. The tester works by emulating Fivetran's core. However, it is running on your local machine, so it isn't as performant as you will experience when running your connector in production. We recommend running fivetran debug on multiple small samples of your data to fully test your connector's logic. Deploy to production once you are ready to test with larger data sets. State management and configuration can be used to control exactly what is tested with each run. You can use fivetran reset to resets the locally saved cursor and warehouse.db files, allowing you to re-run fivetran debug from scratch. It simulates a historical sync.

8. You can connect to the Duck DB warehouse.db file using DBeaver or using the DuckDB CLI commands. Once you are happy with the connector and the Fivetran Local Tester output, proceed to the next step.

9. Deploy the connector by running the following command from the root directory of your project:

   fivetran deploy --api-key <FIVETRAN-API-KEY> --destination <DESTINATION-NAME> --connection <CONNECTION-NAME>
   

   Your <FIVETRAN-API-KEY> is the base64-encoded {API-key}:{API-secret} used in all other Fivetran API calls. See our System API key documentation to learn how to obtain your API key.

   Your <DESTINATION-NAME> identifies the destination you want your new connector to deliver to. It is the name of your destination in the Fivetran dashboard.

   

   NOTE: <DESTINATION-NAME> is required only if multiple destinations are created for the account. If there is only a single destination, it can be omitted, as it will be automatically identified.

   Choose a name for your connection. Your <CONNECTION_NAME> is the connection name that is displayed in your Fivetran dashboard and that you use for future updates of your connection. The connection name must conform to our naming conventions defined by the renaming rules, ensuring it matches the pattern and character set of transformed names.

   

   TIP: Use environment variables to avoid having to enter your API key repeatedly in your terminal.

Related articles

description Connector Overview

home Documentation Home