Best Practices

The following sections describe best practices that help leverage the Connector SDK.

Declaring primary keys

To optimize the uploading of your data to your destination, we recommend that you use the Schema() method to declare the primary key for each of your tables.

The use of the Schema() method is optional - Fivetran automatically syncs any table regardless of whether you have declared the table in the Schema() method or not. For tables not declared, Fivetran creates a surrogate primary key column named _fivetran_id. See our documentation for more details.

Declaring columns and data types

Using the Schema() method to declare all data columns and their data types is NOT recommended. Connector SDK will infer the type of data in any column you send to us using Fivetran's standard data type inference. Declaring data types is useful if you have a particular data type you want to ensure we use.

Use a revision control system

Any time you are writing code, unexpected events can happen. We strongly recommend using a revision control system like GitHub or Bitbucket to keep all your code changes tracked and secure. This is also hugely beneficial when collaborating on a custom connector with other members of your team.

Consider centralizing deployment to a code deployment system

Deploying code from individual machines is generally not considered good practice. The Connector SDK is a command line-based tool designed to work smoothly with standard automated code integration and code deployment tools.

Upgrading your Fivetran Connector SDK

We recommend using the latest version of the Fivetran Connector SDK, as it contains all the latest definitions and updates. You can easily upgrade it via pip by running the following command:

pip install --upgrade fivetran_connector_sdk

Using fivetran debug

Use fivetran debug to test and troubleshoot your connector's behavior with your source's actual data. The tester works by emulating Fivetran's core. However, it is running on your local machine, so it isn't as performant as you will experience when running your connector in production. We recommend running fivetran debug on multiple small samples of your data to fully test your connector's logic. Deploy to production once you are ready to test with larger data sets. State management and configuration can be used to control exactly what is tested with each run.