Working with Environment Variables

It's much more convenient to use environment variables for parameters you would otherwise have to enter repeatedly into the command line. We recommend using environment variables for the required deployment values so you do not have to repeat them every time you deploy your connector.

How to use environment variables

There are multiple ways to set environment variables:

Using python-dotenv
Through your terminal shell
Creating an .env file (easy to get started with)

If you choose to use an .env file, create it in the root directory of your connector project with the following format:

FIVETRAN_API_KEY=<your 64 bit encoded Fivetran API key>

Load your .env file for use in your terminal by running the following command:

export $(grep -v '^#' .env | xargs)

Check that your environment variables are now available for use by running the following command:

echo $FIVETRAN_API_KEY

You should use configuration files for any configuration used within your connector.py code so that it will be available when the code is deployed and running in Fivetran and stored securely within Fivetran.

.env files are not deployed to Fivetran with the rest of your code.

For the required fivetran deploy values, the CLI resolves them in the following order:

Explicit flag value
Matching environment variable
Command failure with a clear error message

Once the local environment variables are set, use the following command to deploy your connector.py:

fivetran deploy

You can also use environment variables and .env files to set defaults for other parameters used in fivetran debug and fivetran deploy. This helps you test and deploy connectors and switch between connections more efficiently.

Supported environment variables for setting command parameter defaults are:

FIVETRAN_API_KEY=<your 64-bit encoded Fivetran API key>
FIVETRAN_DESTINATION_NAME=<your destination name>
FIVETRAN_CONNECTION_NAME=<your connection name>
FIVETRAN_CONFIGURATION=<your configuration file's absolute or relative path with its extension; for example, `config.json`, `~/Documents/config.json`>
FIVETRAN_PYTHON_VERSION=<the python version you will to use; for example, `3.11`>
FIVETRAN_HD_AGENT_ID=<your non default HD agent ID>
FIVETRAN_NAMING=<naming strategy for the connection schema; either `FIVETRAN` (default) or `SOURCE`. See naming conventions: https://fivetran.com/docs/core-concepts/syncoverview/naming-conventions>

If you use them, make sure to include your .env files in your .gitignore file to ensure you don't accidentally check your FIVETRAN_API_KEY into your code repository.
You can use $My_ENV_VAR as an argument in the fivetran deploy and fivetran debug commands.

Referencing environment variables in your code

You can reference the following environment variables in your connector.py code using the os library.

FIVETRAN_CONNECTION_ID: This variable stores the unique identifier for the connection. It lets you interact with the Fivetran REST API during a sync. For example, you can retrieve or update the connector's configuration dynamically during runtime conditions. Learn more in Fivetran REST API documentation. The value of this variable is test_id when running the connector locally using fivetran debug.
FIVETRAN_GROUP_ID: This variable stores the unique identifier for the group. The value of this variable is test_group_id when running the connector locally using fivetran debug.
FIVETRAN_CONNECTION_NAME: This variable stores the name of the connection. The value of this variable is test_connection_name when running the connector locally using fivetran debug.
FIVETRAN_DEPLOYMENT_MODEL: This variable stores the deployment model of the connector. It lets you implement environment-specific logic. For example, you can use mock data when running locally, while using optimized settings for production deployments. This approach improves both development efficiency and production reliability. The supported values are as follows:
- managed_cloud: This value is returned for the SaaS Deployment model.
- hybrid_cloud: This value is returned for the Hybrid Deployment model.
- local_debug: This value is returned when running your connector.py locally using fivetran debug.

For example, to access the environment variable FIVETRAN_CONNECTION_ID, you can reference it in your code as follows:

import os
connection_id = os.getenv('FIVETRAN_CONNECTION_ID')
print(f"Connection ID: {connection_id}")