How Can I Export Fivetran Connection Configuration to GitHub?

Question

How can I export Fivetran connection, destination, and schema configurations to a GitHub repository?

Environment

Fivetran REST API

Answer

To manage your Fivetran configuration as code, we recommend the Fivetran Provider for Terraform. You can define settings in configuration files, keep them in GitHub, and apply changes through your CI/CD pipeline.

To export a snapshot of your current configuration using the Fivetran REST API, complete the following steps. Replace the placeholders with your API credentials and resource IDs.

1. Retrieve your connections using the List All Connections endpoint. If the response contains a next_cursor value, pass it in the cursor query parameter to get the next page of results:

   curl -u "<api_key>:<api_secret>" "https://api.fivetran.com/v1/connections" -o connections.json
   

2. For each connection, retrieve its configuration using the Retrieve Connection Details endpoint and its schema settings using the Retrieve a Connection Schema Config endpoint:

   curl -u "<api_key>:<api_secret>" "https://api.fivetran.com/v1/connections/<connection_id>" -o connection-<connection_id>.json
   curl -u "<api_key>:<api_secret>" "https://api.fivetran.com/v1/connections/<connection_id>/schemas" -o schema-<connection_id>.json
   

3. Retrieve your destinations using the List All Destinations Within Account endpoint. Then, retrieve each destination's configuration using the Retrieve Destination Details endpoint:

   curl -u "<api_key>:<api_secret>" "https://api.fivetran.com/v1/destinations/<destination_id>" -o destination-<destination_id>.json
   

4. Review the saved files and remove any credentials or other sensitive values.
5. Commit the files to your GitHub repository.

To keep the snapshot current, automate these steps with a scheduled export script or GitHub Actions workflow. For more information about API keys and authentication, see Getting Started with the Fivetran REST API.