Connector SDK Beta
Need to get your connector up and running quickly?
Our team of Professional Services experts is available to provide free advisory services to help you build your first Connector SDK connection. This includes guidance on setup, troubleshooting, and best practices. To get started, simply file a support ticket.
Save time nowFivetran’s Connector SDK allows you to code a custom data connector using Python and deploy it as an extension of Fivetran. Fivetran automatically manages running Connector SDK connections on your scheduled frequency and manages the required compute resources, eliminating the need for a third-party provider.
Connector SDK provides native support for many Fivetran features and relies on existing Fivetran technology. It also eliminates timeout and data size limitations seen in AWS Lambda.
Set up your custom connector
When building a data pipeline using the Connector SDK, given that all prerequisites are in place, the high-level end state workflow is as follows:
- You write code to extract data from your source and pass to the Fivetran Connector SDK.
- Test your code locally.
- Deploy the code to Fivetran to create your connection.
- When running a sync, Fivetran runs your custom code so that it can extract data from your source and send it to Fivetran.
- Fivetran then loads the data into your destination.
Follow our step-by-step Connector SDK setup guide and visit our Connector SDK GitHub Repo to create a custom source connector and connect it with your destination using Fivetran.
Features
Feature Name | Supported | Notes |
---|---|---|
Capture deletes | check | |
History mode | ||
Custom data | check | |
Data blocking | check | |
Column hashing | check | |
Re-sync | check | |
API configurable | check | API configuration |
Priority-first sync | ||
Fivetran data models | ||
Private networking | ||
Authorization via API |
NOTE: You cannot create a connection using Fivetran REST API.
The Connector SDK supports many Fivetran features natively, as if the connector was developed by Fivetran, including:
- Column blocking
- Column hashing
- Data de-deduplication
- Destination ingestion optimizations
- Any connection method supported by Python libraries that you can include in your Python
requirement.txt
file
The following features are available for implementation in your custom code, but only if the source allows it:
- Incremental updates
- Updating and upserting data operations and soft deletion of data
- Table-level re-sync
- Multi-threading for fast extraction of data from the source
- Customizing the data schema delivered to your destination
- Defining data types in the schema used to deliver data to your destination
- Source data type inference. This applies only if you are not defining the data type to be used for a column in your code.
- Automatic schema updates. See our changing data type documentation for more information.
- Controlling the state and connector-level re-syncs of the connection
- Authenticating with OAuth 2.0 Client Credentials flow
Use cases
The Connector SDK service is a good fit for the following use cases:
- Fivetran doesn't have a connector for your source or is unlikely to support it in the near future.
- You are using private APIs, custom applications, unsupported file formats or those that require pre-processing.
- You have sensitive data that needs filtering or anonymizing before entering the destination.
- You don't want to introduce a third party into your data pipeline (e.g., to host a custom function).
Architecture
The following diagram shows a high-level overview of how connectors built using the Fivetran Connector SDK work:
The Connector SDK runtime environment executes your submitted Python script in an isolated environment. It installs the libraries listed in the requirements.txt
file, and after installation, it runs the provided Python script and we pre-install the msodbcsql17
and msodbcsql18
drivers for connecting to Microsoft SQL server. We also pre-install libpq5 and libpq-dev, which is required to support psycopg and psycopg2, respectively.
By default, we host and run your Connector SDK connection in GCP, Fivetran's default infrastructure provider, in the region that you have configured in your destination. If you are on our Enterprise or Business Critical plan, you have the option to choose a cloud service provider and cloud region. In that case, we will host and run your code in the cloud provider and region you selected for the rest of your Fivetran infrastructure.
Secrets management
Configuration values are stored securely and encrypted in the same way as other Fivetran connectors' credentials. See our Data Credential Encryption documentation for more details. To pass configuration values to Fivetran for use in your code, you must upload them using the configuration.json
file. Fivetran does not store this file and we suggest that you delete it from your project after deploying a Connector SDK project. Fivetran reads the values, encrypts them, and stores them securely. The values are only available to your code at runtime. You can confidently use configuration.json
to pass any credentials required to run your connection, like a key required to make an API call.
Support for hosted Python environments
You can use a hosted Python environment, such as Jupyter Notebook, to test Python libraries and approaches to validate how you are going to connect to your data source and manipulate the response data ready for sending to Fivetran.
However, you need to send the actual connector.py
file to Fivetran because we execute it in our infrastructure each time a sync is triggered. As such, it needs to be a standalone Python script or a set of scripts that are called from your connector.py
file.
NOTE: See Technical Reference for a list of supported methods, operations, and
fivetran-connector-sdk
commands.
Connector SDK vs. Functions vs. Lite connectors
Lite connectors are API-based connectors that Fivetran builds by request. They are generally for SaaS services that produce a non-dynamic schema.
In contrast, you or your engineering team build connectors using a cloud function service or the Connector SDK and then deploy them to run either on the cloud service infrastructure or on top of the Fivetran platform.
The following table summarizes the key differences between connectors built in our By Request program, using a Cloud Function, and using Connector SDK:
ASPECT | By Request (LITE) | Cloud Function | Connector SDK |
---|---|---|---|
Use cases | Complete or near-complete use case coverage | Customer-decided use case. Business logic triggers, Source-Side filtering, etc. | Customer-decided use case. Custom solution hosted by Fivetran |
Release cycle | Private Preview, then Generally Available. No Beta phase | Live upon deployment. Setup can't be fully automated | Live upon deployment. Setup can be fully automated |
SLA | Full SLA based on pricing plan | Customer-driven SLAs | Customer-driven SLAs |
Hosting or requirement for the 3rd party to run the code you write. | Fivetran | The 3rd party runs the code | Fivetran |
Supported Language | Not applicable since Fivetran writes the code | Language options are defined by the 3rd party | Python |
Connector SDK vs. Partner-Built connectors
Partner-Built connectors are built by Fivetran partners using our Partner SDK and are available for all Fivetran customers. The Partner SDK supports the following types of connectors:
You build custom connectors using the Connector SDK, which are only available for you to use. Our Connector SDK only supports building source connectors.
Limitations
The Connector SDK doesn't support the following features:
- Webhooks for receiving event data
- History Mode
- Table-level re-syncs from the Fivetran dashboard
- Hybrid Deployment
- Powered by Fivetran
- Private Networking
- Authorization via API
- Fivetran data models
Support
Our Support Team triages all tickets submitted though the Support Portal:
- If the issue is with the Fivetran platform, our Support Team will work with you on the issue until it is resolved.
- If the issue is with the connection’s custom code or logic, our Professional Services Team can assist you. See Custom connector development support for more information.
NOTE: The scope of our Customer Support Team's assistance is limited when it comes to troubleshooting custom code issues.
Custom connector development support
If you need assistance in developing a custom Connector SDK connector, our Services team can help. Contact your Account Representative to use existing service hours you may already have or submit a support request to get started with our Services team.
Usage and pricing
Connector SDK is available on all pricing plans. It uses Monthly Active Rows (MAR) to calculate usage, just like any other Fivetran connector.
To calculate the MAR for a table in your Connector SDK, see How to Count Daily MAR Usage per Table.