Building Custom Connectors with AI Agents
AI coding agents let you build, test, and deploy Fivetran Connector SDK connectors using natural language prompts, without writing boilerplate code or running CLI commands manually.
Fivetran provides Connector SDK-specific tools to AI coding agents so that they work more effectively for connector development. The connector_sdk_tools repository hosts plugins that integrate with various coding agents, enabling agents to scaffold connectors, run them locally, troubleshoot issues, assess production readiness, and deploy them to Fivetran. We offer plugins for Claude Code, Codex CLI, Gemini CLI, and GitHub Copilot CLI, and an IDE plugin for GitHub Copilot in VS Code and JetBrains.
We recommend reviewing AI-generated connector code before deploying to production. For more information, see What the agent cannot do.
What the agent can do
Your agent handles the full connector development lifecycle through skills that the plugin provides.
| Phase | Skill | What the agent does |
|---|---|---|
| Build | build-connector | Checks whether Fivetran already has a managed connector for your source; if not, researches your source, scaffolds a project with fivetran init, generates connector.py, configuration.json, requirements.txt, and README.md, runs a local test, and attempts to auto-fix any code errors |
| Test | test-connector | Runs the connector locally, inspects the output database, reports tables and row counts, and classifies any errors |
| Evaluate | evaluate-connector | Analyzes your connector code for SDK compliance, security, and reliability issues; produces a scored report with REQUIRED and GOOD_TO_HAVE categories |
| Deploy | deploy-connector | Validates the connector, runs a final local test, then deploys to your Fivetran account; the new connection starts paused |
| Migrate | migrate-functions-connectormigrate-meltano-connectormigrate-airbyte-connector | Ports an existing connector from Fivetran Functions (AWS Lambda, Azure Functions, Google Cloud Functions), Meltano/Singer, or Airbyte to Connector SDK |
For fixes and code changes, describe the problem or change in plain language and the agent handles it.
Installing the plugin
Quickest path: If you are starting a new project, run fivetran init from a new project directory. The wizard detects which agent you have installed and configures the plugin:
fivetran init
Manual install: For an existing project, or if you skipped agent setup in fivetran init, follow the install instructions for your agent in the Connector SDK Tools repository. The repository covers install and update commands for Claude Code, Codex CLI, Gemini CLI, GitHub Copilot CLI, and IDE-integrated Copilot, as well as additional setup steps such as configuring your Fivetran API key for deployment.
GitHub Copilot in IDEs such as VS Code and JetBrains doesn't support installation through fivetran init or the CLI plugin commands. Install it from the source repository using your IDE's agent plugin guide.
Working with your agent
Invoking skills
Your AI agent uses the appropriate skill automatically based on your prompt. However, you can also invoke it explicitly. To invoke skills manually, type the skill command followed by your prompt in the agent chat.
In Claude Code, Gemini CLI, and GitHub Copilot CLI:
/fivetran-connector-sdk:build-connector Build a connector for the Example API. The base URL is https://api.example.com/v2. Auth uses an API key. Replicate the users and events endpoints.
For Codex CLI, use $skill_name in snake_case, for example $build_connector. In IDEs, access skills through the agent panel.
Giving the agent context
For all skills, the more specific you are in your initial prompt, the fewer clarifying questions the agent needs to ask and the more accurate the result. For build-connector specifically, include the following:
- A link to the API documentation.
- Details about how to connect to your source.
- What data to extract.
- What a sample record looks like. Without sample data, the agent infers table structure from documentation alone, which can lead to incorrect assumptions about nested fields or data types.
For context guidance on building and extending connectors, see the common workflows expandables sections below.
Understanding Connector SDK development concepts such as state management, schema, pagination, and authentication patterns helps you write more precise prompts.
The agent retains context across skill invocations within a session. If any skill produces unexpected results, describe the problem in the same session and the agent picks up where it left off.
Common workflows
Build a new connector
Invoke build-connector with context about your data source in the prompt. The agent checks whether Fivetran already has a managed or Lite Connector for your source before proceeding to custom code generation. If none exists, it researches your source, scaffolds the project, and runs a local test.
/fivetran-connector-sdk:build-connector
Build a connector for the Example API. The base URL is https://api.example.com/v2.
Auth uses an API key passed in the header as X-API-Key.
Replicate the users and events endpoints with incremental sync.
Sample response (GET /users):
{"data": [{"id": "u1", "email": "user@example.com", "created_at": "2024-01-01T00:00:00Z"}], "next_cursor": "abc123"}
If Fivetran already has a managed connector for your source, the agent tells you and offers to stop. Tell it to proceed if you need a custom connector.
Before the local test runs, the agent asks you to enter credentials in a separate terminal. Run the command it provides. Credentials are never passed through the agent chat.
Extend an existing connector
Extending an existing connector requires rebuilding it from scratch. Describe in plain language the customization you want:
Add a new table for the /reports endpoint. Primary key is id, incremental cursor is created_at.
Sample response: {"id": "r1", "name": "Q1 Report", "created_at": "2024-01-01", "status": "published"}
If you change a table's primary key or column types, the destination table must be dropped and resynced. If you change the state or cursor structure, existing checkpoints won't align with the new shape and a resync may be needed. The agent flags both before making changes. Adding new tables or columns does not require a resync.
Debug an issue
Describe the error or unexpected behavior in natural language:
The connector runs without errors but the events table only has 3 rows.
It should have hundreds. Here is the debug output: [paste output]
The agent classifies the error into one of three types:
| Error type | What it means | What the agent does |
|---|---|---|
INFRA | Network failure, JVM crash, or infrastructure issue unrelated to your code | Explains the issue; makes no code changes |
FIRST_RUN | Failure on first execution, typically a credential or configuration problem | Explains it's likely a configuration issue and asks you to verify your credentials and config values; makes no code changes |
CODE | Logic bug: wrong pagination, incorrect cursor, bad data type, missing field | Asks if you'd like it to fix the issue; if yes, edits connector.py and reruns the test |
For issues that persist after the agent's fix attempt, see the Connector SDK Troubleshooting guide.
Evaluate connector quality
We recommend running evaluate-connector before deploying to production. The skill performs a read-only static analysis of your connector code and does not modify any files.
/fivetran-connector-sdk:evaluate-connector
The report groups issues into two categories:
required— fix these before deploying:Category Examples Memory Loading an entire dataset into memory before processing; unclosed file handles or HTTP connections SDK compliance Missing op.checkpoint(); usingyield op.upsert()instead of calling it directly;update()has a return statement;connector = Connector(...)not at module scope; usingprint()orlogging.*instead of the SDK loggerSecurity Credentials written to state or logs; hardcoded secrets in code Data reliability Missing raise_for_status()after API calls; cursor updated beforeop.upsert()(it must be updated after); no pagination handlingException handling Bare except: passblocks; silently caught exceptions; missing error handling around I/O operationsgood-to-have— recommended improvements:Category Examples Performance No timeouton HTTP requests; no retry logic for transient failuresCode quality requestsorfivetran_connector_sdklisted inrequirements.txt(both are pre-installed in the SDK runtime environment); all columns explicitly typed (prevents Fivetran from inferring types automatically when new columns appear)Reliability Retry logic without exponential backoff; string timestamp comparison without datetime parsing
The report produces four scores, each starting at 100: overall, SDK adherence, required, and good-to-have. required issues deduct 10–30 points each; good-to-have issues deduct 1–5. Fix all required issues before running deploy-connector.
Migrate an existing connector
The plugin includes three migration skills. Each one reads your source connector, maps its concepts to Connector SDK equivalents, generates new connector.py, configuration.json, requirements.txt, and README.md files, and runs a local test.
Migration skills are not deterministic converters. Read the generated code and compare it against the original logic, particularly for authentication, pagination, and state handling, before running the connector against your data.
Migrate a Fivetran Functions connector
Fivetran Functions connectors run on AWS Lambda, Azure Functions, or Google Cloud Functions. The migration removes cloud provider handler code and replaces it with standard Connector SDK patterns.
/fivetran-connector-sdk:migrate-functions-connector Migrate my Lambda connector at ./my-lambda-connector to Connector SDK.
Key concept mapping:
| Functions connector | Connector SDK |
|---|---|
| Cloud provider handler function | Removed; replaced with connector = Connector(update=update, schema=schema) |
request.secrets | configuration dict parameter |
request.state | state dict parameter |
insert[table] records | op.upsert(table=table, data=record) |
delete[table] records | op.delete(table=table, keys={...}) |
softDelete table list | op.truncate(table=table) per table |
Returned state | op.checkpoint(state=new_state) |
hasMore: true | Internal loop with checkpoint inside update() |
Migrate a Meltano or Singer connector
/fivetran-connector-sdk:migrate-meltano-connector Migrate my Meltano extractor at ./tap-example to Connector SDK.
The skill ports extractors and Singer taps only. It does not port loaders, dbt transforms, schedules, or Meltano environments.
Key concept mapping:
| Meltano / Singer | Connector SDK |
|---|---|
meltano.yml extractor settings | Flat string fields in configuration.json |
| Singer catalog stream | schema(configuration) table entry |
key_properties / table-key-properties | primary_key |
Singer RECORD message | op.upsert(table=stream, data=record) |
Singer STATE bookmark | op.checkpoint(state=new_state) |
| Incremental replication key | Cursor logic in update() |
For full-table streams, the agent asks how you want to handle replacement semantics: upsert-only snapshot, or op.truncate() followed by a full reload. Answer before the agent writes code so it implements the correct pattern.
Migrate an Airbyte connector
/fivetran-connector-sdk:migrate-airbyte-connector Migrate my Airbyte source connector at ./source-example to Connector SDK.
The skill handles Python CDK connectors and low-code YAML manifests directly. Java connectors are inventoried from their spec and catalog artifacts, but require reimplementation from source API documentation. It does not port destinations, normalization, Docker setup, or Airbyte workspace schedules.
Key concept mapping:
| Airbyte | Connector SDK |
|---|---|
spec.json / connectionSpecification | Flat string fields in configuration.json |
airbyte_secret: true fields | Entered via the credential tool; never in plaintext |
check command | validate_configuration(configuration) |
discover catalog stream | schema(configuration) table entry |
RECORD message | op.upsert(table=stream, data=record) |
STATE message | op.checkpoint(state=new_state) |
incremental sync mode | Cursor logic in update() |
full_refresh + overwrite | op.truncate(table=table) then upsert |
append_dedup | op.upsert(...) with declared primary key |
Append-only streams (streams without a primary key in the Airbyte connector) require an explicit primary key strategy. The agent flags these before writing code and asks how you want to handle them.
Deploy a connector
Set FIVETRAN_API_KEY as an environment variable before deploying. The value must be base64-encoded {key}:{secret} with "manage connections, read destinations" permissions. See the Connector SDK Tools repository for setup instructions.
Before deploying, ensure:
- At least one destination is configured in your Fivetran account
README.mdexists in your connector directory alongsideconnector.py,configuration.json, andrequirements.txt
/fivetran-connector-sdk:deploy-connector
The agent validates the connector, runs a final local test, and deploys to your Fivetran account. The connection name defaults to your connector directory name. The new connection starts paused. The agent asks before starting the initial sync, as it consumes MAR.
To redeploy an updated connector, specify the existing connection name:
/fivetran-connector-sdk:deploy-connector Deploy to the same connection as before. The connection name is my_connector.
Prompt examples
Build prompts
Start from an OpenAPI spec:
/fivetran-connector-sdk:build-connector Build a connector for <api_name> using the OpenAPI spec at <openapi_spec_url>. Replicate the <resource_1> and <resource_2> endpoints. Primary key for <resource_1> is <primary_key_field>. Primary key for <resource_2> is <primary_key_field>. Use incremental sync with <cursor_field> as the cursor.
OAuth 2.0 client credentials:
/fivetran-connector-sdk:build-connector
Build a connector for <api_name>. Auth uses the OAuth 2.0 client credentials grant.
Don't use refresh_token. Write custom token acquisition code.
POST to <token_endpoint_url> with <client_id>, <client_secret>, and <scope> as a space-separated string.
Response includes <access_token> and <expires_in>.
Use <expires_in> for proactive token refresh.
Replicate:
- GET [endpoint_path_1] — primary key: [field], incremental sync on [cursor_field], [pagination_type] pagination
- GET [endpoint_path_2] — primary key: [field], full refresh
Debug prompts
Rate limit errors:
The connector gets 429 errors after the first few requests.
The API allows [N] requests per minute. Add retry logic with exponential backoff.
Here is the debug output: [paste debug output]
Pagination not working:
The connector only returns the first page of results. The API uses [pagination type: cursor | offset | page_token] pagination. Here is the relevant response structure: [paste response structure] Here is the debug output: [paste debug output]
Add a missing column:
The [table] table is missing the [field] column. It is returned in the API response but not currently included in the schema. Add it without changing any existing column types.
Anti-patterns
Instead of specifying every column type:
// Don't do this Generate the full schema with data types for all columns in the users table.
// Do this Define the schema for the users table. Only declare the primary key ([primary_key_field]). Let Fivetran infer the rest.
Specifying every column type removes Fivetran's ability to adapt automatically when the source schema changes and new columns appear.
Instead of pasting credentials in chat:
// Don't do this Here is my API key: <api_key>. Build the connector.
// Do this Build the connector. I'll enter credentials in a separate terminal when prompted.
The plugin enforces credential entry through a separate local process. Credentials never enter the agent conversation.
What the agent cannot do
- Private or undocumented APIs: The agent cannot access internal or undocumented APIs on its own. You must provide endpoint details and sample responses before running
build-connector. - Local testing only: The agent tests your connector on your machine. It can't run the connector in the Fivetran cloud environment or diagnose post-deployment sync failures.
- Migration skills are not one-shot converters: Complex connectors with custom pagination, non-standard authentication, or highly stateful sync logic require human review and iteration after the migration skill runs.
- Source connectors only: The plugin builds source connectors. It cannot build destination connectors.