Connector Development Concepts

To efficiently build a reliable custom connector with Connector SDK, you need to make several key decision, including authentication method, data extraction, sync strategy, pagination, error handling, and schema and state management. While some decisions, such as how to authenticate to access the data, must be made right away, others like fine-tuning data handling or pagination, can be finalized as you get your initial connector working and gain a better understanding of the specific data patterns used by the source.

This section helps you understand and prepare for each area, and highlights key decisions to consider while coding.

Fundamental principles for building custom connectors

• Authentication: How your connector successfully connects to the source to access data.
• Data Extraction and Handling: How your connector fetches data from a source system, shapes it into destination-ready records, and sends those records to Fivetran.
• Schema Management: How your connector defines the structure of the data it sends to Fivetran, including tables, columns, data types, and primary keys.
• State Management: How your connector saves sync progress, such as a timestamp or page token, so later syncs can resume without re-fetching all data.
• Pagination: How your connector fetches large datasets in smaller chunks so syncs are efficient and resumable.
• Error Handling: How your connector responds when something goes wrong during a sync.