Write to an Iceberg Catalog

You can use fivetran run to write your Connector SDK connector's data directly to an Apache Iceberg REST catalog to test how your tables, data types, and primary keys are represented in Iceberg. The command works similarly to fivetran debug, with the same options and logs, but writes the data directly to your Iceberg catalog instead of the local files/warehouse.db file.

Prerequisites

• An Iceberg REST catalog that your machine can reach and that signs you in with a token or OAuth2 client credentials, and the storage behind it, which is Amazon S3 or S3-compatible storage.
• Credentials for the catalog, and for the storage unless the catalog vends credentials.
• A namespace used only for your connector. fivetran run does not ask for confirmation, and it adds columns to, rewrites, and merges rows into any table with the same name that is already in the namespace.
• Internet access on the first run. DuckDB, which fivetran run uses to write the data, downloads its Iceberg extension the first time it is used.

Describe the catalog

fivetran run reads the catalog configuration from a destination.json file in your project folder, beside connector.py. The file is a flat JSON object that uses the same setting names as Iceberg clients such as PyIceberg and Spark. namespace is the only Fivetran-specific setting.

Manage storage credentials

For a catalog that vends storage credentials, provide the storage region and no storage keys:

{
  "uri": "https://my-catalog.example.com/api/catalog",
  "warehouse": "my_warehouse",
  "namespace": "my_connector",
  "client_id": "<your client ID>",
  "client_secret": "<your client secret>",
  "header.X-Iceberg-Access-Delegation": "vended-credentials",
  "s3.region": "us-east-1"
}

If you sign in with a token, replace client_id and client_secret with "token": "<your token>".

For a catalog that does not vend credentials, provide it with storage as well:

{
  "uri": "http://localhost:8181/api/catalog",
  "warehouse": "local_catalog",
  "namespace": "tester",
  "client_id": "<your client ID>",
  "client_secret": "<your client secret>",
  "s3.region": "us-east-1",
  "s3.endpoint": "http://localhost:8333",
  "s3.force-virtual-addressing": false,
  "s3.access-key-id": "<your access key ID>",
  "s3.secret-access-key": "<your secret access key>"
}

Specify catalog settings

Provide catalog details:

Now, configure how fivetran run signs in to your catalog. Provide a token, or a client ID and secret, not both:

Configure how fivetran run reaches the storage:

Run your connector against a catalog

With destination.json in your project folder, run:

fivetran run

fivetran run reads the same files/state.json as fivetran debug. Before the first run against a new catalog or namespace, run fivetran reset, or replace the contents of your state file with {}. Otherwise the connector starts from the cursor of an earlier local run and the catalog does not get the earlier records.

To use a destination file that is not in the project folder, or has another name, use --destination-file:

fivetran run --destination-file ~/catalogs/my_catalog.json

For the options, see fivetran run.

A successful run ends with the same summary as a fivetran debug run, followed by where the data is and how to read it:

SYNC SUCCEEDED — total elapsed 00:00:26.880
Operation       | Counts
----------------+------------
Upserts         | 1
Updates         | 0
Deletes         | 0
Truncates       | 0
Schema changes  | 0
Checkpoints     | 1
File uploads    | 0

results are in the Iceberg catalog at https://my-catalog.example.com/api/catalog: warehouse my_warehouse, namespace my_connector. To read them in DuckDB, with the iceberg extension and a secret for this catalog: ATTACH 'my_warehouse' AS lake (TYPE ICEBERG, ENDPOINT 'https://my-catalog.example.com/api/catalog'); SELECT * FROM lake."my_connector".<table>

If your connector writes to more than one schema, the last line lists all the namespaces it wrote to.

Where your data goes

• Tables. A table your connector writes becomes an Iceberg table in the namespace. It has the same columns as in a fivetran debug run, including the system columns.
• Schemas. A table with no declared schema goes in namespace. With Fivetran naming, a table that declares a schema goes in <namespace>_<schema> (lower-cased, with characters a name cannot hold replaced by _); with source naming, it goes in a namespace named exactly as declared. For example, with the namespace my_connector, a table in the schema Sales goes in my_connector_sales with Fivetran naming. Fivetran creates namespaces that don't exist.
• Naming strategy. The naming strategy is stored with the namespace, so subsequent runs from any machine use the same strategy. A later --naming that differs is ignored, with a warning.
• Primary keys. The key columns are required in the Iceberg table, and rows are merged on the full key. The key columns are also named as the table's identifier fields, unless a key column is a FLOAT or DOUBLE. Then the table has no identifier fields, and other Iceberg clients cannot discover the key from the table.
• Deletes. Like in a local run, delete() and truncate() set _fivetran_deleted to true. They do not remove rows.

Read the results

Open the catalog with any Iceberg client. This example uses DuckDB with client credentials, for a catalog that vends storage credentials:

INSTALL iceberg;
LOAD iceberg;
CREATE SECRET lake_secret (
    TYPE ICEBERG,
    CLIENT_ID '<your client ID>',
    CLIENT_SECRET '<your client secret>',
    OAUTH2_SCOPE '<scope>',
    OAUTH2_SERVER_URI 'https://my-catalog.example.com/api/catalog/v1/oauth/tokens'
);
ATTACH 'my_warehouse' AS lake (
    TYPE ICEBERG,
    SECRET lake_secret,
    ENDPOINT 'https://my-catalog.example.com/api/catalog',
    ACCESS_DELEGATION_MODE 'vended_credentials',
    DEFAULT_REGION 'us-east-1'
);
SELECT * FROM lake."my_connector".<table>;

If you sign in with a token, give TOKEN '<your token>' in the secret instead of the client settings. For a catalog that does not vend credentials, set ACCESS_DELEGATION_MODE 'none' and add a DuckDB secret for your storage. See the DuckDB Iceberg documentation.

Errors

If destination.json is missing or invalid, fivetran run fails before running your connector. The following table lists common errors and how to resolve them.

If the catalog cannot be reached, the fivetran run command fails with could not connect to the Iceberg catalog at <uri> and the reason.

Limitations

• The upsert() and update() operations don't support file uploads using the file parameter.
• The Iceberg extension for DuckDB can't write table or namespace names that contain non-ASCII characters. The FIVETRAN naming strategy converts table and schema names to ASCII, but doesn't convert the namespace value in destination.json. Use an ASCII-only namespace value regardless of your naming strategy.
• fivetran run supports only token-based authentication or client credentials, and doesn't send additional request headers. Catalogs that require no authentication, or that sign requests differently, for example, with AWS Signature Version 4, aren't supported.
• Only Amazon S3 and S3-compatible storage are supported. Google Cloud Storage and Azure Data Lake Storage aren't supported.
• Data files and delete files accumulate with each sync. Use the maintenance tools your catalog provides to compact them regularly.
• Running multiple syncs that write to the same table at the same time isn't supported.