Write to an Iceberg Catalog

fivetran run runs your connector the way fivetran debug does, with the same options and the same logs. The difference is where the data goes: instead of the local files/warehouse.db file, fivetran run writes to an Apache Iceberg REST catalog that you describe in a destination.json file. The data goes to your catalog, not to Fivetran.

Use fivetran run to see how your connector's tables, types, and keys look in an Iceberg catalog, and to test the connector against the catalog and storage you use.

Before you start

You need:

• An Iceberg REST catalog that your machine can reach and that signs you in with a token or OAuth2 client credentials, and the storage behind it, which is Amazon S3 or S3-compatible storage.
• Credentials for the catalog, and for the storage unless the catalog vends credentials.
• A namespace used only for your connector. fivetran run does not ask for confirmation, and it adds columns to, rewrites, and merges rows into any table with the same name that is already in the namespace.
• Internet access on the first run. DuckDB, which fivetran run uses to write the data, downloads its Iceberg extension the first time it is used.

Describe the catalog

fivetran run reads the catalog configuration from a destination.json file in your project folder, beside connector.py. The file is a flat JSON object that uses the same setting names as Iceberg clients such as PyIceberg and Spark. namespace is the only Fivetran-specific setting.

Which example do I need?

Choose an example based on whether your catalog provides credentials for its storage.

• It does (vended credentials). Use the first example. You give the storage region and no storage keys.
• It does not. Use the second example. You give the storage settings yourself.

For a catalog that vends storage credentials:

{
  "uri": "https://my-catalog.example.com/api/catalog",
  "warehouse": "my_warehouse",
  "namespace": "my_connector",
  "client_id": "<your client ID>",
  "client_secret": "<your client secret>",
  "header.X-Iceberg-Access-Delegation": "vended-credentials",
  "s3.region": "us-east-1"
}

If you sign in with a token, replace client_id and client_secret with "token": "<your token>".

For a catalog that does not vend credentials, give it the storage as well:

{
  "uri": "http://localhost:8181/api/catalog",
  "warehouse": "local_catalog",
  "namespace": "tester",
  "client_id": "<your client ID>",
  "client_secret": "<your client secret>",
  "s3.region": "us-east-1",
  "s3.endpoint": "http://localhost:8333",
  "s3.force-virtual-addressing": false,
  "s3.access-key-id": "<your access key ID>",
  "s3.secret-access-key": "<your secret access key>"
}

Settings

The catalog:

How fivetran run signs in to the catalog. Give a token, or a client ID and secret, not both:

How fivetran run reaches the storage:

Run your connector against a catalog

With destination.json in your project folder, run:

fivetran run

fivetran run reads the same files/state.json as fivetran debug. Before the first run against a new catalog or namespace, run fivetran reset, or replace the contents of your state file with {}. Otherwise the connector starts from the cursor of an earlier local run and the catalog does not get the earlier records.

To use a destination file that is not in the project folder, or has another name, use --destination-file:

fivetran run --destination-file ~/catalogs/my_catalog.json

For the options, see fivetran run.

A successful run ends with the same summary as a fivetran debug run, followed by where the data is and how to read it:

SYNC SUCCEEDED — total elapsed 00:00:26.880
Operation       | Counts
----------------+------------
Upserts         | 1
Updates         | 0
Deletes         | 0
Truncates       | 0
Schema changes  | 0
Checkpoints     | 1
File uploads    | 0

results are in the Iceberg catalog at https://my-catalog.example.com/api/catalog: warehouse my_warehouse, namespace my_connector. To read them in DuckDB, with the iceberg extension and a secret for this catalog: ATTACH 'my_warehouse' AS lake (TYPE ICEBERG, ENDPOINT 'https://my-catalog.example.com/api/catalog'); SELECT * FROM lake."my_connector".<table>

If your connector writes to more than one schema, the last line lists all the namespaces it wrote to.

Where your data goes

• Tables. A table your connector writes becomes an Iceberg table in the namespace. It has the same columns as in a fivetran debug run, including the system columns.
• Schemas. A table with no declared schema goes in namespace. With Fivetran naming, a table that declares a schema goes in <namespace>_<schema> (lower-cased, with characters a name cannot hold replaced by _); with source naming, it goes in a namespace named exactly as declared. For example, with the namespace my_connector, a table in the schema Sales goes in my_connector_sales with Fivetran naming. Fivetran creates namespaces that don't exist.
• Naming strategy. The naming strategy is stored with the namespace, so subsequent runs from any machine use the same strategy. A later --naming that differs is ignored, with a warning.
• Primary keys. The key columns are required in the Iceberg table, and rows are merged on the full key. The key columns are also named as the table's identifier fields, unless a key column is a FLOAT or DOUBLE. Then the table has no identifier fields, and other Iceberg clients cannot discover the key from the table.
• Deletes. Like in a local run, delete() and truncate() set _fivetran_deleted to true. They do not remove rows.

What is not supported

• File uploads. A connector that passes file to upsert() or update() is refused.
• Table and namespace names that have characters other than ASCII. The Iceberg extension of DuckDB cannot write them. Fivetran naming converts the names of tables and schemas, but not the namespace in destination.json, so choose an ASCII namespace with either naming strategy.
• Catalogs that need no sign-in, or that sign requests another way, for example with AWS Signature Version 4. fivetran run needs a token or client credentials, and sends no other request headers than the access delegation header.
• Storage that is not Amazon S3 or S3-compatible, for example Google Cloud Storage or Azure Data Lake Storage.
• Compaction. Data files and delete files pile up: use the maintenance your catalog offers.
• Several syncs writing to one table at the same time.

Errors

A problem with destination.json stops the command before your connector runs:

If the catalog cannot be reached, the run fails with could not connect to the Iceberg catalog at <uri> and the reason.

Read the results

Open the catalog with any Iceberg client. This example uses DuckDB with client credentials, for a catalog that vends storage credentials:

INSTALL iceberg;
LOAD iceberg;
CREATE SECRET lake_secret (
    TYPE ICEBERG,
    CLIENT_ID '<your client ID>',
    CLIENT_SECRET '<your client secret>',
    OAUTH2_SCOPE '<scope>',
    OAUTH2_SERVER_URI 'https://my-catalog.example.com/api/catalog/v1/oauth/tokens'
);
ATTACH 'my_warehouse' AS lake (
    TYPE ICEBERG,
    SECRET lake_secret,
    ENDPOINT 'https://my-catalog.example.com/api/catalog',
    ACCESS_DELEGATION_MODE 'vended_credentials',
    DEFAULT_REGION 'us-east-1'
);
SELECT * FROM lake."my_connector".<table>;

If you sign in with a token, give TOKEN '<your token>' in the secret instead of the client settings. For a catalog that does not vend credentials, set ACCESS_DELEGATION_MODE 'none' and add a DuckDB secret for your storage. See the DuckDB Iceberg documentation.