Fivetran Platform Connector

Fivetran generates and logs several types of data related to your account and destinations:

Log events related to connections, dashboard user actions, Fivetran API calls, and schema change data.
Account- and destination-related metadata that includes:
- Role/membership information
- Column lineage (available for Enterprise and Business Critical accounts)
- Granular consumption information

You can use this data for the following purposes:

Monitoring and troubleshooting of connections
Tracking your usage
Conducting audits

The Fivetran Platform Connector delivers your logs and account or destination metadata to a schema in your destination. We automatically add a Fivetran Platform connection to every new destination you create. The schema name we use for these automatically created connections is fivetran_metadata. By default, we configure the connection to sync once a day, but you can set a different sync frequency and change other default settings on the Setup tab of the connection details page.

NOTE: We automatically add a Fivetran Platform connection to every destination, but it doesn't start syncing logs and metadata until you add a connection to your destination.

IMPORTANT: If you are an Account Administrator, you can manually add the Fivetran Platform connection on an account level so that it syncs all the metadata and logs for all the destinations in your account to a single destination. If an account-level Fivetran Platform connection is already configured in a destination in your Fivetran account, then we don't add destination-level Fivetran Platform connections to the new destinations you create.

IMPORTANT: The MAR that Fivetran Platform connections generate is free, though you may incur costs in your destination. Learn more in our pricing documentation.

Features

Feature Name	Supported	Notes
Capture deletes		`CONNECTOR`, `RESOURCE_MEMBERSHIP`, `ROLE`, `ROLE_PERMISSION`, `TEAM`, `TEAM_MEMBERSHIP`, and `USER` tables
History mode
Custom data
Data blocking		Column level
Column hashing
Re-sync		Connection level
API configurable		API configuration
Priority-first sync
Fivetran data models		Get the model; Supports Quickstart data models
Private networking
Authorization via API

Supported deployment models

We support the SaaS and Hybrid deployment models for the connector.

NOTE: You must have an Enterprise or Business Critical plan to use the Hybrid Deployment model.

Setup guide

Follow our step-by-step setup guide to manually set up your Fivetran Platform connection account-wide.

Sync overview

Most of the tables in the Fivetran Platform Connector use incremental sync so that they are updated when there is new or changed data.

For the LOG and CONNECTOR_SDK_LOG tables, new data is appended to the table. To ensure log data integrity, all connection data tables should have data present, complete, and valid following the initial sync of the relevant connection. Where the data exists, we sync the previous 7 days' worth of data prior to the date of the initial sync of the Fivetran Platform connection. This also means that when a full re-sync is triggered for the Fivetran Platform connection, the LOG and CONNECTOR_SDK_LOG tables will have 7 days' worth of historical data .

Schema information

This schema applies to all Fivetran Platform connections.

NOTE: The *_METADATA and *_LINEAGE tables are available only on the Enterprise plan and above.

Explore Fivetran Platform schema ERD

Using Fivetran Platform Connector with its data models

You can use our Fivetran Platform Connector along with the Fivetran Platform Connector data model, with each of the model tables storing specific type of data.

You can also set up our free, pre-built Quickstart models. These models are autonomously created using the logic defined in our Fivetran Platform dbt Package, and prepare the data from the Fivetran Platform Connector in analysis-ready models covering spend, connection performance and status, and metadata/data lineage. If you are an advanced dbt user, you can also use the package for increased flexibility.

Important tables

You can query the data in the different destination tables to monitor different metrics such as connection performance, destination and source metadata, and MAR usage.

INCREMENTAL_MAR table

The INCREMENTAL_MAR table provides incremental daily active rows (MAR) for each destination schema, the associated tables, and the time when the MAR is calculated.

Column Name	Description
`destination_id`	The ID of the destination for which MAR is calculated.
`free_type`	If it is free MAR, the value indicates the type of free MAR. For paid MAR,the value is `PAID`.
`incremental_rows`	The number of new distinct primary keys on the current day synced for the connection.
`connector_id`	The name of the connection for which MAR is calculated. Deprecated and replaced by `connection_name`.
`connector_name`	The name of the connection for which MAR is calculated. Deprecated and replaced by `connection_name`.
`connection_name`	The name of the connection for which MAR is calculated.
`measured_date`	The date in UTC format of when MAR is calculated.
`schema_name`	The destination schema name for which MAR is calculated.
`sync_type`	This defines whether the sync for which MAR calculated is HISTORICAL or INCREMENTAL. Currently, the available value is `UNKNOWN`.
`table_name`	The table name associated with MAR.
`_fivetran_synced`	The timestamp of when Fivetran last successfully synced the row.
`updated_at`	The timestamp of when MAR is updated.

TRANSFORMATION_RUNS table

A transformation job consists of one or several models run in sequence. The TRANSFORMATION_RUNS table provides the total number of successful transformation model runs in each job and destination schema. Additionally, it provides a timestamp of when the model run count was updated.

Column Name	Description
`destination_id`	The ID of the destination for which model runs is calculated.
`free_type`	If it is a free model run, the value indicates the type of the free run. For paid runs, the value is `PAID`.
`job_id`	The unique identifier of a transformation job.
`job_name`	The name of the transformation job.
`measured_date`	The date in UTC format of when models were run.
`model_runs`	The total number of times the models have run.
`project_type`	The type of project associated with the job: `DBT_CORE`, `QUICKSTART`.
`updated_at`	The timestamp of when the model run count was updated.
`_fivetran_synced`	The timestamp of when Fivetran last successfully synced the row.

LOG table

We write the log events to the LOG table to your destination.

Column Name	Description	Data Type
`id`	The ID of the log event (internal).	STRING
`time_stamp`	Indicates the time when the log event is created.	TIMESTAMP
`_fivetran_synced`	Indicates the time when Fivetran last successfully synced the row.	TIMESTAMP
`connector_id`	The ID of the connection for which the event is logged. Deprecated and replaced by `connection_id`.	STRING
`connection_id`	The ID of the connection for which the event is logged.	STRING
`event`	The event type. Events can be classified as a warning, an error, or just an information log.	STRING
`message_event`	The routine involved in the log.	STRING
`message_data`	The details of the event in JSON format.	STRING

CONNECTOR_SDK_LOG table

We write the Connector SDK log events to the CONNECTOR_SDK_LOG table in your destination.

Column Name	Description	Data Type
`id`	The ID of the log event (internal).	STRING
`event_time`	Indicates the time when the log event is created.	TIMESTAMP
`connector_id`	The ID of the connection for which the event is logged. Deprecated and replaced by `connection_id`.	STRING
`connection_id`	The ID of the connection for which the event is logged.	STRING
`level`	The event type. Events can be classified as a warning, an error, or just an information log.	STRING
`message`	The details of the event in plain text.	STRING
`message_origin`	The source of the log event. For example "connector_sdk".	STRING
`sync_id`	The ID of the sync durinc which the event is appeared.	STRING

SOURCE_SCHEMA_METADATA table

Column Name	Description	Data Type
`id`	Schema ID.	BIGSERIAL NOT NULL
`connector_id`	The ID of the connection for which metadata is collected. Deprecated and replaced by `connection_id`.	STRING
`connection_id`	The ID of the connection for which metadata is collected.	STRING
`name`	The name of the table.	TEXT NOT NULL
`created_at`	Creation timestamp.	TIMESTAMPTZ NOT NULL

DESTINATION_SCHEMA_METADATA table

Column Name	Description	Data Type
`id`	Schema ID.	BIGSERIAL NOT NULL
`connector_id`	The ID of the connection for which metadata is collected.	STRING
`connection_id`	The ID of the connection for which metadata is collected. Deprecated and replaced by `connection_id`.	STRING
`destination_id`	The ID of the destination for which metadata is collected.	STRING
`name`	The name of the table.	TEXT NOT NULL
`created_at`	Creation timestamp.	TIMESTAMPTZ NOT NULL

SOURCE_TABLE_METADATA table

Column Name	Description	Data Type
`id`	Table ID.	BIGSERIAL NOT NULL
`schema_id`	ID of the corresponding schema.	SERIAL NOT NULL
`name`	The name of the table.	TEXT NOT NULL
`connector_id`	The ID of the connection for which metadata is collected. Deprecated and replaced by `connection_id`.	STRING
`connection_id`	The ID of the connection for which metadata is collected.	STRING
`created_at`	Creation timestamp.	TIMESTAMPTZ NOT NULL

DESTINATION_TABLE_METADATA table

Column Name	Description	Data Type
`id`	Table ID.	BIGSERIAL NOT NULL
`schema_id`	ID of the corresponding schema.	SERIAL NOT NULL
`name`	The name of the table.	TEXT NOT NULL
`connector_id`	The ID of the connection for which metadata is collected. Deprecated and replaced by `connection_id`.	STRING
`connection_id`	The ID of the connection for which metadata is collected.	STRING
`destination_id`	The ID of the destination for which metadata is collected.	STRING
`created_at`	Creation timestamp.	TIMESTAMPTZ NOT NULL

SOURCE_COLUMN_METADATA table

Column Name	Description	Data Type
`id`	Column ID.	BIGSERIAL NOT NULL
`table_id`	The ID of the corresponding table.	BIGSERIAL NOT NULL
`name`	The name of the column.	TEXT NOT NULL
`type`	Column type (e.g., `STRING`).	TEXT NOT NULL
`is_primary_key`	A flag indicating if this column is part of the primary key.	BOOLEAN NOT NULL
`connector_id`	The ID of the connection for which metadata is collected. Deprecated and replaced by `connection_id`.	STRING
`connection_id`	The ID of the connection for which metadata is collected.	STRING
`created_at`	Creation timestamp.	TIMESTAMPTZ NOT NULL
`updated_at`	Modification timestamp.	TIMESTAMPTZ NOT NULL

DESTINATION_COLUMN_METADATA table

Column Name	Description	Data Type
`id`	Column ID.	BIGSERIAL NOT NULL
`table_id`	The ID of the corresponding table.	BIGSERIAL NOT NULL
`name`	The name of the column.	TEXT NOT NULL
`type`	Column type (e.g., `STRING`).	TEXT NOT NULL
`connector_id`	The ID of the connection for which metadata is collected. Deprecated and replaced by `connection_id`.	STRING
`connection_id`	The ID of the connection for which metadata is collected.	STRING
`destination_id`	The ID of the destination for which metadata is collected.	STRING
`created_at`	Creation timestamp.	TIMESTAMPTZ NOT NULL
`updated_at`	Modification timestamp.	TIMESTAMPTZ NOT NULL

SOURCE_FOREIGN_KEY_METADATA table

Column Name	Description	Data Type
`id`	Foreign key ID.	BIGSERIAL NOT NULL
`column_id`	The ID of the corresponding column.	BIGSERIAL NOT NULL
`ordinal`	The ordinal position of the column in the foreign key.	INTEGER
`foreign_key_reference`	String representation of the foreign key.	STRING NOT NULL
`created_at`	Creation timestamp.	TIMESTAMPTZ NOT NULL
`updated_at`	Modification timestamp.	TIMESTAMPTZ NOT NULL

SCHEMA_LINEAGE table

Column Name	Description	Data Type
`source_schema_id`	The ID of the corresponding source schema.	SERIAL NOT NULL
`destination_schema_id`	The ID of the corresponding destination schema.	SERIAL NOT NULL
`created_at`	Creation timestamp.	TIMESTAMPTZ NOT NULL

TABLE_LINEAGE table

Column Name	Description	Data Type
`source_table_id`	The ID of the corresponding source table.	BIGSERIAL NOT NULL
`destination_table_id`	The ID of the corresponding destination table.	BIGSERIAL NOT NULL
`created_at`	Creation timestamp.	TIMESTAMPTZ NOT NULL

COLUMN_LINEAGE table

Column Name	Description	Data Type
`source_column_id`	The ID of the corresponding source column.	BIGSERIAL NOT NULL
`destination_column_id`	The ID of the corresponding destination column.	BIGSERIAL NOT NULL
`created_at`	Creation timestamp.	TIMESTAMPTZ NOT NULL

To learn more about which queries you can use to check metadata, see our Sample Queries page.

Error logs

These logs contain information about errors that connections encounter. You can track events, such as a failure or an authentication error from the logs.

The following table lists the errors with a SEVERE severity level:

SEVERITY	EVENT_TYPE	PROPERTY_NAME	VALUE
SEVERE	status	status	FAILURE
SEVERE	status	status	FAILURE_WITH_TASK
SEVERE	error	type	dynamically obtained from response
SEVERE	error	type	open_file
SEVERE	error	type	null_primary_key
SEVERE	error	type	authentication_error
SEVERE	error	type	permission_denied

Destination queries

You can query the data in your destination using simple SQL queries. For example, to track schema and table changes in your destination, query the LOG table for the following events:

create_schema
create_table
drop_table
alter_table

To learn more about the queries you can use, see our Sample Queries page.

Sample query format

SELECT * FROM project_id.schema_name.table_name

In this example,

project_id is your destination's project ID. To find the project ID, on your Fivetran dashboard, go to the Destination section.
schema_name is the destination schema name you chose while configuring your connection. The default is fivetran_log.
table_name is the name of the table you want to query. For example, LOG.