Fivetran Platform Connector
Fivetran generates and logs several types of data related to your account and destinations:
- Log events related to connectors, dashboard user actions, Fivetran API calls, and schema change data.
- Account- and destination-related metadata that includes:
- Role/membership information
- Column lineage (available for Enterprise and Business Critical accounts)
- Granular consumption information
You can use this data for the following purposes:
- Monitoring and troubleshooting of connectors
- Tracking your usage
- Conducting audits
The Fivetran Platform Connector delivers your logs and account or destination metadata to a schema in your destination. We automatically add this connector to every new destination you create. The schema name we use for these automatically created connectors is fivetran_metadata. By default, we configure the connector to sync once a day, but you can set a different sync frequency and change other default settings on the Setup tab of the connector details page.
NOTE: We automatically add a Fivetran Platform Connector to every destination, but it doesn't start syncing logs and metadata until you add a source connector to your destination.
IMPORTANT: If you are an Account Administrator, you can manually add the Fivetran Platform Connector on an account level so that it syncs all the metadata and logs for all the destinations in your account to a single destination. If an account-level Fivetran Platform Connector is already configured in a destination in your Fivetran account, then we don't add destination-level Fivetran Platform Connectors to the new destinations you create.
IMPORTANT: The MAR that the Fivetran Platform Connector generates is free, though you may incur costs in your destination. Learn more in our pricing documentation.
Features
| Feature Name | Supported | Notes |
|---|---|---|
| Capture deletes | check | CONNECTOR, RESOURCE_MEMBERSHIP, ROLE, ROLE_PERMISSION, TEAM, TEAM_MEMBERSHIP, and USER tables |
| History mode | ||
| Custom data | ||
| Data blocking | check | |
| Column hashing | check | |
| Re-sync | check | |
| API configurable | check | API configuration |
| Priority-first sync | ||
| Fivetran data models | check | |
| Private networking | ||
| Authorization via API | check |
Setup guide
Follow our step-by-step setup guide to manually set up your Fivetran Platform Connector account-wide.
Sync overview
Most of the tables in the Fivetran Platform Connector use incremental sync so that they are updated when there is new or changed data.
For the LOG and CONNECTOR_SDK_LOG tables, new data is appended to the table. To ensure log data integrity, all connector data tables should have data present, complete, and valid following the initial sync of the relevant connector. Where the data exists, we sync the previous 7 days' worth of data prior to the date of the initial sync of the Fivetran Platform Connector. This also means that when a full re-sync is triggered for the Fivetran Platform Connector, the LOG and CONNECTOR_SDK_LOG tables will have 7 days' worth of historical data .
Schema information
This schema applies to all Fivetran Platform Connectors.
To zoom, open the ERD in a new window.NOTE: The
*_METADATAand*_LINEAGEtables are available only on the Enterprise plan and above.
Using Fivetran Platform Connector with its data models
You can use our Fivetran Platform Connector along with the Fivetran Platform Connector data model, with each of the model tables storing specific type of data.
You can also set up our free, pre-built Quickstart models. These models are autonomously created using the logic defined in our Fivetran Platform dbt Package, and prepare the data from the Fivetran Platform Connector in analysis-ready models covering spend, connector performance and status, and metadata/data lineage. If you are an advanced dbt user, you can also use the package for increased flexibility.
Important tables
You can query the data in the different destination tables to monitor different metrics such as connector performance, destination and source metadata, and MAR usage.
INCREMENTAL_MAR table
The INCREMENTAL_MAR table provides incremental daily active rows (MAR) for each destination schema, the associated tables, and the time when the MAR is calculated.
| Column Name | Description |
|---|---|
destination_id | The ID of the destination for which MAR is calculated. |
free_type | If it is free MAR, the value indicates the type of free MAR. For paid MAR,the value is PAID. |
incremental_rows | The number of new distinct primary keys on the current day synced for the connector. |
connector_id | The ID of the connector for which MAR is calculated. Deprecated and replaced by connector_name for new connectors created after July 10, 2024. |
connector_name | The name of the connector for which MAR is calculated. |
measured_date | The date in UTC format of when MAR is calculated. |
schema_name | The destination schema name for which MAR is calculated. |
sync_type | This defines whether the sync for which MAR calculated is HISTORICAL or INCREMENTAL. Currently, the available value is UNKNOWN. |
table_name | The table name associated with MAR. |
_fivetran_synced | The timestamp of when Fivetran last successfully synced the row. |
updated_at | The timestamp of when MAR is updated. |
LOG table
We write the log events to the LOG table to your destination.
| Column Name | Description | Data Type |
|---|---|---|
id | The ID of the log event (internal). | STRING |
time_stamp | Indicates the time when the log event is created. | TIMESTAMP |
_fivetran_synced | Indicates the time when Fivetran last successfully synced the row. | TIMESTAMP |
connector_id | The ID of the connector for which the event is logged. | STRING |
event | The event type. Events can be classified as a warning, an error, or just an information log. | STRING |
message_event | The routine involved in the log. | STRING |
message_data | The details of the event in JSON format. | STRING |
CONNECTOR_SDK_LOG table
We write the connector SDK log events to the CONNECTOR_SDK_LOG table in your destination
| Column Name | Description | Data Type |
|---|---|---|
id | The ID of the log event (internal). | STRING |
event_time | Indicates the time when the log event is created. | TIMESTAMP |
connector_id | The ID of the connector for which the event is logged. | STRING |
level | The event type. Events can be classified as a warning, an error, or just an information log. | STRING |
message | The details of the event in plain text. | STRING |
message_origin | The source of the log event. For example "connector_sdk". | STRING |
sync_id | The ID of the sync durinc which the event is appeared. | STRING |
SOURCE_SCHEMA_METADATA table
| Column Name | Description | Data Type |
|---|---|---|
id | Schema ID. | BIGSERIAL NOT NULL |
connector_id | The ID of the connector for which metadata is collected. | STRING |
name | The name of the table. | TEXT NOT NULL |
created_at | Creation timestamp. | TIMESTAMPTZ NOT NULL |
DESTINATION_SCHEMA_METADATA table
| Column Name | Description | Data Type |
|---|---|---|
id | Schema ID. | BIGSERIAL NOT NULL |
connector_id | The ID of the connector for which metadata is collected. | STRING |
destination_id | The ID of the destination for which metadata is collected. | STRING |
name | The name of the table. | TEXT NOT NULL |
created_at | Creation timestamp. | TIMESTAMPTZ NOT NULL |
SOURCE_TABLE_METADATA table
| Column Name | Description | Data Type |
|---|---|---|
id | Table ID. | BIGSERIAL NOT NULL |
schema_id | ID of the corresponding schema. | SERIAL NOT NULL |
name | The name of the table. | TEXT NOT NULL |
connector_id | The ID of the connector for which metadata is collected. | STRING |
created_at | Creation timestamp. | TIMESTAMPTZ NOT NULL |
DESTINATION_TABLE_METADATA table
| Column Name | Description | Data Type |
|---|---|---|
id | Table ID. | BIGSERIAL NOT NULL |
schema_id | ID of the corresponding schema. | SERIAL NOT NULL |
name | The name of the table. | TEXT NOT NULL |
connector_id | The ID of the connector for which metadata is collected. | STRING |
destination_id | The ID of the destination for which metadata is collected. | STRING |
created_at | Creation timestamp. | TIMESTAMPTZ NOT NULL |
SOURCE_COLUMN_METADATA table
| Column Name | Description | Data Type |
|---|---|---|
id | Column ID. | BIGSERIAL NOT NULL |
table_id | The ID of the corresponding table. | BIGSERIAL NOT NULL |
name | The name of the column. | TEXT NOT NULL |
type | Column type (e.g., STRING). | TEXT NOT NULL |
is_primary_key | A flag indicating if this column is part of the primary key. | BOOLEAN NOT NULL |
connector_id | The ID of the connector for which metadata is collected. | STRING |
created_at | Creation timestamp. | TIMESTAMPTZ NOT NULL |
updated_at | Modification timestamp. | TIMESTAMPTZ NOT NULL |
DESTINATION_COLUMN_METADATA
| Column Name | Description | Data Type |
|---|---|---|
id | Column ID. | BIGSERIAL NOT NULL |
table_id | The ID of the corresponding table. | BIGSERIAL NOT NULL |
name | The name of the column. | TEXT NOT NULL |
type | Column type (e.g., STRING). | TEXT NOT NULL |
connector_id | The ID of the connector for which metadata is collected. | STRING |
destination_id | The ID of the destination for which metadata is collected. | STRING |
created_at | Creation timestamp. | TIMESTAMPTZ NOT NULL |
updated_at | Modification timestamp. | TIMESTAMPTZ NOT NULL |
SOURCE_FOREIGN_KEY_METADATA table
| Column Name | Description | Data Type |
|---|---|---|
id | Foreign key ID. | BIGSERIAL NOT NULL |
column_id | The ID of the corresponding column. | BIGSERIAL NOT NULL |
ordinal | The ordinal position of the column in the foreign key. | INTEGER |
foreign_key_reference | String representation of the foreign key. | STRING NOT NULL |
created_at | Creation timestamp. | TIMESTAMPTZ NOT NULL |
updated_at | Modification timestamp. | TIMESTAMPTZ NOT NULL |
SCHEMA_LINEAGE table
| Column Name | Description | Data Type |
|---|---|---|
source_schema_id | The ID of the corresponding source schema. | SERIAL NOT NULL |
destination_schema_id | The ID of the corresponding destination schema. | SERIAL NOT NULL |
created_at | Creation timestamp. | TIMESTAMPTZ NOT NULL |
TABLE_LINEAGE table
| Column Name | Description | Data Type |
|---|---|---|
source_table_id | The ID of the corresponding source table. | BIGSERIAL NOT NULL |
destination_table_id | The ID of the corresponding destination table. | BIGSERIAL NOT NULL |
created_at | Creation timestamp. | TIMESTAMPTZ NOT NULL |
COLUMN_LINEAGE table
| Column Name | Description | Data Type |
|---|---|---|
source_column_id | The ID of the corresponding source column. | BIGSERIAL NOT NULL |
destination_column_id | The ID of the corresponding destination column. | BIGSERIAL NOT NULL |
created_at | Creation timestamp. | TIMESTAMPTZ NOT NULL |
To learn more about which queries you can use to check metadata, see our Sample Queries page.
Error logs
These logs contain information about errors that connectors encounter. You can track events, such as a failure or an authentication error from the logs.
The following table lists the errors with a SEVERE severity level:
| SEVERITY | EVENT_TYPE | PROPERTY_NAME | VALUE |
|---|---|---|---|
| SEVERE | status | status | FAILURE |
| SEVERE | status | status | FAILURE_WITH_TASK |
| SEVERE | error | type | dynamically obtained from response |
| SEVERE | error | type | open_file |
| SEVERE | error | type | null_primary_key |
| SEVERE | error | type | authentication_error |
| SEVERE | error | type | permission_denied |
Destination queries
You can query the data in your destination using simple SQL queries. For example, to track schema and table changes in your destination, query the LOG table for the following events:
create_schemacreate_tabledrop_tablealter_table
To learn more about the queries you can use, see our Sample Queries page.
Sample query format
SELECT * FROM project_id.schema_name.table_name
In this example,
project_idis your destination's project ID. To find the project ID, on your Fivetran dashboard, go to the Destination section.schema_nameis the destination schema name you chose while configuring your connector. The default isfivetran_log.table_nameis the name of the table you want to query. For example,LOG.