Fivetran Platform Connector link

Updated November 16, 2023

Fivetran generates structured log events from connectors, dashboard user actions, and Fivetran API calls. The Fivetran Platform Connector delivers your logs and account metadata to a schema in your destination. This metadata includes granular Fivetran consumption information.

IMPORTANT: The MAR that the Fivetran Platform Connector generates is free, though you may incur costs in your destination. Learn more in our pricing documentation.

NOTE: We have deprecated the Metadata API and replaced it with the Fivetran Platform Connector's capabilities to query source and destination metadata.

Featureslink

Feature Name	Supported	Notes
Capture deletes		`USER` and `CONNECTOR` tables
Custom data
Data blocking		Column level
Column hashing
Re-sync		Connector level
History
API configurable		API configuration
Priority-first sync
Fivetran data models		Get the model; Supports Quickstart data models
Private networking

Setup guidelink

Follow our step-by-step setup guide to set up your Fivetran Platform Connector.

Sync overviewlink

The Fivetran Platform Connector syncs the latest logs generated from your connectors to your destination. It syncs additional information about the account, destination, data flow, connectors, users, and membership present in your Fivetran account. We sync all these details at a destination level. If you are an Account Administrator, you can also choose to sync all the connector logs for all the destinations in your account.

Log eventslink

Fivetran generates multiple log events depending on your use case enabling you to monitor your connectors and track your usage. For more information about the different log events that Fivetran generates, see our Events documentation.

Formatlink

We write the log events in a standardized JSON format to your destination:

{
    "event": <Event name>,
    "created": <Event creation timestamp in UTC>,
    "connector_type": <Connector type>,
    "connector_id": <Connector ID>,
    "connector_name": <Connector name>,
    "data": {
        // Event-specific data. This section varies for each event
    },
    "exception_id": <Fivetran error identifier>
}
content_copy

We write the logs to the LOG table in the destination. See our LOG table documentation for more information.

Schema informationlink

This schema applies to all Fivetran Platform Connectors.

To zoom, open the ERD in a new window.

Important tableslink

You can query the data in the different destination tables to monitor different metrics such as connector performance, destination and source metadata, and MAR usage:

INCREMENTAL_MARlink

The INCREMENTAL_MAR table provides incremental daily active rows (MAR) for each destination schema, the associated tables, and the time when the MAR is calculated.

Column Name	Description
`destination_id`	The ID of the group for which MAR is calculated.
`free_type`	If it is free MAR, the value indicates the type of free MAR. For paid MAR, the value is `PAID`.
`incremental_rows`	The number of new distinct primary keys on the current day synced for the connector.
`connector_id`	The ID of the connector for which MAR is calculated.
`measured_date`	The date in UTC format of when MAR is calculated.
`schema_name`	The destination schema name for which MAR is calculated.
`sync_type`	This defines whether the sync for which MAR calculated is HISTORICAL or INCREMENTAL. Currently, the available value is `UNKNOWN`.
`table_name`	The table name associated with MAR.
`_fivetran_synced`	The timestamp of when Fivetran last successfully synced the row.
`updated_at`	The timestamp of when MAR is updated.

LOGlink

The LOG table provides a record of events that have occurred for a particular connector along with the event details.

Column Name	Description	Data Type
`id`	The ID of the log event (internal).	STRING
`time_stamp`	Indicates the time when the log event is created.	TIMESTAMP
`_fivetran_synced`	Indicates the time when Fivetran last successfully synced the row.	TIMESTAMP
`connector_id`	The ID of the connector for which the event is logged.	STRING
`event`	The event type. Events can be classified as a warning, an error, or just an information log.	STRING
`message_event`	The routine involved in the log.	STRING
`message_data`	The details of the event in JSON format.	STRING
`transformation_id`	The ID of the transformation (internal).	STRING

USAGE_COSTlink

The USAGE_COST table provides information about the monthly usage summary.

Column Name	Description
`destination_id`	The ID of the destination for which the usage occurred.
`measured_month`	The month in which the usage is measured.
`amount`	The total dollar amount spent during the month.

NOTE: If you are using a credits-based plan, see our CREDITS_USED table documentation.

SOURCE_SCHEMA_METADATAlink

Column Name	Description	Data Type
`id`	Schema ID.	BIGSERIAL NOT NULL
`connector_id`	The ID of the connector for which metadata is collected.	STRING
`name`	The name of the table.	TEXT NOT NULL
`created_at`	Creation timestamp.	TIMESTAMPTZ NOT NULL

DESTINATION_SCHEMA_METADATAlink

Column Name	Description	Data Type
`id`	Schema ID.	BIGSERIAL NOT NULL
`destination_id`	The ID of the destination for which metadata is collected.	STRING
`name`	The name of the table.	TEXT NOT NULL
`created_at`	Creation timestamp.	TIMESTAMPTZ NOT NULL

SOURCE_TABLE_METADATAlink

Column Name	Description	Data Type
`id`	Table ID.	BIGSERIAL NOT NULL
`schema_id`	ID of the corresponding schema.	SERIAL NOT NULL
`name`	The name of the table.	TEXT NOT NULL
`connector_id`	The ID of the connector for which metadata is collected.	STRING
`created_at`	Creation timestamp.	TIMESTAMPTZ NOT NULL

DESTINATION_TABLE_METADATAlink

Column Name	Description	Data Type
`id`	Table ID.	BIGSERIAL NOT NULL
`schema_id`	ID of the corresponding schema.	SERIAL NOT NULL
`name`	The name of the table.	TEXT NOT NULL
`destination_id`	The ID of the destination for which metadata is collected.	STRING
`created_at`	Creation timestamp.	TIMESTAMPTZ NOT NULL

SOURCE_COLUMN_METADATAlink

Column Name	Description	Data Type
`id`	Column ID.	BIGSERIAL NOT NULL
`table_id`	The ID of the corresponding table.	BIGSERIAL NOT NULL
`name`	The name of the column.	TEXT NOT NULL
`type`	Column type (e.g., `STRING`).	TEXT NOT NULL
`is_primary_key`	A flag indicating if this column is part of the primary key.	BOOLEAN NOT NULL
`connector_id`	The ID of the connector for which metadata is collected.	STRING
`created_at`	Creation timestamp.	TIMESTAMPTZ NOT NULL
`updated_at`	Modification timestamp.	TIMESTAMPTZ NOT NULL

DESTINATION_COLUMN_METADATAlink

Column Name	Description	Data Type
`id`	Column ID.	BIGSERIAL NOT NULL
`table_id`	The ID of the corresponding table.	BIGSERIAL NOT NULL
`name`	The name of the column.	TEXT NOT NULL
`type`	Column type (e.g., `STRING`).	TEXT NOT NULL
`destination_id`	The ID of the destination for which metadata is collected.	STRING
`created_at`	Creation timestamp.	TIMESTAMPTZ NOT NULL
`updated_at`	Modification timestamp.	TIMESTAMPTZ NOT NULL

SOURCE_FOREIGN_KEY_METADATAlink

Column Name	Description	Data Type
`id`	Foreign key ID.	BIGSERIAL NOT NULL
`column_id`	The ID of the corresponding column.	BIGSERIAL NOT NULL
`ordinal`	The ordinal position of the column in the foreign key.	INTEGER
`foreign_key_reference`	String representation of the foreign key.	STRING NOT NULL
`created_at`	Creation timestamp.	TIMESTAMPTZ NOT NULL
`updated_at`	Modification timestamp.	TIMESTAMPTZ NOT NULL

SCHEMA_LINEAGElink

Column Name	Description	Data Type
`source_schema_id`	The ID of the corresponding source schema.	SERIAL NOT NULL
`destination_schema_id`	The ID of the corresponding destination schema.	SERIAL NOT NULL
`created_at`	Creation timestamp.	TIMESTAMPTZ NOT NULL

TABLE_LINEAGElink

Column Name	Description	Data Type
`source_table_id`	The ID of the corresponding source table.	BIGSERIAL NOT NULL
`destination_table_id`	The ID of the corresponding destination table.	BIGSERIAL NOT NULL
`created_at`	Creation timestamp.	TIMESTAMPTZ NOT NULL

COLUMN_LINEAGElink

Column Name	Description	Data Type
`source_column_id`	The ID of the corresponding source column.	BIGSERIAL NOT NULL
`destination_column_id`	The ID of the corresponding destination column.	BIGSERIAL NOT NULL
`created_at`	Creation timestamp.	TIMESTAMPTZ NOT NULL

To learn more about which queries you can use to check metadata, see our Sample Queries page.

Error logslink

These logs contain information about errors that connectors encounter. You can track events, such as a failure or an authentication error from the logs.

The following table lists the errors with a SEVERE severity level:

SEVERITY	EVENT_TYPE	PROPERTY_NAME	VALUE
SEVERE	status	status	FAILURE
SEVERE	status	status	FAILURE_WITH_TASK
SEVERE	error	type	dynamically obtained from response
SEVERE	error	type	open_file
SEVERE	error	type	null_primary_key
SEVERE	error	type	authentication_error
SEVERE	error	type	permission_denied

Destination querieslink

You can query the data in your destination using simple SQL queries. For example, to track schema and table changes in your destination, query the LOG table for the following events:

create_schema
create_table
drop_table
alter_table

To learn more about the queries you can use, see our Sample Queries page.

Sample query formatlink

SELECT * FROM project_id.schema_name.table_name 
content_copy

In this example,

project_id is your destination's project ID. To find the project ID, on your Fivetran dashboard, go to the Destination section.
schema_name is the destination schema name you chose while configuring your connector. The default is fivetran_log.
table_name is the name of the table you want to query. For example, LOG.

CREDITS_USED tablelink

You can query the data in the CREDITS_USED table for information about the monthly credit usage summary.

Column Name	Description
`destination_id`	The ID of the destination for which the credits are used.
`measured_month`	The month in which the credits are used.
`credits_consumed`	The number of credits used during the month.

Using Fivetran Platform Connector with its data modellink

You can use our Fivetran Platform Connector along with the Fivetran Platform Connector data model to recreate the deprecated _fivetran_audit table in your destination. The pre-built data model uses the data stored in the log tables to create fivetran_platform__audit_table matching the _fivetran_audit table.

Documentation