Fivetran Platform Connector link
Updated November 16, 2023
Fivetran generates structured log events from connectors, dashboard user actions, and Fivetran API calls. The Fivetran Platform Connector delivers your logs and account metadata to a schema in your destination. This metadata includes granular Fivetran consumption information.
IMPORTANT: The MAR that the Fivetran Platform Connector generates is free, though you may incur costs in your destination. Learn more in our pricing documentation.
NOTE: We have deprecated the Metadata API and replaced it with the Fivetran Platform Connector's capabilities to query source and destination metadata.
Featureslink
Feature Name | Supported | Notes |
---|---|---|
Capture deletes | check | USER and CONNECTOR tables |
Custom data | ||
Data blocking | check | Column level |
Column hashing | check | |
Re-sync | check | Connector level |
History | ||
API configurable | check | API configuration |
Priority-first sync | ||
Fivetran data models | check | Get the model; Supports Quickstart data models |
Private networking |
Setup guidelink
Follow our step-by-step setup guide to set up your Fivetran Platform Connector.
Sync overviewlink
The Fivetran Platform Connector syncs the latest logs generated from your connectors to your destination. It syncs additional information about the account, destination, data flow, connectors, users, and membership present in your Fivetran account. We sync all these details at a destination level. If you are an Account Administrator, you can also choose to sync all the connector logs for all the destinations in your account.
Log eventslink
Fivetran generates multiple log events depending on your use case enabling you to monitor your connectors and track your usage. For more information about the different log events that Fivetran generates, see our Events documentation.
Formatlink
We write the log events in a standardized JSON format to your destination:
{
"event": <Event name>,
"created": <Event creation timestamp in UTC>,
"connector_type": <Connector type>,
"connector_id": <Connector ID>,
"connector_name": <Connector name>,
"data": {
// Event-specific data. This section varies for each event
},
"exception_id": <Fivetran error identifier>
}
content_copy
We write the logs to the LOG
table in the destination. See our LOG
table documentation for more information.
Schema informationlink
This schema applies to all Fivetran Platform Connectors.
To zoom, open the ERD in a new window.Important tableslink
You can query the data in the different destination tables to monitor different metrics such as connector performance, destination and source metadata, and MAR usage:
INCREMENTAL_MARlink
The INCREMENTAL_MAR
table provides incremental daily active rows (MAR) for each destination schema, the associated tables, and the time when the MAR is calculated.
Column Name | Description |
---|---|
destination_id | The ID of the group for which MAR is calculated. |
free_type | If it is free MAR, the value indicates the type of free MAR. For paid MAR, the value is PAID . |
incremental_rows | The number of new distinct primary keys on the current day synced for the connector. |
connector_id | The ID of the connector for which MAR is calculated. |
measured_date | The date in UTC format of when MAR is calculated. |
schema_name | The destination schema name for which MAR is calculated. |
sync_type | This defines whether the sync for which MAR calculated is HISTORICAL or INCREMENTAL. Currently, the available value is UNKNOWN . |
table_name | The table name associated with MAR. |
_fivetran_synced | The timestamp of when Fivetran last successfully synced the row. |
updated_at | The timestamp of when MAR is updated. |
LOGlink
The LOG
table provides a record of events that have occurred for a particular connector along with the event details.
Column Name | Description | Data Type |
---|---|---|
id | The ID of the log event (internal). | STRING |
time_stamp | Indicates the time when the log event is created. | TIMESTAMP |
_fivetran_synced | Indicates the time when Fivetran last successfully synced the row. | TIMESTAMP |
connector_id | The ID of the connector for which the event is logged. | STRING |
event | The event type. Events can be classified as a warning, an error, or just an information log. | STRING |
message_event | The routine involved in the log. | STRING |
message_data | The details of the event in JSON format. | STRING |
transformation_id | The ID of the transformation (internal). | STRING |
USAGE_COSTlink
The USAGE_COST
table provides information about the monthly usage summary.
Column Name | Description |
---|---|
destination_id | The ID of the destination for which the usage occurred. |
measured_month | The month in which the usage is measured. |
amount | The total dollar amount spent during the month. |
NOTE: If you are using a credits-based plan, see our
CREDITS_USED
table documentation.
SOURCE_SCHEMA_METADATAlink
Column Name | Description | Data Type |
---|---|---|
id | Schema ID. | BIGSERIAL NOT NULL |
connector_id | The ID of the connector for which metadata is collected. | STRING |
name | The name of the table. | TEXT NOT NULL |
created_at | Creation timestamp. | TIMESTAMPTZ NOT NULL |
DESTINATION_SCHEMA_METADATAlink
Column Name | Description | Data Type |
---|---|---|
id | Schema ID. | BIGSERIAL NOT NULL |
destination_id | The ID of the destination for which metadata is collected. | STRING |
name | The name of the table. | TEXT NOT NULL |
created_at | Creation timestamp. | TIMESTAMPTZ NOT NULL |
SOURCE_TABLE_METADATAlink
Column Name | Description | Data Type |
---|---|---|
id | Table ID. | BIGSERIAL NOT NULL |
schema_id | ID of the corresponding schema. | SERIAL NOT NULL |
name | The name of the table. | TEXT NOT NULL |
connector_id | The ID of the connector for which metadata is collected. | STRING |
created_at | Creation timestamp. | TIMESTAMPTZ NOT NULL |
DESTINATION_TABLE_METADATAlink
Column Name | Description | Data Type |
---|---|---|
id | Table ID. | BIGSERIAL NOT NULL |
schema_id | ID of the corresponding schema. | SERIAL NOT NULL |
name | The name of the table. | TEXT NOT NULL |
destination_id | The ID of the destination for which metadata is collected. | STRING |
created_at | Creation timestamp. | TIMESTAMPTZ NOT NULL |
SOURCE_COLUMN_METADATAlink
Column Name | Description | Data Type |
---|---|---|
id | Column ID. | BIGSERIAL NOT NULL |
table_id | The ID of the corresponding table. | BIGSERIAL NOT NULL |
name | The name of the column. | TEXT NOT NULL |
type | Column type (e.g., STRING ). | TEXT NOT NULL |
is_primary_key | A flag indicating if this column is part of the primary key. | BOOLEAN NOT NULL |
connector_id | The ID of the connector for which metadata is collected. | STRING |
created_at | Creation timestamp. | TIMESTAMPTZ NOT NULL |
updated_at | Modification timestamp. | TIMESTAMPTZ NOT NULL |
DESTINATION_COLUMN_METADATAlink
Column Name | Description | Data Type |
---|---|---|
id | Column ID. | BIGSERIAL NOT NULL |
table_id | The ID of the corresponding table. | BIGSERIAL NOT NULL |
name | The name of the column. | TEXT NOT NULL |
type | Column type (e.g., STRING ). | TEXT NOT NULL |
destination_id | The ID of the destination for which metadata is collected. | STRING |
created_at | Creation timestamp. | TIMESTAMPTZ NOT NULL |
updated_at | Modification timestamp. | TIMESTAMPTZ NOT NULL |
SOURCE_FOREIGN_KEY_METADATAlink
Column Name | Description | Data Type |
---|---|---|
id | Foreign key ID. | BIGSERIAL NOT NULL |
column_id | The ID of the corresponding column. | BIGSERIAL NOT NULL |
ordinal | The ordinal position of the column in the foreign key. | INTEGER |
foreign_key_reference | String representation of the foreign key. | STRING NOT NULL |
created_at | Creation timestamp. | TIMESTAMPTZ NOT NULL |
updated_at | Modification timestamp. | TIMESTAMPTZ NOT NULL |
SCHEMA_LINEAGElink
Column Name | Description | Data Type |
---|---|---|
source_schema_id | The ID of the corresponding source schema. | SERIAL NOT NULL |
destination_schema_id | The ID of the corresponding destination schema. | SERIAL NOT NULL |
created_at | Creation timestamp. | TIMESTAMPTZ NOT NULL |
TABLE_LINEAGElink
Column Name | Description | Data Type |
---|---|---|
source_table_id | The ID of the corresponding source table. | BIGSERIAL NOT NULL |
destination_table_id | The ID of the corresponding destination table. | BIGSERIAL NOT NULL |
created_at | Creation timestamp. | TIMESTAMPTZ NOT NULL |
COLUMN_LINEAGElink
Column Name | Description | Data Type |
---|---|---|
source_column_id | The ID of the corresponding source column. | BIGSERIAL NOT NULL |
destination_column_id | The ID of the corresponding destination column. | BIGSERIAL NOT NULL |
created_at | Creation timestamp. | TIMESTAMPTZ NOT NULL |
To learn more about which queries you can use to check metadata, see our Sample Queries page.
Error logslink
These logs contain information about errors that connectors encounter. You can track events, such as a failure or an authentication error from the logs.
The following table lists the errors with a SEVERE
severity level:
SEVERITY | EVENT_TYPE | PROPERTY_NAME | VALUE |
---|---|---|---|
SEVERE | status | status | FAILURE |
SEVERE | status | status | FAILURE_WITH_TASK |
SEVERE | error | type | dynamically obtained from response |
SEVERE | error | type | open_file |
SEVERE | error | type | null_primary_key |
SEVERE | error | type | authentication_error |
SEVERE | error | type | permission_denied |
Destination querieslink
You can query the data in your destination using simple SQL queries. For example, to track schema and table changes in your destination, query the LOG
table for the following events:
create_schema
create_table
drop_table
alter_table
To learn more about the queries you can use, see our Sample Queries page.
Sample query formatlink
SELECT * FROM project_id.schema_name.table_name
content_copy
In this example,
project_id
is your destination's project ID. To find the project ID, on your Fivetran dashboard, go to the Destination section.schema_name
is the destination schema name you chose while configuring your connector. The default isfivetran_log
.table_name
is the name of the table you want to query. For example,LOG
.
CREDITS_USED tablelink
You can query the data in the CREDITS_USED
table for information about the monthly credit usage summary.
Column Name | Description |
---|---|
destination_id | The ID of the destination for which the credits are used. |
measured_month | The month in which the credits are used. |
credits_consumed | The number of credits used during the month. |
Using Fivetran Platform Connector with its data modellink
You can use our Fivetran Platform Connector along with the Fivetran Platform Connector data model to recreate the deprecated _fivetran_audit
table in your destination. The pre-built data model uses the data stored in the log tables to create fivetran_platform__audit_table
matching the _fivetran_audit
table.