Monitor File Sync Activity

Question

How can I monitor which files are being synced, track file processing patterns, and audit my file sync operations?

Environment

File connectors

Answer

File connectors record read_start and read_end events as user-facing logs (UFLs) to provide visibility into your file sync operations. This enables you to monitor file processing patterns, track sync performance, troubleshoot issues, and audit data ingestion workflows. You must have a Fivetran Platform connector and complete the initial sync. Once you have configured you Fivetran Platform connection, we create the LOG table in your destination schema. This table captures the file metadata through two events.

• read_start events - Extracts the following values :

  • Source - the file name
  • Table name - the destination table name
  • File size - the file size in bytes
  • File modified time - the last modified timestamp of the file

• read_end events - Extracts the following values:

  • Extracted rows count - the number of rows extracted from the file
  • Skipped rows count - the number of rows skipped during processing

How to access file sync metadata

To access the metadata Fivetran captures in the LOG table, perform this step.

1. Query the LOG table using the following SQL pattern:

SELECT
    message_event,
    message_data,
FROM `<Destination-Schema-Name>.log`
WHERE message_event IN ('read_start', 'read_end');