Monitor File Sync Activity

Question

How can I monitor which files are being synced, track file processing patterns, and audit my file sync operations?

Answer

File connectors generate two log events, read_start and read_end, that record and provide visibility to your file sync activities. Together, these events provide file-specific metadata that you can use to:

• Identify the files we process and the time of processing
• Monitor sync performances
• Troubleshoot skipped rows and unexpected row counts
• Maintain an audit trail of ingestion activities

You must have a Fivetran Platform connector with a completed initial sync. Once your connection is configured, Fivetran creates a LOG table in your destination schema and captures file metadata through the following two events:

• read_start - Captures the following values:

  • Source - file name
  • Table name - destination table name
  • File size - file size in bytes
  • File modified time - last modified timestamp of the file

• read_end - Captures the following values:

  • Extracted row count - number of rows extracted from the file
  • Skipped row count - number of rows skipped while processing the files

How to access file sync metadata

Use the following SQL to query the LOG table:

SELECT
    message_event,
    message_data,
FROM `<Destination-Schema-Name>.log`
WHERE message_event IN ('read_start', 'read_end');