What Data Does Fivetran Receive From My Local Environment?

Question

What data does Fivetran receive from my local environment?

Answer

Fivetran receives only metadata from your local environment.

The agent in your environment sends the following information to the Fivetran cloud environment:

• Registration information: When you start the agent on your machine, it sends the following data to the orchestration component of the Fivetran cloud:

  • Orchestration server hostname
  • Orchestration server port number
  • Client certificate and key

• Hybrid Deployment Agent logs: Fivetran logs only the internal tracing messages for your agent, and the agent sends them securely to the Fivetran cloud. These logs contain the agent registration status, sync job status, container errors, etc. A copy of these logs is available on your local machine for review. By default, these logs are available in ~/Fivetran/logs/<processing_agent_name>. However, you can change their location. You can also disable the logging in your local environment.

• Sync (data pipeline processing) logs: The sync logs contain internal tracing messages of all internal Fivetran processes and events, such as sync start and end times and sync errors. These logs contain only the metadata of synced objects to indicate when Fivetran processes the objects and their status. A copy of these logs is available on your local machine for review. By default, these logs are available in ~/Fivetran/logs/<processing_agent_name>/jobs/<process_id>. However, you can disable logging in your local environment.

• Hybrid Deployment Agent metrics: The agent sends the following metrics to indicate its performance and status:

  • Start time and run duration of the agent
  • Initiation status of the sync jobs and their failures, internal integration ID, process ID, and Docker container metadata for debugging purposes
  • Initiation status of the test jobs (for validating connection credentials for your source and destination) and their internal integration ID, process ID, and Docker container metadata
  • Initiation status of the schema config jobs (for retrieving connection schema and object details), internal integration ID, process ID, and Docker container metadata

• Sync job metrics: The agent sends the following sync job metrics:

  • Number of rows extracted or loaded per object
  • Volume of data extracted or loaded
  • Processing time for data extraction or loading

• Table and column metadata: The agent sends the following metadata of the synced tables and columns:

  • The name of the schema in the source and destination
  • The names of the tables in the source and destination
  • The names and data types of the columns in the source and destination