Local Data Processing for Fivetran Managed Service Private Previewlink

Updated November 6, 2023

IMPORTANT: You must have a Business Critical plan to use this feature.

With Local Data Processing for Fivetran managed service, customers with stringent data privacy and security requirements can utilize all the data integration capabilities of Fivetran within the privacy and security of their local on-premises or cloud environment.

Local Data Processing allows you to extract your data and load it into your destination, without the data ever leaving your infrastructure environment. It provides a secure solution that deploys Fivetran's services and connectors in your IT infrastructure and enables you to create a secure pipeline for your business-critical data.

Architecturelink

Local Data Processing loads your data directly into your destination, without the data going through Fivetran's cloud. To do so, Local Data Processing uses the Local Processing Agent. The Local Processing Agent is hosted on your infrastructure and it creates a network connection between your environment and Fivetran Managed SaaS. The agent keeps checking with Fivetran to determine if there are any syncs to perform. Whenever a sync needs to be performed, Fivetran provides the details of the sync to the Local Processing Agent, which then triggers the sync in your environment.

You can opt to use our Local Data Processing while setting up your destination. To use Local Data Processing, download the agent configuration files to a Linux machine with Docker installed on it. For more information about installing the Local Processing Agent, see the Install Local Processing Agent section of this guide.

Once you run the Local Processing Agent on your machine, the agent starts the services that are configured to load your data into your destination. The agent also sends the metadata of your syncs to Fivetran's cloud environment. The metadata contains the details of your syncs, such as sync metrics and logs. We use this metadata only to keep a track of your syncs, enabling you to monitor their status and details on the Fivetran dashboard.

NOTE: Each Local Processing Agent supports only one set of connector and destination.

The following diagram illustrates the Local Data Processing architecture:

Supported connectorslink

We support Local Data Processing for the following connectors:

• High-Volume Agent Oracle
• Generic PostgreSQL
• Generic SQL Server
• PostgreSQL RDS

Supported destinationslink

We support Local Data Processing for the following destinations:

• Generic PostgreSQL
• PostgreSQL RDS
• Snowflake

Install Local Processing Agentlink

Prerequisiteslink

To install the Local Processing Agent, you need a machine with the following:

• Minimum 4 vCPUs with x86-64 processors.
• Minimum 10 GB of RAM.
• Minimum 40 GB allocated disk space.
• Linux distribution that allows you to run Docker containers using Docker Engine. For example:
  • CentOS
  • Oracle Linux
  • Red Hat Enterprise Linux
  • Rocky Linux
  • SUSE Linux
  • Ubuntu
• Latest version of Docker Engine. For information about how to install Docker Engine, see Docker Engine's documentation.

  NOTE: If you want to use Docker Desktop which already includes Docker Engine, make sure your Docker subscription plan supports it.

• Docker Compose to start the Local Processing Agent after installing it. You can either use the docker compose sub command, or the latest docker-compose standalone utility.

  NOTE: For more information about installing the Compose standalone, see Docker's documentation.

Instructionslink

1. On the machine where you want to host the Local Processing Agent, run the following commands to create a user group and user for Fivetran:

   i. sudo groupadd <group_name>.

   ii. Create the user and also add it to the docker group. sudo useradd -g <group_name> -G docker -m <username>.

   NOTE: Replace <group_name> with a user group name and <username> with a username of your choice.

2. Log in using the Fivetran user you created in Step 1 and run the following commands to create folders for the agent configuration files:

   i. mkdir -p $HOME/fivetran/ldp to create the base folder.

   ii. mkdir -p $HOME/fivetran/ldp/logs to create a folder for the log files.

   iii. mkdir -p $HOME/fivetran/ldp/data to create a folder for the temporary files created during your syncs.

   NOTE: You can replace the default fivetran/ldp location with a folder name of your choice, however you must make sure to update the configuration files with the correct path values if you change it.

3. Log in to your Fivetran account.

4. Go to the Destinations page, and then click Add Destination.

5. On the Add Destination To Fivetran page, enter a Destination name of your choice.

6. Click Add.

7. Select your destination type.

8. In the destination setup form, set the Local Data Processing toggle to ON.

9. Click + Configure new Local Processing Agent.

   

10. In the Configure a new Local Processing Agent pane, download the config.json and docker-compose.yml files to the base folder you created in Step 2(i). For example, $HOME/fivetran/ldp.

    

11. Open the config.json file, and then replace <Path to storage location on host machine> with the path to the folder you created for temporary files in Step 2(iii). For example, ~/fivetran/ldp, where ~ indicates the $HOME directory.

    

12. Open the docker-compose.yml file and do the following:

    • Replace ./auth.json with $HOME/fivetran/ldp/auth.json. Note that the auth.json file will be downloaded in Step 16 below.

    • Replace ./config.json with $HOME/fivetran/ldp/config.json.

    • Replace <Path to log location on host machine, typically '$HOME/fivetran/ldp/logs'> with the path to the folder you created for log files in Step 2(ii). For example, "$HOME/fivetran/ldp/logs".

    • Verify whether the Docker socket file location specified in the file is correct for your environment. For example, "/var/run/docker.sock".

    • Verify whether the docker daemon.json file location specified in the file is correct for your environment. For example, "$HOME/.docker".

    

13. Go to the destination setup form and check the I have downloaded the configuration files checkbox.

    

14. Click Next.

15. Enter an Agent name, and then click Generate agent credentials.

16. Download the auth.json file to the base folder you created in Step 2(i). For example, $HOME/fivetran/ldp.

17. Check the I have downloaded the file checkbox, and then click Next.

18. From the Start the agent section, copy the Docker Compose command and run it on your machine to start the Local Processing Agent.

    NOTE: Docker Compose is used to start the Local Processing Agent. You can use the docker compose subcommand or the latest version of the Docker Compose standalone utility (docker-compose).

    

19. Once the agent is up and running on your machine, check the I have started the agent and it is running checkbox.

20. Click Save and continue to set up your destination.