Hybrid Deployment Setup Guide Private Previewlink
Follow our setup guide to set up the Hybrid Deployment architecture for your data pipeline.
Supported container management toolslink
Hybrid Deployment supports the following container management tools for Hybrid Deployment:
The setup instructions for Hybrid Deployment architecture vary based on the container management tool you choose to manage Fivetran's containers in your local environment.
Hybrid Deployment with Docker - Setup instructionslink
Learn how to set up the Hybrid Deployment architecture with Docker containers.
Expand for instructions
Prerequisiteslink
To use Hybrid Deployment with Docker containers, you need a machine in your local environment with the following:
- CPU: Minimum 4 vCPUs with x86-64 processors.
- Memory: Minimum 10 GB of RAM.
- Storage:
- Minimum 50 GB allocated disk space for the Docker storage location (usually
/var/lib/docker
). You need the Docker images for your local processing agent, sync processing runs, etc.- PostgreSQL destinations: We recommend that the size of the Docker storage location must be equal to the volume of data per connector.
- Snowflake destinations: We recommend that the size of the Docker storage location must be at least 30% of the volume of data per connector.
- Minimum 50 GB allocated disk space for the persistent local storage (usually
$HOME/fivetran/data
). We use the persistent local storage during the pipeline processing and to allow the data to be persisted across sync processing runs.- PostgreSQL destinations: We recommend that the size of the persistent local storage must be equal to the volume of data per connector.
- Snowflake destinations: We recommend that the size of the persistent local storage must be at least 30% of the volume of data per connector.
- Minimum 50 GB allocated disk space for the Docker storage location (usually
- Linux distribution that allows you to run Docker containers using Docker Engine. For example:
- Ubuntu
- CentOS
- Rocky Linux
- SUSE Linux
- Oracle Linux
- Red Hat Enterprise Linux
- Latest version of Docker Engine. For information about how to install Docker Engine, see Docker Engine's documentation.
NOTE: If you want to use Docker Desktop which already includes Docker Engine, make sure your Docker subscription plan supports it.
- Docker Compose (version 2.x or above) to start the local processing agent after installing it. You can either use the
docker compose
sub command, or the latestdocker-compose
standalone utility.NOTE: For more information about installing the Compose standalone, see Docker's documentation.
- A storage container or bucket to stage your data before loading it into your destination. The staging location is necessary only for the following destinations:
- Databricks
- Redshift
- Snowflake
NOTE: You can find more information about configuring the staging location on the individual destination setup guides.
Setup instructionslink
Configure local environmentlink
Log in to the machine where you want to host the local processing agent.
Run the following command to create a user group for Fivetran:
sudo groupadd <group_name>
NOTE: Replace
<group_name>
in the command with a user group name of your choice.Run the following command to create a user for Fivetran:
sudo useradd -g <group_name> -G docker -m <username>
NOTE: Replace
<group_name>
in the command with the name of the user group you created and<username>
with a username of your choice.Log out from the machine, and then log back in using the Fivetran user you created.
Create the following folders:
i. Base folder: Run
mkdir -p $HOME/<base_folder_name>
to create the base folder for the local processing agent. For example,mkdir -p $HOME/fivetran
.ii. Logs folder: Run
mkdir -p $HOME/<base_folder_name>/logs
to create a folder for the logs. For example,mkdir -p $HOME/fivetran/logs
.iii. Temporary files folder: Run
mkdir -p $HOME/<base_folder_name>/data
to create a folder for the temporary files created during your syncs. For example,mkdir -p $HOME/fivetran/data
.NOTE: Replace
<base_folder_name>
in the commands with a folder name of your choice.Make a note of the paths to all the new folders you created. You will need them to configure your local processing agent.
Install local processing agentlink
Log in to your Fivetran account.
Go to the Destinations page and click Add destination.
Enter a Destination name of your choice.
Click Add.
Select your destination type.
In the destination setup form, set the Enable local data processing toggle to ON.
Click + Configure new local processing agent.
In the Configure a new local processing agent pane, read the On-Prem Software License Addendum, and select the I have read and agree to the terms of the License Addendum and the Software Specific Requirements checkbox.
Click Next.
Download the
config.json
anddocker-compose.yml
files to the base folder you created in Step 1.Select the I have downloaded the configuration files checkbox and click Next.
Enter an Agent name and click Generate agent credentials.
Download the
auth.json
file to the base folder you created in Step 1.Select the I have downloaded the file checkbox and then click Next.
From the Start the agent section, make a note the Docker Compose command. You will need it to start the local processing agent.
Configure local processing agentlink
Log in to your local machine using the Fivetran user you created in Step 1.
Go to the base folder you created in Step 1.
Open the
config.json
file you downloaded in Step 2.Set the value of
host_persistent_storage_mount_path
to the path to the temporary files folder you created in Step 1. For example,"host_persistent_storage_mount_path": "~/fivetran/data"
.Save and close the
config.json
file.
Start local processing agentlink
On your local machine, open the
docker-compose.yml
file.NOTE: The local processing agent container uses the Docker volume mounts to read the configuration files and persist the metadata on your local host.
(Optional) If you want the controller to restart automatically after every machine reboot, set the
restart
option toalways
. The default value isno
because Docker's default restart policy does not restart a container under any circumstance.Verify whether the file and folder paths specified in the
volumes
section are correct for your environment. If not, replace the default values with the correct paths. Thevolumes
section contains the paths to the following:auth.json
file you downloaded in Step 2 (default path:$HOME/fivetran/auth.json
)config.json
file you downloaded in Step 2 (default path:$HOME/fivetran/config.json
)- Logs folder you created in Step 1 (default path:
$HOME/fivetran/logs
) - Docker configuration file (default path:
$HOME/.docker
) - Docker socket file (default path:
/var/run/docker.sock
)NOTE: If you want to specify a different Docker socket file path, make sure you edit or replace
/var/run/docker.sock
specified before the colon (:). You must not change the value after the colon (:).
Save and close the
docker-compose.yml
file.Run the Docker Compose command you found in Step 2 to start the local processing agent.
NOTE: Docker Compose is used to start the local processing agent. You can use the
docker compose
subcommand or the latest version of the Docker Compose standalone utility (docker-compose).
Confirm local processing agent installationlink
Verify whether the local processing agent is up and running.
On the Fivetran dashboard, go to the destination setup form.
Select the I have started the agent and it is running checkbox.
Click Save and continue to set up your destination.
Hybrid Deployment with Podman - Setup instructionslink
Learn how to set up the Hybrid Deployment architecture with Podman containers.
Expand for instructions
Prerequisiteslink
To use Hybrid Deployment with Podman containers, you need a machine in your local environment with the following:
- CPU: Minimum 4 vCPUs with x86-64 processors.
- Memory: Minimum 10 GB of RAM.
- Storage:
- Minimum 50 GB allocated disk space for the Podman storage location (usually
/.local/share/containers/storage/
). You need the Podman images for your local processing agent, sync processing runs, etc.- PostgreSQL destinations: We recommend that the size of the Podman storage location must be equal to the volume of data per connector.
- Snowflake destinations: We recommend that the size of the Podman storage location must be at least 30% of the volume of data per connector.
- Minimum 50 GB allocated disk space for the persistent local storage (usually
$HOME/fivetran/data
). We use the persistent local storage during the pipeline processing and to allow the data to be persisted across sync processing runs.- PostgreSQL destinations: We recommend that the size of the persistent local storage must be equal to the volume of data per connector.
- Snowflake destinations: We recommend that the size of the persistent local storage must be at least 30% of the volume of data per connector.
- Minimum 50 GB allocated disk space for the Podman storage location (usually
- Linux distribution that allows you to run Podman containers. For example:
- Ubuntu
- CentOS
- Rocky Linux
- SUSE Linux
- Oracle Linux
- Red Hat Enterprise Linux
- Podman (v4.6.1 or above).
NOTE: For more information about installing Podman on Linux distributions, see Podman's documentation.
- A storage container or bucket to stage your data before loading it into your destination. The staging location is necessary only for the following destinations:
- Databricks
- Redshift
- Snowflake
NOTE: You can find more information about configuring the staging location on the individual destination setup guides.
Setup instructionslink
Configure local environmentlink
Log in to the machine where you want to host the local processing agent.
Create the following folders:
i. Base folder: Run
mkdir -p $HOME/<base_folder_name>
to create the base folder for the local processing agent. For example,mkdir -p $HOME/fivetran
.ii. Logs folder: Run
mkdir -p $HOME/<base_folder_name>/logs
to create a folder for the logs. For example,mkdir -p $HOME/fivetran/logs
.iii. Temporary files folder: Run
mkdir -p $HOME/<base_folder_name>/data
to create a folder for the temporary files created during your syncs. For example,mkdir -p $HOME/fivetran/data
.NOTE: Replace
<base_folder_name>
in the commands with a folder name of your choice.Make a note of the paths to all the new folders you created. You will need them to configure your local processing agent.
Run the following commands to start the Podman socket in rootless mode:
i.
systemctl --user enable --now podman.socket
ii.
systemctl --user start --now podman.socket
iii.
sudo loginctl enable-linger <user_name>
NOTE:
- In the
sudo loginctl enable-linger <user_name>
command, replace<user_name>
with your username. - For better security, we recommend that you always run Podman in rootless mode.
- In the
Install local processing agentlink
Log in to your Fivetran account.
Go to the Destinations page and click Add destination.
Enter a Destination name of your choice.
Click Add.
Select your destination type.
In the destination setup form, set the Enable local data processing toggle to ON.
Click + Configure new local processing agent.
In the Configure a new local processing agent pane, read the Fivetran On-Prem Software License Addendum, and select the I accept the terms of the License Addendum checkbox.
Click Next.
Download the
config.json
file to the base folder you created in Step 1.Select the I have downloaded the configuration files checkbox and click Next.
Enter an Agent name and click Generate agent credentials.
Select the I have downloaded the file checkbox and then click Next.
Download the
auth.json
file to the base folder you created in Step 1.Select the I have downloaded the file checkbox and then click Next.
Configure local processing agentlink
Log in to your local machine.
Go to the base folder you created in Step 1.
Open the
config.json
file you downloaded in Step 2.Add the following text to the file:
"container_env_type": 3,
.Replace the default value of
host_docker_sock_file_path
withunix:///run/user/<user_id>/podman/podman.sock
.NOTE: In the socket file path, replace
<user_id>
with your system user ID. For example,unix:///run/user/1000/podman/podman.sock
.Set the value of
host_persistent_storage_mount_path
to the path to the temporary files folder you created in Step 1. For example,"host_persistent_storage_mount_path": "$HOME/fivetran/data"
.Save and close the
config.json
file.
Start local processing agentlink
On your local machine, run the following command to create a Podman network:
podman network create -d bridge fivetran_ldp > /dev/null 2>&1
Run the following command to start the local processing agent:
podman run \ -d \ --rm \ --security-opt label=disable \ --name controller \ --network fivetran_ldp \ --env HOST_USER_HOME_DIR=$HOME \ -v <auth-file-path>:/tmpauth.json:ro \ -v <config-file-path>:/tmpconfig.json:ro \ -v <logs-folder-path>:/logs \ -v $XDG_RUNTIME_DIR/podman/podman.sock:/run/user/<user_id>/podman/podman.sock \ us-docker.pkg.dev/prod-eng-fivetran-ldp/public-docker-us/ldp-agent:production \ /tmpauth.json /tmpconfig.json
NOTE: Replace the following values in the command:
<auth-file-path>
with the absolute path to the folder where you stored theauth.json
file in Step 2. For example,$HOME/fivetran/auth.json
.<config-file-path>
with the absolute path to the folder where you stored theconfig.json
file in Step 2. For example,$HOME/fivetran/config.json
.<logs-folder-path>
with the absolute path to the folder you created for logs in Step 1. For example,$HOME/fivetran/logs
.<user_id>
with your system user ID. For example,/run/user/1000/podman/podman.sock
.
Expand for sample scripts configured to start and stop the local processing agent
start-agent.shlink
#!/bin/bash
podman network create -d bridge fivetran_ldp > /dev/null 2>&1
podman run \
-d \
--rm \
--security-opt label=disable \
--name controller \
--network fivetran_ldp \
--env HOST_USER_HOME_DIR=$HOME \
-v $HOME/fivetran/auth.json:/tmpauth.json:ro \
-v $HOME/fivetran/config.json:/tmpconfig.json:ro \
-v $HOME/fivetran/logs:/logs \
-v $XDG_RUNTIME_DIR/podman/podman.sock:/run/user/1000/podman/podman.sock \
us-docker.pkg.dev/prod-eng-fivetran-ldp/public-docker-us/ldp-agent:production \
/tmpauth.json /tmpconfig.json
stop-agent.shlink
#!/bin/bash
podman stop controller
podman rm controller
Confirm local processing agent installationlink
Verify whether the local processing agent is up and running.
On the Fivetran dashboard, go to the destination setup form.
Select the I have started the agent and it is running checkbox.
Click Save and continue to set up your destination.
Related articleslink
description Hybrid Deployment Overview