Hybrid Deployment Setup Guide Beta
Follow our setup guide to set up the Hybrid Deployment model for your data pipeline.
Supported container management tools
Hybrid Deployment supports the following container management tools for Hybrid Deployment:
The setup instructions for the Hybrid Deployment model vary based on the container management tool you choose to manage Fivetran's containers in your local environment.
Hybrid Deployment with Docker - Setup instructions
Learn how to set up the Hybrid Deployment model with Docker containers.
Expand for instructions
Prerequisites
To use Hybrid Deployment with Docker containers, you need a machine in your local environment with the following:
- CPU: Minimum 4 vCPUs with x86-64 processors.
- Memory: Minimum 10 GB of RAM.
- Storage:
- Minimum 50 GB allocated disk space for the Docker storage location (usually
/var/lib/docker
). You need the Docker images for your local processing agent, sync processing runs, etc.- PostgreSQL destinations: We recommend that the size of the Docker storage location must be equal to the volume of data per connector.
- Snowflake destinations: We recommend that the size of the Docker storage location must be at least 30% of the volume of data per connector.
- Minimum 50 GB allocated disk space for the persistent local storage (usually
$HOME/fivetran/data
). We use the persistent local storage during the pipeline processing and to allow the data to be persisted across sync processing runs.- PostgreSQL destinations: We recommend that the size of the persistent local storage must be equal to the volume of data per connector.
- Snowflake destinations: We recommend that the size of the persistent local storage must be at least 30% of the volume of data per connector.
- Minimum 50 GB allocated disk space for the Docker storage location (usually
- Linux distribution that allows you to run Docker containers using Docker Engine. For example:
- Ubuntu
- CentOS
- Rocky Linux
- SUSE Linux
- Oracle Linux
- Red Hat Enterprise Linux
- Docker Engine(v20.10.17 or above). We recommend that you use the latest version of Docker Engine.
NOTE:
- For information about how to install Docker Engine, see Docker Engine's documentation.
- If you want to use Docker Desktop which already includes Docker Engine, make sure your Docker subscription plan supports it.
- Docker Compose (v2.17.x or above) to start the local processing agent after installing it. You can either use the
docker compose
sub command, or the latestdocker-compose
standalone utility.NOTE: For more information about installing the Compose standalone, see Docker's documentation.
- A storage container or bucket to stage your data before loading it into your destination. The staging location is necessary only for the following destinations:
- BigQuery
- Databricks
- Redshift
- Snowflake
NOTE: You can find more information about configuring the staging location on the individual destination setup guides.
Setup instructions
Configure local environment
Log in to the machine where you want to host the local processing agent.
Run the following command to create a Fivetran user group:
sudo groupadd <group_name>
NOTE: Replace
<group_name>
in the command with a user group name of your choice.Add a user to the Fivetran user group.
If you want Fivetran to use an existing user, run the following command:
sudo usermod -aG docker <username>
If you want to create a new user for Fivetran, run the following command:
sudo useradd -g <group_name> -G docker -m <username>
NOTE: In the above commands, you must replace
<group_name>
with the name of the Fivetran user group you created and<username>
with a username of your choice.Log back in as the Fivetran user.
Create the following folders:
i. Base folder: Run
mkdir -p $HOME/<base_folder_name>
to create the base folder for the local processing agent. For example,mkdir -p $HOME/fivetran
.ii. Logs folder: Run
mkdir -p $HOME/<base_folder_name>/logs
to create a folder for the logs. For example,mkdir -p $HOME/fivetran/logs
.iii. Temporary files folder: Run
mkdir -p $HOME/<base_folder_name>/data
to create a folder for the temporary files created during your syncs. For example,mkdir -p $HOME/fivetran/data
.NOTE: Replace
<base_folder_name>
in the commands with a folder name of your choice.Make a note of the paths to all the new folders you created. You will need them to configure your local processing agent.
Install local processing agent
Log in to your Fivetran account.
Go to the Destinations page and click Add destination.
Enter a Destination name of your choice.
Click Add.
Select your destination type.
In the destination setup form, set the Enable hybrid deployment toggle to ON.
Click + Configure new local processing agent.
In the Configure a new local processing agent pane, read the On-Prem Software License Addendum, and select the I have read and agree to the terms of the License Addendum and the Software Specific Requirements checkbox.
Click Next.
Download the
config.json
anddocker-compose.yml
files to the base folder you created in Step 1.Select the I have downloaded the configuration files checkbox and click Next.
Enter an Agent name and click Generate agent credentials.
Download the
auth.json
file to the base folder you created in Step 1.Select the I have downloaded the file checkbox and then click Next.
From the Start the agent section, make a note the Docker Compose command. You will need it to start the local processing agent.
Configure local processing agent
Log in to your local machine using the Fivetran user.
Go to the base folder you created in Step 1.
Open the
config.json
file you downloaded in Step 2.Set the value of
host_persistent_storage_mount_path
to the path to the temporary files folder you created in Step 1. For example,"host_persistent_storage_mount_path": "~/fivetran/data"
.If SELinux is enabled in your local environment, set the value of
host_selinux_enabled
totrue
. The default value isfalse
.Save and close the
config.json
file.
Find controller ID
Open the
auth.json
file you downloaded in Step 2.Make a note of the
controller_id
value. You will need it to start your local processing agent.Close the
auth.json
file.
Start local processing agent
On your local machine, open the
docker-compose.yml
file.NOTE: The local processing agent container uses the Docker volume mounts to read the configuration files and persist the metadata on your local host.
(Optional) If you want the controller to restart automatically after every machine reboot, set the
restart
option toalways
. The default value isno
because Docker's default restart policy does not restart a container under any circumstance.Replace the default placeholder value of
ldp_controller_id
with thecontroller_id
value you found in Step 4. The default placeholder value is:<controller-id>
.Verify whether the file and folder paths specified in the
volumes
section are correct for your environment. If not, replace the default values with the correct paths. Thevolumes
section contains the paths to the following:auth.json
file you downloaded in Step 2 (default path:$HOME/fivetran/auth.json
)config.json
file you downloaded in Step 2 (default path:$HOME/fivetran/config.json
)- Logs folder you created in Step 1 (default path:
$HOME/fivetran/logs
) - Docker socket file (default path:
/var/run/docker.sock
)NOTE: If you want to specify a different Docker socket file path, make sure you edit or replace
/var/run/docker.sock
specified before the colon (:). You must not change the value after the colon (:).
Save and close the
docker-compose.yml
file.Run the Docker Compose command you found in Step 2 to start the local processing agent.
NOTE: Docker Compose is used to start the local processing agent. You can use the
docker compose
subcommand or the latest version of the Docker Compose standalone utility (docker-compose).
Confirm local processing agent installation
Verify whether the local processing agent is up and running.
On the Fivetran dashboard, go to the destination setup form.
Select the I have started the agent and it is running checkbox.
Click Save and continue to set up your destination.
Hybrid Deployment with Podman - Setup instructions
Learn how to set up the Hybrid Deployment model with Podman containers.
Expand for instructions
Prerequisites
To use Hybrid Deployment with Podman containers, you need a machine in your local environment with the following:
- CPU: Minimum 4 vCPUs with x86-64 processors.
- Memory: Minimum 10 GB of RAM.
- Storage:
- Minimum 50 GB allocated disk space for the Podman storage location (usually
/.local/share/containers/storage/
). You need the Podman images for your local processing agent, sync processing runs, etc.- PostgreSQL destinations: We recommend that the size of the Podman storage location must be equal to the volume of data per connector.
- Snowflake destinations: We recommend that the size of the Podman storage location must be at least 30% of the volume of data per connector.
- Minimum 50 GB allocated disk space for the persistent local storage (usually
$HOME/fivetran/data
). We use the persistent local storage during the pipeline processing and to allow the data to be persisted across sync processing runs.- PostgreSQL destinations: We recommend that the size of the persistent local storage must be equal to the volume of data per connector.
- Snowflake destinations: We recommend that the size of the persistent local storage must be at least 30% of the volume of data per connector.
- Minimum 50 GB allocated disk space for the Podman storage location (usually
- Linux distribution that allows you to run Podman containers. For example:
- Ubuntu
- CentOS
- Rocky Linux
- SUSE Linux
- Oracle Linux
- Red Hat Enterprise Linux
- Podman (v4.6.1 or above).
NOTE: For more information about installing Podman on Linux distributions, see Podman's documentation.
- A storage container or bucket to stage your data before loading it into your destination. The staging location is necessary only for the following destinations:
- BigQuery
- Databricks
- Redshift
- Snowflake
NOTE: You can find more information about configuring the staging location on the individual destination setup guides.
Setup instructions
Configure local environment
Log in to the machine where you want to host the local processing agent.
Run the following command to create a user group for Fivetran:
sudo groupadd <group_name>
NOTE: Replace
<group_name>
in the command with a user group name of your choice.Run the following command to create a user for Fivetran:
sudo useradd -g <group_name> -m <username>
NOTE: Replace
<group_name>
in the command with the name of the user group you created and<username>
with a username of your choice.Run the following command to switch to the user you created for Fivetran:
sudo su - <username>
NOTE: Replace
<username>
in the command with the name of the user you created for Fivetran.Create the following folders:
i. Base folder: Run
mkdir -p $HOME/<base_folder_name>
to create the base folder for the local processing agent. For example,mkdir -p $HOME/fivetran
.ii. Logs folder: Run
mkdir -p $HOME/<base_folder_name>/logs
to create a folder for the logs. For example,mkdir -p $HOME/fivetran/logs
.iii. Temporary files folder: Run
mkdir -p $HOME/<base_folder_name>/data
to create a folder for the temporary files created during your syncs. For example,mkdir -p $HOME/fivetran/data
.NOTE: Replace
<base_folder_name>
in the commands with a folder name of your choice.Make a note of the paths to all the new folders you created. You will need them to configure your local processing agent.
Run the following command:
export XDG_RUNTIME_DIR=/run/user/$(id -u)
Open the
.bashrc
file associated with the user you created for Fivetran and add the following command to the file:export XDG_RUNTIME_DIR=/run/user/$(id -u)
Save and close the
.bashrc
file.Run the following commands to start the Podman socket in rootless mode:
i. systemctl --user enable --now podman.socket
ii. systemctl --user start --now podman.socket
NOTE: For better security, we recommend that you always run Podman in rootless mode.
Switch to a root user and run the following command to allow services to run even after the user logs out:
sudo loginctl enable-linger <username>
NOTE: Replace
<username>
in the command with the username of the user you created for Fivetran.
Install local processing agent
Log in to your Fivetran account.
Go to the Destinations page and click Add destination.
Enter a Destination name of your choice.
Click Add.
Select your destination type.
In the destination setup form, set the Enable hybrid deployment toggle to ON.
Click + Configure new local processing agent.
In the Configure a new local processing agent pane, read the Fivetran On-Prem Software License Addendum, and select the I accept the terms of the License Addendum checkbox.
Click Next.
Download the
config.json
file to the base folder you created in Step 1.Select the I have downloaded the configuration files checkbox and click Next.
Enter an Agent name and click Generate agent credentials.
Select the I have downloaded the file checkbox and then click Next.
Download the
auth.json
file to the base folder you created in Step 1.Select the I have downloaded the file checkbox and then click Next.
Configure local processing agent
Log in to your local machine.
Go to the base folder you created in Step 1.
Open the
config.json
file you downloaded in Step 2.Add the following text to the file:
"container_env_type": 3,
.Replace the default value of
host_docker_sock_file_path
withunix:///run/user/<user_id>/podman/podman.sock
.NOTE: In the socket file path, replace
<user_id>
with your system user ID. For example,unix:///run/user/1000/podman/podman.sock
. You can run theid -u
command to find your user ID.Set the value of
host_persistent_storage_mount_path
to the path to the temporary files folder you created in Step 1. For example,"host_persistent_storage_mount_path": "~/fivetran/data"
.If SELinux is enabled in your local environment, set the value of
host_selinux_enabled
totrue
. The default value isfalse
.Save and close the
config.json
file.
Find controller ID
Open the
auth.json
file you downloaded in Step 2.Make a note of the
controller_id
value. You will need it to start your local processing agent.Close the
auth.json
file.
Start local processing agent
On your local machine, run the following command as a non-root user to create a Podman network:
podman network create -d bridge fivetran_ldp > /dev/null 2>&1
Run the following command as the same non-root user to start the local processing agent:
podman run \ -d \ --rm \ --security-opt label=disable \ --label fivetran=ldp \ --label ldp_process_id=default-controller-process-id \ --label ldp_controller_id=<controller-id> \ --name controller \ --network fivetran_ldp \ --env FIVETRAN_ENV=prod \ --env HOST_USER_HOME_DIR=$HOME \ -v <auth-file-path>:/tmpauth.json:ro \ -v <config-file-path>:/tmpconfig.json:ro \ -v <logs-folder-path>:/logs \ -v $XDG_RUNTIME_DIR/podman/podman.sock:/run/user/1000/podman/podman.sock \ us-docker.pkg.dev/prod-eng-fivetran-ldp/public-docker-us/ldp-agent:production /tmpauth.json /tmpconfig.json
NOTE: Replace the following values in the command:
<controller-id>
with thecontroller_id
value you found in Step 4.<auth-file-path>
with the absolute path to the folder where you stored theauth.json
file in Step 2. For example,$HOME/fivetran/auth.json
.<config-file-path>
with the absolute path to the folder where you stored theconfig.json
file in Step 2. For example,$HOME/fivetran/config.json
.<logs-folder-path>
with the absolute path to the folder you created for logs in Step 1. For example,$HOME/fivetran/logs
.
Expand for sample scripts configured to start and stop the local processing agent
start-agent.sh
#!/bin/bash
podman network create -d bridge fivetran_ldp > /dev/null 2>&1
podman run \
-d \
--rm \
--security-opt label=disable \
--label fivetran=ldp \
--label ldp_process_id=default-controller-process-id \
--label ldp_controller_id=controller123 \
--name controller \
--network fivetran_ldp \
--env FIVETRAN_ENV=prod \
--env HOST_USER_HOME_DIR=$HOME \
-v $HOME/fivetran/auth.json:/tmpauth.json:ro \
-v $HOME/fivetran/config.json:/tmpconfig.json:ro \
-v $HOME/fivetran/logs:/logs \
-v $XDG_RUNTIME_DIR/podman/podman.sock:/run/user/1000/podman/podman.sock \
us-docker.pkg.dev/prod-eng-fivetran-ldp/public-docker-us/ldp-agent:production /tmpauth.json /tmpconfig.json
stop-agent.sh
#!/bin/bash
podman stop controller
podman rm controller
Confirm local processing agent installation
Verify whether the local processing agent is up and running.
On the Fivetran dashboard, go to the destination setup form.
Select the I have started the agent and it is running checkbox.
Click Save and continue to set up your destination.
Related articles
description Hybrid Deployment Overview