Hybrid Deployment with Podman Setup Guide

Configure local Linux environment

1. Log in to your local machine using the Fivetran user.

2. Go to the /etc/profile.d/ directory and add a new file named proxy.sh.

3. In the proxy.sh file, set appropriate values for the following system-wide environment variables:

   • http_proxy: Specifies the proxy server to use for HTTP requests.
   • https_proxy: Specifies the proxy server to use for HTTPS requests.
   • no_proxy: Excludes specific domains or IPs from using the proxy.

   For example:

   export http_proxy="http://my-squid-proxy.example.com:3128"
   export https_proxy="http://my-squid-proxy.example.com:3128"
   export no_proxy="localhost,127.0.0.1"
   

   • We recommend using the Fully Qualified Domain Name (FQDN) or IP address of the proxy host.
   • The no_proxy value can include a custom list of exclusions. In most environments, this includes localhost,127.0.0.1.
   • If the host is in an AWS environment, make sure the no_proxy variable includes the instance metadata address 169.254.169.254.

Configure container runtime

Podman primarily operates in rootless mode, meaning it runs under a regular user account without needing root privileges. To ensure Podman uses the appropriate proxy settings when pulling images or running containers, you must configure both the podman.service and podman.socket user-level systemd units.

Update Podman service configuration

1. Open the override file (~/.config/systemd/user/podman.service.d/override.con) for the Podman service.

   systemctl --user edit podman.service
   

2. Add the environment variables to the [Service] section. For example:

   [Service]
   Environment="HTTP_PROXY=http://my-squid-proxy.example.com:3128"
   Environment="HTTPS_PROXY=http://my-squid-proxy.example.com:3128"
   Environment="NO_PROXY=localhost,127.0.0.1"
   

Update Podman socket configuration

1. Open the override file for the Podman socket.

   systemctl --user edit podman.socket
   

2. Add the environment variables to the [Service] section. For example:

   [Service]
   Environment="HTTP_PROXY=http://my-squid-proxy.example.com:3128"
   Environment="HTTPS_PROXY=http://my-squid-proxy.example.com:3128"
   Environment="NO_PROXY=localhost,127.0.0.1"
   

Reload and restart Podman services

After updating the configuration, reload the systemd manager and restart the Podman services to apply the changes.

systemctl --user daemon-reexec
systemctl --user daemon-reload
systemctl --user restart podman.service
systemctl --user restart podman.socket

Configure local environment for agent

1. Log in to your local machine using the Fivetran user.

2. Run the following commands to create the agent directories:

   mkdir -p $HOME/fivetran
   cd $HOME/fivetran
   mkdir -p data conf logs tmp
   

   These commands create the agent directories in the following structure:

   $HOME/fivetran        --> Agent home directory              
         ├── conf        --> Configuration file location               
         ├── data        --> Persistent storage used during data pipeline processing           
         ├── logs        --> Log file location       
         └── tmp         --> Local temporary storage used during data pipeline processing        
   

3. Create a configuration file, config.json, in $HOME/fivetran/conf directory.

4. In the config.json file, add the agent token Fivetran generated for your agent.

   {
     "token": "YOUR_AGENT_TOKEN"
   }
   

   • We recommend that you add the agent token to the config.json file. However, you can skip this step and use the token as an environment variable when starting the agent container.
   • By default, you do not have to add any additional configuration values to the config.json file. However, you can add additional values to the config.json file based on your requirements. For more information about the configuration parameters, see our Agent Configuration Parameters documentation.

Start agent

1. Log in to your local machine using the Fivetran user.

2. Go to the base folder you created.

3. Create a Podman network and start the container.

   #!/bin/bash
   
   # Config file is expected in the conf/ sub folder
   CONFIG_FILE=conf/config.json
   
   # Token will be extracted from config file
   TOKEN=$(grep -o '"token": *"[^"]*"' "$CONFIG_FILE" | sed 's/.*"token": *"\([^"]*\)".*/\1/')
   
   # Extract controller id from token
   CONTROLLER_ID=$(echo $TOKEN | base64 -d | cut -f1 -d":")
   
   # Podman socket 
   XDG_RUNTIME_DIR=/run/user/$(id -u)
   SOCKET=$XDG_RUNTIME_DIR/podman/podman.sock
   
   # Create podman network for agent container
   podman network create -d bridge fivetran_ldp > /dev/null 2>&1
   
   # Start agent container
   podman run \
    -d \
    --restart "on-failure:3" \
    --pull "always" \
    --security-opt label=disable \
    --label fivetran=ldp \
    --label ldp_process_id=default-controller-process-id \
    --label ldp_controller_id=$CONTROLLER_ID \
    --name controller \
    --network fivetran_ldp \
    --env HOST_USER_HOME_DIR=$HOME \
    --env CONTAINER_ENV_TYPE="podman" \
    -v $HOME/fivetran/conf:/conf \
    -v $HOME/fivetran/logs:/logs \
    -v $SOCKET:/run/user/1000/podman/podman.sock \
    us-docker.pkg.dev/prod-eng-fivetran-ldp/public-docker-us/ldp-agent:production -f /conf/config.json
   

Stop agent

The following script identifies the agent container, and then stops and removes it:

#!/bin/bash
CONTAINER_ID=$(podman ps -a -q -f name="^/controller" -f label=fivetran=ldp)
podman stop $CONTAINER_ID
podman rm $CONTAINER_ID
podman network rm fivetran_ldp

1. Open the config.json file for the Hybrid Deployment Agent.

2. Add the necessary proxy environment variables you specified for your local environment. For example:

   ...
   "http_proxy": "http://my-squid-proxy.example.com:3128",
   "https_proxy": "http://my-squid-proxy.example.com:3128",
   "no_proxy": "localhost,127.0.0.1",
   ...
   

3. Restart the agent to apply the new settings.

   ./hdagent.sh stop
   ./hdagent.sh start