How Can I Avoid Docker or Podman Network Limits per Host When Running Many Concurrent Hybrid Deployment Jobs?

Question

How can I avoid Docker or Podman network limits per host when running many concurrent Hybrid Deployment jobs?

Environment

Container platform: Docker and Podman

Answer

Each Hybrid Deployment Agent creates its own Docker or Podman bridge network called fivetran_ldp. By default, every pipeline processing job runs in a separate, on-demand network (for example, ldp_net_<process_id>), which is removed when the job completes. If you run a large number of jobs concurrently on a single host, you may exceed the Docker or Podman network limit.

Each Hybrid Deployment Agent runs its own Docker or Podman bridge network, called fivetran_ldp. By default, every pipeline processing job managed by an agent runs in a separate network. For example, a job may run in a network named ldp_net_<process_id>, where <process_id> is the job's unique ID. These networks are created on demand and removed when jobs complete. If you run a large number of jobs concurrently on a single host, you may exceed the Docker or Podman network limit.

To avoid hitting this limit, configure the Hybrid Deployment Agent to run all jobs in the shared fivetran_ldp agent network instead of creating a separate network for each job.

Update config.json

Set the following parameter in your agent’s config.json file:

{
  "use_docker_networks_for_jobs": false
}

Configure Helm chart

If you are deploying the Hybrid Deployment Agent with Helm, include the necessary configuration in your values.yaml file.

Example:

config:
  namespace: YOUR_NAMESPACE
  data_volume_pvc: YOUR_PERSISTENT_VOLUME_CLAIM_NAME
  token: YOUR_TOKEN_HERE
  annotations:
    gke-gcsfuse/volumes: true
  service_account: YOUR_SERVICE_ACCOUNT_NAME
agent:
    image: "us-docker.pkg.dev/prod-eng-fivetran-ldp/public-docker-us/ldp-agent:production"
    image_pull_policy: "Always"