How to Use Kubernetes Pod Affinity to Run Hybrid Deployment Jobs on Specific Nodes?

Question

How do I use Kubernetes pod affinity to run Hybrid Deployment jobs on specific nodes?

Environment

Container platform: Kubernetes

Answer

You can use Kubernetes pod affinity to control which nodes run your Hybrid Deployment jobs, excluding the agent, based on the presence of other pods on the same node.

You can enable either node selector or pod affinity, not both. To use pod affinity, disable node selector by setting kubernetes_node_selector_enable to false.

Configure pod affinity rule associations for connections, in values.yaml file

In the config section of your Helm values.yaml file, define the affinity rules under the kubernetes_affinity branch that associates connection IDs with named scheduling rules.

Each rule maps to one or more connections.

Example:

config:
  namespace: YOUR_NAMESPACE_HERE
  data_volume_pvc: YOUR_PERSISTENT_VOLUME_CLAIM_HERE
  token: YOUR_TOKEN_HERE
  kubernetes_affinity:
    - rule: pod_affinity_rule
      connectors:
        - demo_connector5
        - demo_connector6

In this example:

demo_connector5 and demo_connector6 connections pods will be associated with the pod_affinity_rule rule.

Define pod affinity rules in config section

Define the pod affinity criteria for each rule inside the config section using the affinity_rules branch. These settings control which nodes the corresponding connections can run on, based on the presence of other pods, such as the agent pod, on the same node.

Use standard Kubernetes pod affinity syntax. In the example below, jobs are scheduled with a preference for running on the same node where the Hybrid Deployment Agent pod labeled with app.kubernetes.io/name=hd-agent is running.

Example:

config:
  affinity_rules:
    pod_affinity_rule:
      affinity:
        podAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 1
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app.kubernetes.io/name
                      operator: In
                      values:
                        - hd-agent
                topologyKey: kubernetes.io/hostname

Detailed example:

The following example brings together all the above settings:

config:
    namespace: YOUR_NAMESPACE_HERE
    data_volume_pvc: YOUR_PERSISTENT_VOLUME_CLAIM_HERE
    token: YOUR_TOKEN_HERE
    kubernetes_affinity:
    - rule: pod_affinity_rule
      connectors:
        - demo_connector5
        - demo_connector6
   
    affinity_rules:
      pod_affinity_rule:
        affinity:
          podAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 1
                podAffinityTerm:
                  labelSelector:
                    matchExpressions:
                      - key: app.kubernetes.io/name
                        operator: In
                        values:
                          - hd-agent
                  topologyKey: kubernetes.io/hostname
agent:
  image: "us-docker.pkg.dev/prod-eng-fivetran-ldp/public-docker-us/ldp-agent:production"
  image_pull_policy: "Always"