How to Use Kubernetes Node Affinity to Run Hybrid Deployment Jobs on Specific Nodes?

Question

How to use Kubernetes node affinity to run Hybrid Deployment jobs on specific nodes?

Environment

Container platform: Kubernetes

Answer

You can use Kubernetes node affinity to control which nodes run your Hybrid Deployment jobs (excluding the agent). This method provides more flexible scheduling than Node Selector and allows you to define complex rules, such as running most jobs on a small node group while directing specific connectors to larger nodes.

You can enable either node selector or node affinity, not both. To use node affinity, disable node selector by setting kubernetes_node_selector_enable to false.

Configure node affinity rule associations for connections in values.yaml file

In the config section of your Helm values.yaml file, define the named affinity rules associated with connection IDs.

Each rule maps to one or more connections. You can also set one rule as the default, which applies to all connections not specifically listed.

Example:

config:
  namespace: YOUR_NAMESPACE_HERE
  data_volume_pvc: YOUR_PERSISTENT_VOLUME_CLAIM_HERE
  token: YOUR_TOKEN_HERE
  kubernetes_affinity:
    - rule: small
      connectors:
        - demo_connection1
        - demo_connection2
      default: true
    - rule: large
      connectors:
        - demo_connection3
        - demo_connection4

In this example:

demo_connection1 and demo_connection2 run on nodes labeled as SMALL.
demo_connection3 and demo_connection4 run on nodes labeled as LARGE.
All other connections use the small rule by default.

Define node affinity rules in config section

Define the node affinity criteria for each rule in the affinity_rules section under the config section of your values.yaml file. These settings control which nodes the corresponding connections can run on.

Use standard Kubernetes node affinity syntax. In the following example, nodes are labeled with HD_SIZE=SMALL or HD_SIZE=LARGE.

Example:

config:
    affinity_rules:
      small:
        affinity:
          nodeAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 1
                preference:
                  matchExpressions:
                    - key: HD_SIZE
                      operator: In
                      values:
                        - "SMALL"
    
      large:
        affinity:
          nodeAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 1
                preference:
                  matchExpressions:
                    - key: HD_SIZE
                      operator: In
                      values:
                        - "LARGE"

(Optional) Set node selector for Hybrid Deployment Agent

You can continue using node selector for the agent while using node affinity for connector jobs. For example, run the agent on the same nodes used for small jobs:

agent:
  node_selector:
    HD_SIZE: "small"

Detailed example:

The following example brings together all the above settings:

config:
    namespace: YOUR_NAMESPACE_HERE
    data_volume_pvc: YOUR_PERSISTENT_VOLUME_CLAIM_HERE
    token: YOUR_TOKEN_HERE
    kubernetes_affinity:
    - rule: small
      connectors:
        - demo_connector1
        - demo_connector2
      default: true
    - rule: large
      connectors:
        - demo_connector3
        - demo_connector4
   
    affinity_rules:
      small:
        affinity:
          nodeAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 1
                preference:
                  matchExpressions:
                    - key: HD_SIZE
                      operator: In
                      values:
                        - "SMALL"
    
      large:
        affinity:
          nodeAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 1
                preference:
                  matchExpressions:
                    - key: HD_SIZE
                      operator: In
                      values:
                        - "LARGE"
                      
agent:
  image: "us-docker.pkg.dev/prod-eng-fivetran-ldp/public-docker-us/ldp-agent:production"
  image_pull_policy: "Always"
  node_selector:
    HD_SIZE: "small"
  resources:
    requests:
      cpu: "2"
      memory: "1500Mi"
    limits:
      cpu: "2"
      memory: "2Gi"