How to Use Kubernetes Node Tolerations for Hybrid Deployment?

Question

How to use Kubernetes node tolerations for Hybrid Deployment?

Environment

Container platform: Kubernetes

Answer

You can use Kubernetes taints and tolerations to control where the Hybrid Deployment Agent and pipeline processing jobs run. When you apply taints to specific nodes, Kubernetes prevents pods from running on those nodes unless the pods explicitly include matching tolerations.

You must use Helm v0.14.0 or above to configure taints and tolerations.

Configuration

To configure tolerations for the agent, define the tolerations under the agent section in your values.yaml file.
To configure tolerations for the pipeline processing jobs, define the tolerations under the config section.

Example

Taint nodes using kubectl

#example commands to create the node taints
kubectl taint nodes <node_name> key1=value1:NoSchedule
kubectl taint nodes <node_name> key2:NoExecute

Add tolerations to values.yaml file

# Sets the tolerations for the Hybrid Deployment pipeline processing jobs.
config: 
  tolerations: [
    {
      key: key1,
      operator: Equal,
      value: value1,
      effect: NoSchedule
    }
  ]

# Sets the tolerations for the Hybrid Deployment agent.
agent:
    tolerations: [
      {
      key: key1,
      operator: Equal,
      value: value1,
      effect: NoSchedule
      },
      {
        key: key2, 
        operator: Exists,
        effect: NoExecute
      }
    ]

Runtime behavior

Kubernetes applies the following tolerations at runtime:

Hybrid Deployment Agent pods:

tolerations:
      - effect: NoSchedule
        key: key1
        operator: Equal
        value: value1
      - effect: NoExecute
        key: key2
        operator: Exists

Pipeline processing job pods:

tolerations:
      - effect: NoSchedule
        key: key1
        operator: Equal
        value: value1