How to Use Kubernetes Node Selector for Hybrid Deployment Jobs?

Question

How to use Kubernetes Node Selector for Hybrid Deployment jobs?

Environment

Container platform: Kubernetes

Answer

You can use Kubernetes node selector to control which nodes run the Hybrid Deployment pipeline processing jobs. You can do this by assigning specific labels to your nodes and configuring the appropriate settings in the values.yaml file. However, we recommend using node affinity instead, as it offers greater flexibility and more advanced scheduling options.

Configuration

To enable and configure node selector, update the config section of your Helm values.yaml file with the following settings:

Enable node selector: kubernetes_node_selector_enable: true
Choose one of the following options:
- Use the default fivetran.role=job label: kubernetes_node_selector_include_role: true. Ensure that the label fivetran.role=job is applied to the target nodes.
- Use a custom selector label: kubernetes_node_selector_custom_selector: key1:value1. Replace key1:value1 with your custom key-value pair. Ensure the label exists on the target nodes.

Example configurations

Example 1: Use the default fivetran.role=job label

config:
  kubernetes_node_selector_enable: true
  kubernetes_node_selector_include_role: true

At runtime, the following node selector will be applied to the pod specification of the Hybrid Deployment pipeline processing jobs:

nodeSelector:
  fivetran.role: "job"

Example 2: Use a custom label `key1=value1`

config:
  kubernetes_node_selector_enable: true
  kubernetes_node_selector_custom_selector: key1:value1