LAB04a: Taint & Toleration

Understanding in One Line

• Taint = node says “don’t place pods here unless they tolerate this.”

• Toleration = pod says “I accept your taint, I can be scheduled.”

Taints & Tolerations

Make control-plane schedulable

Apply taints on nodes

Test pod scheduling behavior

Verify when pods are blocked or allowed

This is exactly how senior DevOps/SREs demonstrate the concept.

LAB OVERVIEW

1. Check cluster nodes

2. Remove control-plane taint → make CP schedulable

3. Create a taint on a worker node

4. Deploy a pod WITHOUT toleration → should fail to schedule

5. Deploy a pod WITH toleration → should successfully schedule

6. Understand taint effects (NoSchedule, PreferNoSchedule, NoExecute)

7. Optional: Add taints BACK to control-plane node

PRE-REQUISITE

A kubeadm or k3s or minikube cluster.

STEP 1: Check Nodes

kubectl get nodes -o wide

Typical output:

NAME            ROLES           AGE   STATUS
cp-node         control-plane   1d    Ready
worker1         <none>          1d    Ready
worker2         <none>          1d    Ready

STEP 2: Make Control Plane Schedulable

Kubeadm clusters have this taint:

node-role.kubernetes.io/control-plane:NoSchedule

Remove it:

kubectl taint nodes cp-node node-role.kubernetes.io/control-plane:NoSchedule-

Verify:

kubectl describe node cp-node | grep Taint

If nothing shows → control plane is now schedulable.

STEP 3: Add a Taint to a Node

Let's taint worker1:

kubectl taint nodes worker1 environment=dev:NoSchedule

Verify:

kubectl describe node worker1 | grep Taint

You should see:

Taints: environment=dev:NoSchedule

Meaning:

• Pods without toleration → ❌ cannot be scheduled on worker1

• Pods with matching toleration → ✔ allowed

---

STEP 4: Deploy a Pod WITHOUT Toleration

Create file:

pod-no-toleration.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx-no-toleration
spec:
  containers:
  - name: nginx
    image: nginx

Apply:

kubectl apply -f pod-no-toleration.yaml

Check pod:

kubectl get pod -o wide

Expected:

• Pod will NOT schedule onto worker1 (tainted)

• It will only schedule onto cp-node or worker2

Check events:

kubectl describe pod nginx-no-toleration

You should see message like:

node(s) had taints that the pod didn't tolerate

STEP 5: Deploy a Pod WITH Toleration

Create file:

pod-with-toleration.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx-tolerate
spec:
  tolerations:
  - key: "environment"
    operator: "Equal"
    value: "dev"
    effect: "NoSchedule"
  containers:
  - name: nginx
    image: nginx

Apply:

kubectl apply -f pod-with-toleration.yaml

Check:

kubectl get pod -o wide

Expected:

• Pod will schedule on worker1

• Because it has matching toleration.

Check:

kubectl describe pod nginx-tolerate | grep Node

LAB UNDERSTANDING: WHY THIS WORKS

Taint applied:

environment=dev:NoSchedule

Meaning:

• Kubernetes will not schedule pods on worker1 unless they tolerate the taint.

Toleration used:

tolerations:
- key: "environment"
  operator: "Equal"
  value: "dev"
  effect: "NoSchedule"

This tells scheduler:

• “I can live on tainted nodes”

STEP 6: Add Taints Back to Control Plane (Optional)

Re-taint CP:

kubectl taint nodes cp-node node-role.kubernetes.io/control-plane=:NoSchedule

Or for older labels:

kubectl taint nodes cp-node node-role.kubernetes.io/master=:NoSchedule

Check:

kubectl describe node cp-node | grep Taint

SUMMARY OF LAB RESULTS

Node	Taint	Pod Behavior
cp-node	No taint	Any pod can run
worker1	environment=dev:NoSchedule	Only pods WITH toleration can run
worker2	No taint	Any pod can run

Yes you can do the entire taint & toleration lab using ONLY kubectl commands (no YAML). This is how SREs quickly test taints in real clusters.

Below is the command-only lab.

LAB: Taints & Tolerations Using Only Commands

1. Make control-plane schedulable

Remove default taint:

kubectl taint nodes <control-plane-node> node-role.kubernetes.io/control-plane:NoSchedule-

Or older clusters:

kubectl taint nodes <control-plane-node> node-role.kubernetes.io/master:NoSchedule-

2. Add taint to a worker node

Example: taint worker1

kubectl taint nodes worker1 environment=dev:NoSchedule

3. Create a pod WITHOUT toleration (command-only)

kubectl run no-toleration --image=nginx

Check where it got scheduled:

kubectl get pod no-toleration -o wide

Describe event:

kubectl describe pod no-toleration | grep -i taint -A3

It should show:

node(s) had taints that the pod didn't tolerate

4. Create a pod WITH toleration (command-only)

Using kubectl run --overrides

kubectl run tolerate-dev \
  --image=nginx \
  --overrides='
{
  "apiVersion": "v1",
  "spec": {
    "tolerations": [
      {
        "key": "environment",
        "operator": "Equal",
        "value": "dev",
        "effect": "NoSchedule"
      }
    ]
  }
}'

Check where scheduled:

kubectl get pod tolerate-dev -o wide

It should schedule on worker1.

Shortcut: Create pod with toleration using oneliners

If you want fast testing, this is easier:

Option A: kubectl run with --dry-run pipe to apply

kubectl run tolerate --image=nginx \
  --dry-run=client -o yaml \
  | kubectl set toleration -f - environment=dev:NoSchedule --local -o yaml \
  | kubectl apply -f -

Option B: use kubectl-krew plugin (if installed)

kubectl run tolerate --image=nginx
kubectl taint-set tolerate environment=dev:NoSchedule

5. Remove a taint from worker

kubectl taint nodes worker1 environment=dev:NoSchedule-

6. Add taint back to control-plane (optional)

kubectl taint nodes <control-plane-node> node-role.kubernetes.io/control-plane=:NoSchedule