LAB03a: Kubernetes Lifecycle Management Lab (v1.33 → v1.34)

Cluster Topology

Role	Hostname	kube version	CNI	OS
Control plane	master	1.33.x	Calico	Ubuntu 24.04
Worker node	node1	1.33.x	Calico	Ubuntu 24.04

PART 1 — Install containerd

Installation guide

PART 2 — Create a Test Deployment

We use nginx:

kubectl create deployment web --image=nginx --replicas=3
kubectl get pods -o wide

PART 3 — Node Lifecycle Management

1️ Cordon the node

kubectl cordon node1

Expected behavior

• No new pods will be scheduled to node1.

• Existing pods continue running, not disturbed.

Verify:

kubectl describe node node1 | grep -i unsched

2️ Drain the node

kubectl drain node1 --ignore-daemonsets --delete-emptydir-data

What happens now?

✔ Kubernetes evicts pods on node1 ✔ Scheduler moves pods to master (if resources available) ✔ If cluster cannot schedule → pods stay in Pending

Watch:

kubectl get pods -w -o wide

PART 4 — Node Crash Simulation

Simulate crash:

sudo systemctl stop kubelet
sudo systemctl stop containerd

Observe:

kubectl get nodes

Expected behavior:

⏱ After ~40 seconds: NotReady ⏱ After grace period: Pods on that node are terminated and rescheduled ❗ Except DaemonSets — they remain because they’re node-local ✔ Calico will mark node as "down" in BGP

Check pod movement:

kubectl get pods -o wide -w

PART 5 — Upgrade Cluster to v1.34

1️ Upgrade Master Node

Unhold and install new kubeadm:

sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm=1.34.0-1.1

Check plan:

sudo kubeadm upgrade plan

Apply:

sudo kubeadm upgrade apply v1.34.0

Upgrade kubelet and kubectl:

sudo apt-get install -y kubelet=1.34.0-1.1 kubectl=1.34.0-1.1
sudo systemctl restart kubelet

Check:

kubectl get nodes

2️ Upgrade Worker Node

Drain:

kubectl drain node1 --ignore-daemonsets

Upgrade kubeadm:

sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm=1.34.0-1.1
sudo kubeadm upgrade node

Upgrade kubelet:

sudo apt-get install -y kubelet=1.34.0-1.1 kubectl=1.34.0-1.1
sudo systemctl restart kubelet

Uncordon:

kubectl uncordon node1

---

PART 6 — Add a NEW Node (v1.34)

From master:

kubeadm token create --print-join-command

Run the join command on new node (node2), which already has v1.34 installed.

---

PART 7 — Test Scheduling After Upgrade

kubectl scale deployment web --replicas=6
kubectl get pods -o wide

✔ Pods should now be well distributed across upgraded nodes.

---

What Students Learn

✔ Kubernetes lifecycle fundamentals ✔ How cordon/drain affects scheduling ✔ Pod evacuation + rescheduling ✔ BGP/Calico behavior when node goes down ✔ How Kubernetes handles node crash ✔ Clean cluster upgrade from v1.33 → v1.34 ✔ Node join workflow in real production

References:

1. BLOG07: Node maintenance - cordon/drain