LAB03: Joining New Node

Objective:

✔ joining new worker nodes ✔ performing maintenance ✔ draining & cordon/uncordon ✔ permanently removing nodes ✔ understanding DaemonSets and eviction behavior

---

Kubernetes Cluster Management

SCENARIO

You have:

• A Kubernetes cluster with 1 control-plane + 1 worker node (kubeadm-based, Kubernetes v1.33).

• You want to:

  1. Prepare a new worker node and join it to the cluster.

  2. Simulate maintenance on an existing worker (cordon, drain, uncordon).

  3. Permanently remove a node from the cluster.

  4. Understand DaemonSet behavior during drain.

PART 1 — Prepare a NEW Node and Join to Cluster

1. Prepare the new worker node

On your new Linux server (Ubuntu):

A. Install required packages

sudo apt update
sudo apt install -y apt-transport-https curl containerd

B. Configure containerd

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
sudo systemctl restart containerd
sudo systemctl enable containerd

C. Disable swap (required for Kubernetes)

sudo swapoff -a
sudo sed -i '/swap/d' /etc/fstab

D. Install Kubernetes binaries

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

2. Get join command from Control Plane

On the control-plane node:

kubeadm token create --print-join-command

Output example:

kubeadm join 10.10.10.10:6443 --token abcd12.xxxxxxxx \
  --discovery-token-ca-cert-hash sha256:123456789abcdef.....

3. Run join command on worker node

Paste on new worker node:

sudo kubeadm join 10.10.10.10:6443 --token abcd12.xxxxxxxx \
  --discovery-token-ca-cert-hash sha256:123456789abcdef.....

Wait 10–20 seconds.

4. Verify from control-plane

kubectl get nodes -o wide

You should now see the new node in Ready state.

PART 2 — Cordon, Uncordon, Drain (Maintenance Mode)

Scenario:

Node worker-1 needs maintenance. You must drain workloads safely, then bring node back.

1. Mark node unschedulable (cordon)

kubectl cordon worker-1

Verify:

kubectl get nodes

You will see:

worker-1   Ready,SchedulingDisabled

2. Drain the node safely

kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data

Flags meaning:

• --ignore-daemonsets → DaemonSet pods stay on the node

• --delete-emptydir-data → deletes local emptyDir data

• Without this flag, drain fails if pods use emptyDir

Check rescheduling:

kubectl get pods -o wide -n <your-namespace>

Pods should automatically move to other nodes.

3. Perform maintenance (simulated)

# e.g. update OS, restart node
sudo apt update
sudo reboot

4. Bring node back (uncordon)

kubectl uncordon worker-1

Verify:

kubectl get nodes

Node will be Ready.

PART 3 — DaemonSet Behavior Lab

Test that DaemonSet pods are NOT evicted

List all DaemonSet pods:

kubectl get daemonsets -A

Example:

• kube-proxy

• calico-node

• node-exporter

Now drain a node again:

kubectl drain worker-1 --ignore-daemonsets

You will see warnings like:

pod kube-proxy-xxxx is managed by DaemonSet, skipping

✔ DaemonSet pods stay on the node ✔ Kubernetes never evicts them ✔ They are recreated only if node restarts or DS is updated

This reinforces the maintenance concept.

PART 4 — Permanently Remove a Node From Cluster

Scenario:

You are removing worker-2 from the cluster completely.

STEP 1: Drain the node

Even for permanent removal, always drain:

kubectl drain worker-2 --ignore-daemonsets --delete-emptydir-data

STEP 2: Delete node from cluster

kubectl delete node worker-2

STEP 3: Reset kubeadm on the worker

Run on worker-2:

sudo kubeadm reset -f
sudo rm -rf /etc/cni/net.d
sudo ip link delete cni0
sudo ip link delete flannel.1 2>/dev/null

Optional clean-up:

sudo systemctl stop kubelet
sudo systemctl stop containerd

STEP 4: (Optional) Rejoin later

If you want this node back later:

• Reset

• Install kubeadm again

• Run new join command

OPTIONAL ADVANCED SCENARIOS FOR TRAINING

Scenario A: Drain with force

kubectl drain worker-1 --force --ignore-daemonsets

Use when:

• Pod is not part of a controller (bare pods)

• Pod cannot be recreated automatically ⚠️ Risk: you will cause downtime

Scenario B: Node is unreachable

If node is NotReady or Unknown:

kubectl delete node worker-3 --force --grace-period=0

Scenario C: Upgrade workflow

1. cordon

2. drain

3. OS upgrade + reboot

4. kubeadm upgrade

5. uncordon