LAB18: Etcd Backup & Restore

LAB 900a: Etcd Backup & Restore

1. Listing Kubernetes objects from etcd (pods, deployments, secrets…)

2. Taking a FULL etcd snapshot (correct way for kubeadm)

3. Restoring the cluster from the snapshot (disaster recovery)

PART 1 — LIST OBJECTS IN ETCD (Kubernetes registry)

List ALL keys (huge output)

sudo ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  get /registry/ --prefix --keys-only

---

List all PODS stored in etcd

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  get /registry/pods/ --prefix --keys-only

Example output:

/registry/pods/default/nginx
/registry/pods/kube-system/coredns-12345

---

List deployments

sudo etcdctl ... get /registry/deployments/ --prefix --keys-only

---

List secrets

sudo etcdctl ... get /registry/secrets/ --prefix --keys-only

---

List configmaps

sudo etcdctl ... get /registry/configmaps/ --prefix --keys-only

---

View a single object (raw protobuf)

sudo etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  get /registry/pods/default/nginx

(Remember: RAW output is binary protobuf and unreadable.)

---

PART 2 — FULL ETCD BACKUP (Kubeadm Production-Safe)

Create a snapshot

sudo ETCDCTL_API=3 etcdctl snapshot save /root/etcd-snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key

Confirm:

etcdutl snapshot status /root/etcd-snapshot.db

You will see:

total keys: 12345
revision: 6789

---

PART 3 — RESTORE ETCD (DISASTER RECOVERY)

⚠️ Must stop kube-apiserver and etcd first. ⚠️ Restore must be done on ONE control-plane node at a time.

---

Stop the static etcd pod

sudo mv /etc/kubernetes/manifests/etcd.yaml /etc/kubernetes/manifests/etcd.yaml.bak

Wait 20 seconds → kubelet stops etcd.

---

Restore snapshot to a new data dir

sudo ETCDCTL_API=3 etcdctl snapshot restore /root/etcd-snapshot.db \
  --data-dir=/var/lib/etcd-restore

This creates:

/var/lib/etcd-restore/member/

---

Update etcd.yaml to point to restored data-dir

Edit:

/etc/kubernetes/manifests/etcd.yaml

Find:

--data-dir=/var/lib/etcd

Replace with:

--data-dir=/var/lib/etcd-restore

Save → kubelet will automatically restart etcd.

---

Wait until etcd is running again

Check:

docker ps | grep etcd

Or:

crictl ps | grep etcd

---

Bring kube-apiserver back

If needed:

mv /etc/kubernetes/manifests/kube-apiserver.yaml.bak /etc/kubernetes/manifests/kube-apiserver.yaml

---

PART 4 — Verify cluster health after restore

kubectl get nodes
kubectl get pods -A
kubectl get deployments -A
kubectl get svc -A

If everything works → recovery successful.

---

PART 5 — Optional: Automate backups

Create cron job:

sudo crontab -e

Add:

0 */6 * * * ETCDCTL_API=3 etcdctl snapshot save /var/backups/etcd-$(date +\%F-\%H).db --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key