LAB03c: Upgrade & Disaster Recovery

LAB700: Upgrade & Disaster Recovery

Install etcdtl

Install latest etcdctl (official upstream)

1. Download the latest etcd release

ETCD_VER=v3.5.15
wget https://github.com/etcd-io/etcd/releases/download/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz

1. Extract

tar xvf etcd-${ETCD_VER}-linux-amd64.tar.gz

1. Move binaries to /usr/local/bin

sudo mv etcd-${ETCD_VER}-linux-amd64/etcdctl /usr/local/bin/
sudo mv etcd-${ETCD_VER}-linux-amd64/etcdutl /usr/local/bin/
sudo chmod +x /usr/local/bin/etcdctl /usr/local/bin/etcdutl

1. Verify

etcdctl version

Connect to etcd inside Kubernetes (kubeadm)

Your etcd is secured and listens on:

https://127.0.0.1:2379

Use kubeadm’s built-in certs:

sudo ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  get "" --prefix --keys-only

Check cluster health:

sudo etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  endpoint health

Snapshot etcd:

sudo etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  snapshot save snapshot.db

K3s cluster

Default k3s cluster database can be Sqlite

1. Check What Type of DB It Is

Run:

file /var/lib/rancher/k3s/server/db/state.db

You should see something like:

SQLite 3.x database

1. Install SQLite Client (If Not Installed)

sudo apt update
sudo apt install -y sqlite3

1. Open the Database in Read-Only Mode (SAFE)

sudo sqlite3 -readonly /var/lib/rancher/k3s/server/db/state.db

You’ll enter the SQLite shell:

sqlite>

Exit anytime with:

.quit

---

1. List All Tables

Inside sqlite:

.tables

You’ll see tables like:

• kine

• cluster

• endpoints

• registry

• leases

K3s stores Kubernetes data via Kine (SQL-backed etcd replacement).

---

1. View Table Schema

Example:

.schema kine

---

1. Query Data (Read-Only)

Example: View latest records

SELECT * FROM kine ORDER BY id DESC LIMIT 10;

Search for nodes:

SELECT * FROM kine WHERE name LIKE '%node%';

Search for secrets (metadata only, still encrypted):

SELECT * FROM kine WHERE name LIKE '%secret%';

Never open without -readonly in production:

sqlite3 state.db   ❌ DANGEROUS

Never edit this DB while K3s is running.

---

1. Safe Backup Before Any Inspection

Always back up first:

sudo cp /var/lib/rancher/k3s/server/db/state.db \
/root/state.db.backup.$(date +%F)

If the database is Etcd

Connect to etcd in K3s K3s stores certs in:

/var/lib/rancher/k3s/server/tls/

Command:

sudo ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
  --cert=/var/lib/rancher/k3s/server/tls/etcd/client.crt \
  --key=/var/lib/rancher/k3s/server/tls/etcd/client.key \
  get "" --prefix

Using UI Explorer

1. Setup etcd gatekeeper

wget https://github.com/evildecay/etcdkeeper/releases/download/v0.7.8/etcdkeeper-v0.7.8-linux_x86_64.tar.gz

1. Next Steps: Extract & run etcdkeeper

tar xvf etcdkeeper-v0.7.8-linux_x86_64.tar.gz
cd etcdkeeper-v0.7.8-linux_x86_64
chmod +x etcdkeeper
# ./etcdkeeper

1. Start EtcdKeeper WITH connection parameters

sudo ./etcdkeeper \
  -h 0.0.0.0 \
  -p 8080 \
  -usetls \
  -cacert /etc/kubernetes/pki/etcd/ca.crt \
  -cert /etc/kubernetes/pki/etcd/healthcheck-client.crt \
  -key /etc/kubernetes/pki/etcd/healthcheck-client.key

Reference

1. Etcd Backup & Restore Lab