LAB09: Fetching Kubelet Metrics

LAB 20.1: Fetching Kubelet metric endpoint

STEP 0 — Start the Proxy (required)

kubectl proxy --port=8001

This opens a local gateway:

http://127.0.0.1:8001/

Now your curl commands DO NOT need TLS, CA bundles, or certs. API server authenticates you automatically.

---

STEP 1 — List All Nodes (raw JSON)

curl -s http://127.0.0.1:8001/api/v1/nodes | jq

Only print node names:

curl -s http://127.0.0.1:8001/api/v1/nodes \
  | jq -r '.items[].metadata.name'

Example output:

k8s-cluster-m1
k8s-cluster-w1
k8s-cluster-w2

---

STEP 2 — KUBELET SUMMARY METRICS (MAIN ENTRYPOINT)

This is where Metrics Server gets its data.

Replace <node> with one from above:

curl -s http://127.0.0.1:8001/api/v1/nodes/<node>/proxy/stats/summary | jq

This gives:

• CPU usage (node + pods + containers)

• Memory usage

• Network stats

• Disk stats

• Ephemeral storage

• cgroup stats

This is the heart of kubelet metrics.

---

STEP 3 — List Pods on That Node (from kubelet)

curl -s http://127.0.0.1:8001/api/v1/nodes/<node>/proxy/pods | jq

This gives full pod specs as kubelet sees them.

---

STEP 4 — cAdvisor Metrics (raw Prometheus-style)

Yes — kubelet exposes cAdvisor through:

/metrics/cadvisor

Curl it:

curl -s http://127.0.0.1:8001/api/v1/nodes/<node>/proxy/metrics/cadvisor

This prints ALL low-level container metrics in Prometheus format, including:

• throttling

• cgroup CPU stats

• container memory pages

• IO stats

• filesystem stats

• network packets

This is massive — hundreds of metrics.

---

STEP 5 — Kubelet Internal Metrics

From /metrics endpoint:

curl -s http://127.0.0.1:8001/api/v1/nodes/<node>/proxy/metrics

Expose kubelet internals:

• pod lifecycle

• garbage collector

• volume plugin timing

• runtime stats

• kubelet request latency

• pod admission latency

---

STEP 6 — Resource Metrics (new endpoint)

curl -s http://127.0.0.1:8001/api/v1/nodes/<node>/proxy/metrics/resource

Contains:

• container CPU usage

• container memory usage

• node CPU

• node memory

Metrics Server reads from Summary API, not from this one. But Prometheus scrapes this too.

---

STEP 7 — Kubelet Health Endpoints

Live

curl -s http://127.0.0.1:8001/api/v1/nodes/<node>/proxy/healthz

Subsystems:

curl -s http://127.0.0.1:8001/api/v1/nodes/<node>/proxy/healthz/log
curl -s http://127.0.0.1:8001/api/v1/nodes/<node>/proxy/healthz/etcd
curl -s http://127.0.0.1:8001/api/v1/nodes/<node>/proxy/healthz/ping

---

STEP 8 — Get Container Logs Directly via Kubelet

Format:

/containerLogs/{namespace}/{pod}/{container}

Example:

curl -s \
  http://127.0.0.1:8001/api/v1/nodes/<node>/proxy/containerLogs/kube-system/coredns-xyz/coredns

---

STEP 9 — Direct Logs of Kubelet (if exposed)

Some clusters expose:

curl -s http://127.0.0.1:8001/api/v1/nodes/<node>/proxy/logs

(Not always available)

---

STEP 10 — FULL MENU of Kubelet Endpoints

Here is everything exposed behind kubelet’s secure port 10250, but through the API server proxy (so no TLS/Cert issues):

Endpoint	Purpose
/stats/summary	Main Metrics Server source
/metrics/cadvisor	Raw container metrics
/metrics/resource	Node+container CPU/mem
/metrics	Kubelet internal metrics
/pods	Pod list (as seen by kubelet)
/healthz	Kubelet health
/containerLogs/..	Pod/container logs
/configz	Kubelet config
/run/..	Runtime info
/debug/pprof/...	Profiling

You can curl all of these through:

http://127.0.0.1:8001/api/v1/nodes/<node>/proxy/<endpoint>

---

Correct usage: You must get one of the API resources in that group

The API resources inside metrics.k8s.io/v1beta1 are:

NodeMetrics
PodMetrics

Their resource names:

nodes.metrics.k8s.io
pods.metrics.k8s.io

So you can do:

---

1. Get node metrics

kubectl get nodes.metrics.k8s.io

or

kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes | jq

---

2. Get pod metrics

kubectl get pods.metrics.k8s.io -A

or

kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods | jq

---

3. Get schema via explain

kubectl explain nodes.metrics.k8s.io
kubectl explain pods.metrics.k8s.io

---

Helpful Table (This will make everything click)

What you type	What it is	Valid?
metrics.k8s.io	API Group	❌ cannot kubectl get
metrics.k8s.io/v1beta1	GroupVersion	❌ cannot kubectl get
nodes.metrics.k8s.io	API Resource	✅ can kubectl get
pods.metrics.k8s.io	API Resource	✅ can kubectl get
NodeMetrics	Kind	❌ cannot kubectl get (unless plural)
PodMetrics	Kind	❌ cannot kubectl get

---

Final working commands

All node metrics:

kubectl get nodes.metrics.k8s.io

All pod metrics:

kubectl get pods.metrics.k8s.io -A

Raw JSON:

kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes
kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods