BLOG09c: Prometheus Metric Collection Flow

BLOG26: Prometheus Metric collection flow

---

What type of resource is Prometheus Adapter?

Prometheus Adapter is a Deployment

Installed via Helm → creates:

• Deployment (pods running the adapter)

• Service

• APIService (custom.metrics.k8s.io/v1beta1)

• ConfigMap (rules)

• ClusterRole / ClusterRoleBinding

• ServiceAccount

Important: Prometheus Adapter registers itself as a Kubernetes API extension:

custom.metrics.k8s.io

But the core object running it is a Deployment.

---

What type of resource is PodMonitor?

PodMonitor is a Custom Resource (CRD)

Created by the Prometheus Operator (kube-prometheus-stack).

CRD:

monitoring.coreos.com/v1
kind: PodMonitor

It tells Prometheus:

“Scrape these pods, at this path, on this port.”

It does NOT run anything — it’s just configuration that the Prometheus Operator converts into Prometheus scrape configs.

---

Now the Whole Metric Exposure Flow — Brief, Beautiful & Complete

The following sections provide a 10-second summary you can use in your slides or training:

---

Step 1 — Application exposes metrics

Your dummy app exposes:

/metrics → myapp_queue_length

in Prometheus format.

---

Step 2 — PodMonitor tells Prometheus to scrape

PodMonitor selects the pods and creates the scrape config.

Prometheus Operator loads the config → Prometheus begins scraping.

---

Step 3 — Prometheus stores the metric

Prometheus stores:

myapp_queue_length{pod="dummy-queue-app-XYZ"}

---

Step 4 — Prometheus Adapter translates Prometheus metrics → K8s Custom Metrics API

Adapter rule converts:

Prometheus query → Kubernetes API metric

Example internal query:

myapp_queue_length{namespace="queue-demo", pod="..."}

Adapter exposes it at:

/apis/custom.metrics.k8s.io/v1beta1/namespaces/queue-demo/pods/*/myapp_queue_length

Now Kubernetes can “see” the metric.

---

Step 5 — HPA fetches metric from Custom Metrics API

HPA runs a loop:

GET custom.metrics.k8s.io/v1beta1/.../myapp_queue_length

Compares:

• current queue length

• desired target

---

Step 6 — HPA scales the Deployment

If queue > target:

scale up (replicas++)

If queue < target:

scale down (replicas--)

---

Full Metric Flow Diagram (ASCII)

Copy-paste friendly:

                      ┌──────────────────────┐
                      │   Dummy App (Pods)   │
                      │ exposes /metrics     │
                      │ myapp_queue_length   │
                      └──────────┬───────────┘
                                 │
                                 ▼
                     ┌────────────────────────┐
                     │   PodMonitor (CRD)     │
                     │  “scrape these pods”   │
                     └──────────┬────────────┘
                                │ generated config
                                ▼
                     ┌────────────────────────┐
                     │   Prometheus Operator  │
                     │   creates scrape jobs  │
                     └──────────┬────────────┘
                                │
                                ▼
                     ┌────────────────────────┐
                     │      Prometheus        │
                     │ stores metric values   │
                     └──────────┬────────────┘
                                │ PromQL query
                                ▼
              ┌────────────────────────────────────┐
              │       Prometheus Adapter (Deploy)  │
              │ translates PromQL → K8s API metric │
              └──────────┬─────────────────────────┘
                         │ exposes:
                         │ custom.metrics.k8s.io
                         ▼
      ┌─────────────────────────────────────────────────────┐
      │     Kubernetes Custom Metrics API (APIService)      │
      └──────────┬──────────────────────────────────────────┘
                 │ HPA queries:
                 │ /apis/custom.metrics.k8s.io/…/myapp_queue_length
                 ▼
        ┌──────────────────────────┐
        │ HPA (HorizontalPodAutoscaler)
        │ decides scale up/down
        └──────────┬──────────────┘
                   ▼
         ┌──────────────────────┐
         │  Deployment replicas │
         └──────────────────────┘

---

In One Line

Pod → PodMonitor → Prometheus → Prometheus Adapter → Custom Metrics API → HPA → Scaling

---