LAB09a: The 3-Piece Orchestra

LAB21: Scale your Deployment based on custom application metrics already exposed in Prometheus.

---

The 3-Piece Orchestra

To make HPA understand Prometheus metrics, you need:

1. Application exposes metrics → Prometheus scrapes them

2. Prometheus Adapter → Translates Prometheus metrics into Kubernetes Custom Metrics API

3. HPA using custom metric → References the metric name via metrics.custom.metrics.k8s.io

---

Know more about app

image

Setup

Custom metric autoscaling with:

• Your Node.js queue app

• Prometheus (scraping the pod)

• Prometheus Adapter (exposing metric to Kubernetes)

• HPA that scales when myapp_queue_length > 10

This is the full, end-to-end pipeline for K3s, K8s, or RKE2. If you follow these steps in order, autoscaling will work.

---

PART 0 — Prerequisites

You need:

• kubectl

• helm

• k3s/k8s cluster running

• internet access for images

---

PART 1 — Deploy Your Dummy Queue App

Your app exposes one metric:

myapp_queue_length

Let's deploy it properly with annotations so Prometheus scrapes it.

---

1.1 — Namespace

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: queue-demo

Apply:

kubectl apply -f namespace.yaml

---

1.2 — Deployment (with scrape annotations)

queue-app-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dummy-queue-app
  namespace: queue-demo
  labels:
    app: dummy-queue-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: dummy-queue-app
  template:
    metadata:
      labels:
        app: dummy-queue-app
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - name: dummy-queue-app
        image: openlabfree/tiny-toy-traffic:v1
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          name: http
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 2
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10

---

1.3 — Service

queue-app-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: dummy-queue-app
  namespace: queue-demo
spec:
  selector:
    app: dummy-queue-app
  ports:
  - name: http
    port: 80
    targetPort: 8080

Apply:

kubectl apply -f queue-app-deployment.yaml
kubectl apply -f queue-app-service.yaml

---

---

For HPA custom metric scaling, you MUST have:

• Prometheus running

• Prometheus scraping your dummy queue app

• Prometheus Adapter running

• Adapter rules mapping your metric

No Prometheus → HPA custom metrics will never appear.

Why Metrics Server Cannot Do Custom Metrics

The Metrics Server is a tiny gardener in the Kubernetes garden. It only harvests two crops:

• CPU usage

• Memory usage

That’s it. No queue length, no RPS, no latency, no business metrics. It’s intentionally minimal and lightweight.

Setting up . . .

1. Add Helm Repo

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

1. Install Prometheus Stack

helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

This gives you:

• Prometheus server

• Alertmanager

• Grafana

• Node exporter, kube-state-metrics, etc.

By default, this Prometheus will scrape pods with prometheus.io/scrape: "true".

Check pods:

kubectl get pods -n monitoring

1. Install Prometheus Adapter This exposes custom metrics to Kubernetes.

helm install prometheus-adapter prometheus-community/prometheus-adapter \
  -n monitoring

1. Create Adapter Rules file adapter-values.yaml

prometheus:
  url: http://monitoring-kube-prometheus-prometheus.monitoring.svc
  port: 9090
  portName: prometheus

rules:
  custom:
    - seriesQuery: 'myapp_queue_length'
      resources:
        overrides:
          namespace:
            resource: namespace
          pod:
            resource: pod
      name:
        as: myapp_queue_length
      metricsQuery: 'myapp_queue_length{<<.LabelMatchers>>}'

1. Apply the custom rules

helm upgrade prometheus-adapter prometheus-community/prometheus-adapter \
  -n monitoring -f adapter-values.yaml

1. Create PodMonitor to scrape your queue app It only scrapes:

• ServiceMonitor

• PodMonitor

• Node exporter

• Kube-state-metrics

• Control-plane components

• etc. Annotations are ignored.

podmonitor-queue.yaml

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: dummy-queue-monitor
  namespace: monitoring
  labels:
    release: monitoring          # <--- REQUIRED!!!
spec:
  namespaceSelector:
    matchNames:
      - queue-demo
  selector:
    matchLabels:
      app: dummy-queue-app
  podMetricsEndpoints:
    - port: http
      path: /metrics
      interval: 10s

1. Verify adapter exposes metric

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .

curl -s http://localhost:9090/api/v1/targets | jq . | grep dummy-queue-app

Once PodMonitor is active, the chain unlocks:

1. Prometheus scrapes myapp_queue_length

2. Prometheus Adapter exposes it via custom.metrics.k8s.io

3. HPA can finally read it

4. Autoscaling begins

Verify Prometheus is Scraping App

1. Port forward:

kubectl port-forward svc/monitoring-kube-prometheus-prometheus -n monitoring 9090

Open browser

http://localhost:9090/targets

Add HPA with Custom Metric

queue-app-hpa.yaml

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: dummy-queue-hpa
  namespace: queue-demo
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: dummy-queue-app

  minReplicas: 1
  maxReplicas: 10

  metrics:
  - type: Pods
    pods:
      metric:
        name: myapp_queue_length
      target:
        type: AverageValue
        averageValue: "10"

Apply

kubectl apply -f queue-app-hpa.yaml

Check

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/queue-demo/pods/*/myapp_queue_length" | jq .

If success, Your Prometheus adapter is now:

• correctly connected to Prometheus

• successfully discovering your metric

• mapping it to Kubernetes pod identities

• exposing it via custom.metrics.k8s.io API

Test Auto-scaling

for i in $(seq 1 30); do
  curl -s -X POST http://dummy-queue-app.queue-demo.svc.cluster.local/enqueue
done

kubectl get hpa -n queue-demo -w

YOU ARE DONE. Your custom-metric autoscaler is fully working.

Your app → Prometheus → Adapter → HPA → Autoscaling pipeline is now operational.

References:

1. Metric collection flow

Debugging Steps

1. Confirm Prometheus Adapter is actually

kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus-adapter

1. Get logs from the Prometheus Adapter

ADAPTER_POD=$(kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus-adapter -o jsonpath="{.items[0].metadata.name}")
kubectl logs -n monitoring $ADAPTER_POD

1. Test adaptor endpoint

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/queue-demo/pods/*/myapp_queue_length"

Some time we need to Remove ALL default adapter container rules

prometheus:
  url: http://monitoring-kube-prometheus-prometheus.monitoring.svc:9090/

rules:
  default: false   # <-- VERY IMPORTANT
  custom:
    - seriesQuery: 'myapp_queue_length'
      resources:
        overrides:
          namespace:
            resource: "namespace"
          pod:
            resource: "pod"
      name:
        as: "myapp_queue_length"
      metricsQuery: 'myapp_queue_length{<<.LabelMatchers>>}'