BLOG20c: What is the Function of

BLOG12: What is the function of Controller? What is the concept of watch?

---

🟦 How does a controller know when a new CRD object is created?

Answer: The controller does not ask Kubernetes directly. Instead, it uses the Kubernetes Watch API, which is built into every API resource automatically — including CRDs.

Controllers "subscribe" to changes.

---

🟩 1. Kubernetes API provides a Watch endpoint for every resource

When you create a CRD:

greetings.example.com/v1

Kubernetes automatically generates:

GET /apis/example.com/v1/namespaces/<ns>/greetings?watch=true

This is a built-in capability of the API server — no extra code required.

So yes, Kubernetes API helps. The CRD gains watch/stream support from the API server automatically.

---

🟧 2. A Controller uses the Watch API to get events (add/update/delete)

A controller usually does:

watch.Interface = client.Watch(
  resource = "greetings",
  group = "example.com",
  version = "v1",
  namespace = "default",
)

This maintains a streaming connection.

Events it receives:

• ADDED → new object created

• MODIFIED → spec or status changed

• DELETED → object removed

So the controller reacts when the API server sends these events.

---

🟦 3. Controllers use “informers” to efficiently watch objects

In real operators, controllers use client-go informers:

• informer watches API server

• caches objects

• triggers reconcile loops when events happen

Example:

DeploymentInformer -> triggers DeploymentController
GreetingInformer -> triggers your custom controller

This is low-latency and extremely efficient.

---

🟨 4. Does Kubernetes API help? YES — the API server is the hub

🔹 Controllers do not directly read ETCD 🔹 Controllers do not poll for changes 🔹 Controllers do not scan the cluster

The Kubernetes API server is the central mediator.

The flow:

✔️ Step 1: You apply a CRD

API server registers new REST endpoints

✔️ Step 2: You create a CRD instance

API server stores it in etcd

✔️ Step 3: Controller opens a watch request

API server streams events to the controller

✔️ Step 4: Controller reconciles

Uses APIs → create pods, update status, etc.

✔️ Step 5: Status updates

Controller PATCHes status field

---

🟥 5. Think of the API server as a “Message Broker"

Analogy:

Component	Role
API server	Kafka broker
CRD objects	Topics
Controllers	Consumers
Clients applying YAML	Producers

Controllers are continuously “listening" to new messages (events).

---

🟩 6. CRDs automatically get WATCH support — no extra code required

You do not need to modify Kubernetes.

When you install a CRD:

• API server auto-generates:

  • list endpoint

  • get endpoint

  • watch endpoint

  • status subresource

  • validation via OpenAPI

CRDs become first-class API resources.

---

🟧 7. In simple words

Controllers know new CRD objects because:

✔️ API server sends notifications through WATCH streams

✔️ Controller libraries (informers) subscribe to those streams

✔️ CRDs inherit watch behavior from Kubernetes API machinery

✔️ Controller code reacts whenever API server reports an event

---

⭐ FINAL EXPLANATION (SHORT VERSION)

How does a controller detect new CRD objects?

Because Kubernetes API server automatically publishes events for every resource (built-in or CRD). The controller subscribes to these events using the Watch API. This is how controllers remain informed without polling.

---

How do Kubernetes watches differ from generic webhooks?

Let’s make these two concepts crystal clear:

1️⃣ Is WATCH like a webhook? 2️⃣ How the RECONCILIATION LOOP works (the beating heart of Kubernetes)

Both are extremely important to understand Kubernetes internals, CRDs, and Controllers.

---

🟦 1️⃣ Is WATCH like a webhook?

Short answer:

✅ WATCH is not a webhook, but it behaves similar in purpose. ❌ They are not the same mechanism.

Let’s compare.

---

🔵 WATCH (Kubernetes Watch API)

• A long-running HTTP connection

• Controller opens a stream to the API server

• API server pushes events (ADDED, MODIFIED, DELETED) as they happen

• The controller keeps the connection open and continuously listens

• Native Kubernetes feature—no config needed

Example:

GET /apis/apps/v1/deployments?watch=true

The API server keeps the connection open and streams events:

{
  "type": "ADDED",
  "object": {...}
}
{
  "type": "MODIFIED",
  "object": {...}
}

---

🔴 Webhook

• API server calls your HTTP endpoint

• You register it using a webhook configuration

• Used for validate or mutate incoming objects

• Runs BEFORE objects are stored in etcd

• Not for watching changes—only for intercepting API calls

---

🟢 WATCH vs Webhook (Simple Comparison)

Feature	WATCH	Webhook
Mechanism	Long-lived streaming	API server calls external HTTP server
Purpose	Detect changes to objects	Validate/Mutate object on create/update
Direction	API server → Controller	Controller/Webhook server ← API server
When triggered	After object stored	Before object stored
Who uses it	Controllers, operators	Admission controllers

Summary:

WATCH = “Notify me whenever something changes.” Webhook = “I want to approve/modify requests before they are accepted.”

---

🟩 2️⃣ The Reconciliation Loop (The Soul of Kubernetes)

The Reconcile Loop is the core idea behind all Kubernetes controllers (built-in + CRD).

It includes:

• Deployment controller

• ReplicaSet controller

• KEDA controller

• cert-manager controller

• Your custom controller

🟧 WHAT does Reconciliation mean?

“Try to make actual state = desired state. Repeat forever.”

---

🟦 Let’s break it down:

A. USER applies desired state

Example:

kind: Deployment
spec:
  replicas: 3

API server stores:

• Deployment object

• .spec.replicas = 3

This is desired state.

---

B. CONTROLLER sees the event via WATCH

The Deployment controller receives event:

Deployment ADDED/MODIFIED: replicas=3

---

C. CONTROLLER checks the cluster

The controller asks:

• How many ReplicaSets exist?

• How many Pods exist?

Example actual state:

actual pods = 1
desired pods = 3

---

D. CONTROLLER takes action

🟢 If pods missing → create pods 🔴 If pods too many → delete pods 🟡 If pod template changed → do rolling update

Controller attempts to correct the cluster.

---

E. Controller updates STATUS

Example:

status:
  replicas: 3
  availableReplicas: 3
  updatedReplicas: 3

STATUS shows actual state.

---

F. REPEAT FOREVER

Controllers run loops forever:

while true:
    watch → compare → fix → update → repeat

This is the Kubernetes self-healing model.

---

🟨 Reconcile Loop: Simple Example (Concept Demo)

Let’s imagine CRD:

spec:
  message: "Hello"

Reconcile process:

watch event:
  object: Greeting
desired state = message "Hello"
actual state = status.printed = false

action:
  print message
  update status.printed = true

On next iteration:

desired state = Hello
actual state = printed=true
no action needed

This is reconciliation.

---

🟧 Controller Pattern in One Sentence

Every controller is a loop that continuously tries to make the real world match the desired world stored in the API server.

---

🟩 Reconciliation Loop (Diagram)

            +-------------------------+
            |     User Applies YAML   |
            |   (desired state)       |
            +------------+------------+
                         |
                         v
                 +-------+-------+
                 | API Server    |
                 | Stores Object |
                 +-------+-------+
                         |
                 WATCH   |   EVENT
                         v
                 +-------+-------+
                 | Controller    |
                 | Reconcile()   |
                 +-------+-------+
                         |
                 ACTION  |  TO FIX
                         v
                 +-------+-------+
                 | Kubernetes    |
                 | (Pods, RS…)   |
                 +-------+-------+
                         |
                     ACTUAL STATE
                         |
                         v
                 Reconcile checks again...

---

🟦 ⭐ FINAL SUMMARY

Is Watch like webhook?

• ❌ Not the same

• ✔️ Watch = streaming events

• ✔️ Webhook = validation/mutation call

What is Reconciliation Loop?

• Controller continuously watches changes

• Compares desired vs actual

• Fixes the difference

• Updates status

• Repeats forever

• This is how Kubernetes self-heals

---

Does the Kubernetes API server open a persistent streaming connection to controllers, and can you directly see this?**

This is the real internal working of Kubernetes: the API server keeps a persistent HTTP streaming connection open to the controller so it can deliver events in real time. However, you usually cannot see this connection directly (like you would with a typical socket listing), because controller libraries abstract and hide these networking details.

Let’s break down how this works internally, and explain how (or if) you can observe it.

---

🟦 1. What is the “stream connection” between API server → Controller?

When a controller uses:

?watch=true

It calls the Kubernetes API server like this:

GET https://<api-server>/apis/example.com/v1/greetings?watch=true

The API server does:

• Accepts the HTTP request

• Does NOT close it

• Starts streaming JSON events over the same connection

• Sends messages like:

{"type":"ADDED", "object":{...}}
{"type":"MODIFIED", "object":{...}}
{"type":"DELETED", "object":{...}}

This is called:

✔️ HTTP/1.1 Chunked Transfer Encoding

or

✔️ Long-lived HTTP Watch stream

It is NOT WebSocket, NOT gRPC — just a long-running HTTP.

This is how controllers get events in real time.

---

🟧 2. Can you SEE this connection?

❌ You CANNOT normally see the watch connection inside the cluster by default

Why?

• Controllers (like cert-manager, deployment-controller) run internal Go code

• They use client-go libraries

• The connection is inside the controller pod’s memory

• The TCP stream is internal to pod → API server

• You don't get system-level socket tracing by default

BUT…

✔️ You CAN see:

A. API server metrics showing WATCH connections

If you have metrics-server or Prometheus:

apiserver_watch_events_total
apiserver_longrunning_gauge

There is a metric:

apiserver_longrunning_gauge{verb="WATCH",resource="pods"}

This shows active watch streams.

---

B. Inspect the controller logs

Most controllers log:

starting watch
event received: ADDED
event received: MODIFIED

Example (cert-manager controller logs):

I1024 controller.go:123] Starting watch for Certificates
I1024 controller.go:145] Certificate event: ADD

This proves the watch loop is active.

---

C. Using tcpdump (advanced)

You can capture the TCP stream from inside a controller pod:

kubectl exec -it <controller-pod> -- tcpdump -n host <APISERVER_IP> -vv

You'll see one long-running TCP connection from controller → kube-apiserver.

---

D. Using lsof inside controller container

(if the container allows it):

lsof -p 1 | grep TCP

You will see open TCP connections, one of them a long-running session to API server's port 443.

---

E. APIServer access logs (in non-managed clusters)

If you're running kubeadm or k3s, you CAN see API server watch traffic:

journalctl -u kube-apiserver | grep watch

Or on k3s:

sudo cat /var/log/k3s.log | grep WATCH

You’ll see entries like:

WATCH /api/v1/pods?watch=true client=controller-manager

❗ Important:

In managed clusters (EKS, AKS, GKE) — you can't access these logs.

---

🟨 3. How to manually see an API Watch stream (demo)

You can observe a watch connection manually:

A. Watch Pods:

kubectl get pods -w -n default

This uses the SAME mechanism that controllers use.

B. Raw HTTP Watch using curl:

TOKEN=$(kubectl get secret $(kubectl get sa default -o jsonpath='{.secrets[0].name}') -o jsonpath='{.data.token}' | base64 -d)
APISERVER=$(kubectl config view -o jsonpath='{.clusters[0].cluster.server}')

curl -k -H "Authorization: Bearer $TOKEN" \
  "$APISERVER/api/v1/pods?watch=true"

Now open another terminal and create/delete a Pod. You'll see the real watch events stream:

{"type":"ADDED", "object":{...}}
{"type":"MODIFIED", "object":{...}}

This is exactly what controllers see.

---

🟩 4. NOW — Detailed Concept: How Controllers Use Watch + Reconcile Loop

Every controller does:

STEP 1 — Open Watch

GET /apis/.../resources?watch=true

STEP 2 — Receive event

API server pushes:

type="ADDED"
object={...}

STEP 3 — Queue the event

Controller adds to workqueue.

STEP 4 — Reconcile

Controller compares:

• desired state (spec)

• actual state (pods, secrets, etc.)

STEP 5 — Fix differences

Create/delete/update underlying resources.

STEP 6 — Update status

PATCH the .status field.

STEP 7 — Repeat forever

Watch keeps firing events.

---

🟦 ⭐ Final Summary

✔️ The controller knows new CRD objects through WATCH streams

✔️ This is a long-running HTTP streaming connection

✔️ Implemented by the Kubernetes API server

✔️ Controllers subscribe using client-go informers

✔️ You can observe streams with:

• API server metrics

• logs

• tcpdump

• curl watch

• kubectl get -w

Does HTTPS Support Long-Lived Streaming Connections for Kubernetes Watch?

Yes, HTTPS fully supports long-lived streaming connections—and Kubernetes Watch uses standard HTTP/1.1 streaming over TLS.

Let's break it down clearly.

---

🟦 1. WATCH is built on top of standard HTTPS (HTTP/1.1 long-lived streaming)

Even though the connection is HTTPS, it still supports:

• long-lived connections

• chunked transfer encoding

• streaming responses

• server-pushed events

• continuous JSON payloads

This is not WebSocket This is not HTTP/2 push This is pure HTTP/1.1 over TLS.

---

🟩 2. How HTTPS supports this?

HTTPS is simply:

TLS encryption over TCP

Underneath, Kubernetes uses HTTP/1.1 over TLS, and HTTP/1.1 allows:

✔️ Persistent connections (Connection: keep-alive)

✔️ Chunked Transfer Encoding (Transfer-Encoding: chunked)

✔️ Unlimited response time

✔️ Streaming partial responses

Kubernetes uses chunked encoding to send each event as a new JSON chunk.

Example raw stream:

chunk 1:
{"type":"ADDED", "object":{...}}

chunk 2:
{"type":"MODIFIED", "object":{...}}

chunk 3:
{"type":"DELETED", "object":{...}}

Each chunk is sent without closing the connection.

SSL/TLS does NOT block this.

TLS will simply encrypt each chunk and stream it.

---

🟧 3. Why HTTPS doesn’t break streaming

TLS works at the transport layer:

Application (HTTP) → TLS encryption → TCP

So:

• HTTP semantics remain the same

• Only encryption is added

• HTTP chunking works normally

• Long-lived connections remain open

Nothing in TLS stops:

• long-running requests

• large responses

• streaming multiple chunks

• server-push events

This is why things like:

• kubectl logs -f

• kubectl get -w

• Kubernetes WATCH

• Event streaming

• Server-sent events (SSE)

All work perfectly over HTTPS.

---

🟨 4. Let’s see a real WATCH request from API server

The API server responds:

HTTP headers:

HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive

Then the body is streamed chunk-by-chunk.

Chunk example:

24
{"type": "ADDED", "object": { ... }}

19
{"type": "MODIFIED", ...}

0

Where:

• 24 is the hex size of content

• 0 signals end (but API server rarely sends 0, it keeps open)

---

🟪 5. Why not use WebSockets instead?

WebSockets also support streaming, but Kubernetes avoids them because:

❌ WebSockets add protocol overhead

❌ Harder to maintain reconnect logic

❌ Harder to debug and proxy

❌ Not compatible with all HTTP intermediaries

HTTP streaming is:

• simpler

• stable

• works with all reverse proxies

• more compatible with API server watch semantics

---

🟥 6. Can we manually observe HTTPS streaming?

YES.

Try:

kubectl get pods -w

or raw:

curl -k -H "Authorization: Bearer $TOKEN" \
  "$APISERVER/api/v1/pods?watch=true"

This will show:

{"type":"ADDED"...}
{"type":"MODIFIED"...}

That is HTTPS streaming in action.

---

🟫 7. How client-go handles streaming under TLS

client-go uses:

• http.Client

• TLS transport (Go stdlib)

• Goroutine that reads chunks

• Reflector + Informer

• Event queue

• Reconcile loop

So the flow is:

HTTPS STREAM → client-go → informer → event handler → reconcile()

TLS does not change anything except encryption.

---

⭐ FINAL SUMMARY

✔️ Kubernetes WATCH uses HTTP/1.1 streaming over TLS (HTTPS)

✔️ HTTPS fully supports:

• long-lived connections

• chunked responses

• streaming events

• continuous data flow

• no need to close connection

✔️ TLS does NOT prevent streaming

It only encrypts the bytes — protocol behavior is unchanged.

---