BLOG20b: Why Do We Need Kubernetes

BLOG11: Why do we need Kubernetes CRDs?

Where is the Deployment Controller?

The Deployment Controller is a built-in controller that runs inside the Kubernetes kube-controller-manager.

---

Location of the Deployment Controller

✔️ It is not in the API server

The API server only stores objects.

✔️ It is not in the Deployment object

The Deployment object carries only the desired state.

✔️ It is inside the kube-controller-manager binary

Specifically in the controller named:

deployment-controller

kube-controller-manager runs as a Pod in most clusters:

• kubeadm

• K3s

• AKS / EKS / GKE

• Rancher

• RKE2

• OpenShift (different)

Check it in kubeadm clusters:

kubectl -n kube-system get pods | grep controller-manager

Example:

kube-controller-manager-ip-172-31-1-20   Running   1/1

Inside that pod are multiple controllers.

---

What lives inside kube-controller-manager?

The controller-manager contains many controllers, not just Deployment:

Controller	Purpose
deployment-controller	Handles Deployments → manages ReplicaSets
replicaset-controller	Ensures Pod replicas
statefulset-controller	Manages StatefulSets
job-controller	Manages Jobs
cronjob-controller	Manages CronJobs
node-controller	Handles node health
service-controller	Syncs load balancers
namespace-controller	Cleans up namespaces
pv-controller	Manages PersistentVolumes
endpoint-controller	Builds Endpoints

And many more.

---

Where is it in Kubernetes source code?

If you browse the Kubernetes source:

kubernetes/pkg/controller/deployment/

On GitHub:

https://github.com/kubernetes/kubernetes/tree/master/pkg/controller/deployment

Inside you will find:

• deployment_controller.go

• rollout.go

• sync.go

• progress.go

These files implement:

• Rolling updates

• Rollbacks

• ReplicaSet management

• Scale behaviors

• Status updates

---

How It Actually Works

1. Deployment object is stored in etcd via API server

2. Deployment controller watches /apis/apps/v1/deployments

3. It detects new or changed Deployment

4. It creates or updates a ReplicaSet

5. ReplicaSet controller notices ReplicaSet changes

6. ReplicaSet controller creates or deletes Pods

7. Scheduler assigns Pods to nodes

8. Kubelet runs containers

Everything is loosely coupled via the API server. Controllers do not talk directly to each other — only through API objects.

---

Where does Deployment Controller run in managed Kubernetes?

On EKS

Runs inside control plane (you cannot see Pod) ✔️ But still a kube-controller-manager process

On GKE

Runs as part of Google’s internal master image

On AKS

Runs in Azure-managed controller-manager VM

On K3s

It is inside:

/usr/local/bin/k3s server

Because K3s compiles all controllers into a single binary.

On kubeadm

Visible as a Pod:

kube-controller-manager-master

---

Fast Recap

Item	Where it exists
Deployment controller logic	Inside kube-controller-manager
Deployment object	Stored in etcd via API server
Runs as	Process or Pod depending on Kubernetes distribution
Communicates via	Shared API use, not direct calls

---

Perfect — here is the full, end-to-end workflow to demonstrate a simple "Greeting" CRD + Controller that prints a message.

This is the smallest, cleanest CRD demo possible.

---

FULL STEPS: Simple Greeting CRD + Controller Demo

STEP 1 — Create the CRD (Greeting)

Save this as greetings-crd.yaml:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: greetings.example.com
spec:
  group: example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                message:
                  type: string
      subresources:
        status: {}
  scope: Namespaced
  names:
    plural: greetings
    singular: greeting
    kind: Greeting
    shortNames:
      - grt

Apply:

kubectl apply -f greetings-crd.yaml

Check:

kubectl get crd | grep greeting

---

STEP 2 — Create a Custom Greeting Resource

Save as greeting-sample.yaml:

apiVersion: example.com/v1
kind: Greeting
metadata:
  name: hello-demo
spec:
  message: "Hello from Kubernetes CRD demo!"

Apply:

kubectl apply -f greeting-sample.yaml

This creates your custom object:

kubectl get greetings
kubectl get greeting hello-demo -o yaml

So far:

• CRD exists

• Custom object exists But nothing “happens” yet — because we need a controller.

---

STEP 3 — Create a Tiny Controller (runs in a Pod)

This controller simply:

• Watches Greeting objects

• Prints the message

• Updates .status.printed = true

👉 Minimal controller code (Python version for simplicity)

Save as controller.py:

#!/usr/bin/env python3
import time
import json
from kubernetes import client, config, watch

config.load_incluster_config()
api = client.CustomObjectsApi()

while True:
    w = watch.Watch()
    for event in w.stream(api.list_namespaced_custom_object,
        group="example.com", version="v1", namespace="default", plural="greetings"):
        
        obj = event["object"]
        name = obj["metadata"]["name"]
        message = obj["spec"].get("message", "")

        print(f"Greeting '{name}': {message}", flush=True)

        # update status
        status = {"status": {"printed": True}}
        api.patch_namespaced_custom_object(
            "example.com", "v1", "default", "greetings", name, status)

---

STEP 4 — Create a Deployment for the Controller

We need a container that contains:

• the Python script

• kubernetes python client

Use a simple container definition:

Save as controller-deploy.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: greeting-controller
spec:
  replicas: 1
  selector:
    matchLabels:
      app: greeting-controller
  template:
    metadata:
      labels:
        app: greeting-controller
    spec:
      serviceAccountName: greeting-sa
      containers:
        - name: controller
          image: python:3.10
          command: ["python3", "/controller/controller.py"]
          volumeMounts:
            - name: controller-code
              mountPath: /controller
      volumes:
        - name: controller-code
          configMap:
            name: greeting-controller-code

---

STEP 5 — Put Controller Code in a ConfigMap

Create ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: greeting-controller-code
data:
  controller.py: |
    # (paste the python code here)

Apply:

kubectl apply -f greeting-controller-code.yaml

---

STEP 6 — Give Controller Permission (RBAC)

Create service account + permission:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: greeting-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: greeting-role
rules:
  - apiGroups: ["example.com"]
    resources: ["greetings", "greetings/status"]
    verbs: ["get", "list", "watch", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: greeting-rb
subjects:
  - kind: ServiceAccount
    name: greeting-sa
roleRef:
  kind: Role
  name: greeting-role
  apiGroup: rbac.authorization.k8s.io

Apply:

kubectl apply -f greeting-rbac.yaml

---

STEP 7 — Deploy the Controller

kubectl apply -f controller-deploy.yaml

View logs:

kubectl logs -l app=greeting-controller -f

You should see the message:

Greeting 'hello-demo': Hello from Kubernetes CRD demo!

Check status update:

kubectl get greeting hello-demo -o yaml

You will see:

status:
  printed: true

---

Finished — You demonstrated:

• Custom API (CRD)

• Custom object

• Reconciliation controller

• Status updates

• Watch loop

• Real CRD lifecycle

This is a complete, minimal operator-style CRD demo, perfect for:

• training

• presentations

• interviews

• workshops

• showing Kubernetes API-first design

---

Here are the simplest and most useful CRDs you can install with Helm and immediately demonstrate in under 5 minutes — with zero coding, real behavior, and clear understanding.

These CRDs are easy, visual, and perfect for demos.

---

1. Metrics Server (Metrics / CPU Usage CRD)

Helm Chart:

helm upgrade --install metrics-server metrics-server/metrics-server --namespace kube-system

Creates CRDs:

• none, but adds metrics APIs:

  • /apis/metrics.k8s.io/v1beta1/nodes

  • /apis/metrics.k8s.io/v1beta1/pods

Amazing 2-minute demo:

kubectl top nodes
kubectl top pods -A

Shows real CPU & memory usage.

🟢 Simple 🟢 Fast 🟢 Very useful demo

---

2. cert-manager (Certificate CRDs + automatic TLS)

The MOST POPULAR CRD used in real clusters.

Install:

helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --set installCRDs=true

Installs CRDs:

• ClusterIssuer

• Issuer

• Certificate

• CertificateRequest

• Order

• Challenge

Super simple demo:

apiVersion: cert-manager.io/v1
kind: SelfSignedIssuer
metadata:
  name: demo

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: mycert
spec:
  secretName: mycert-tls
  issuerRef:
    name: demo
  dnsNames:
    - demo.example.com

Apply:

kubectl apply -f .
kubectl get certificate
kubectl describe certificate mycert
kubectl get secret mycert-tls -o yaml

⭐ Shows:

• CRD

• Controller

• Status updates

• Secret creation

---

3. ArgoCD (Application CRD)

One of the BEST CRD demos for beginners.

Install:

helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd -n argocd --create-namespace

Installs CRDs:

• Application

• AppProject

Quick demo:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo-app
spec:
  source:
    repoURL: https://github.com/argoproj/argocd-example-apps
    path: guestbook
  destination:
    server: https://kubernetes.default.svc
    namespace: default

Apply:

kubectl apply -f app.yaml
kubectl get applications.argoproj.io

Then watch ArgoCD sync and deploy Pods automatically.

Great for “GitOps powered by CRDs” demo.

---

4. Prometheus Operator (ServiceMonitor / PodMonitor CRDs)

Install:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack -n monitoring --create-namespace

Installs CRDs:

• ServiceMonitor

• PodMonitor

• PrometheusRule

Simple demo:

Monitor nginx using ServiceMonitor.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nginx-monitor
spec:
  selector:
    matchLabels:
      app: nginx
  endpoints:
    - port: http

Shows:

• CRD creates auto-monitoring

• Prometheus reacts instantly

---

5. KEDA (ScaledObject CRD → Autoscaling on external metrics)

KEDA is simple but shows a powerful effect.

Install:

helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda --namespace keda --create-namespace

CRDs created:

• ScaledObject

• TriggerAuthentication

• ScaledJob

2-minute demo:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: nginx-scale
spec:
  scaleTargetRef:
    name: nginx
  triggers:
    - type: cpu
      metadata:
        type: Utilization
        value: "20"

Then create load → autoscaling happens automatically.

---

6. ExternalDNS (DNSRecord CRD)

Perfect demo of cloud DNS automation.

Install:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install externaldns bitnami/external-dns \
  -n externaldns --create-namespace

CRDs (optional depending on provider):

• DNSEndpoint

Demo:

apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint
metadata:
  name: test
spec:
  endpoints:
    - dnsName: demo.example.com
      recordType: A
      targets:
        - "1.2.3.4"

DNS record gets auto-created.

---

7. Sealed Secrets (SealedSecret CRD)

Encrypted secrets, safe in Git.

Install:

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets -n kube-system

CRDs:

• SealedSecret

Super simple demo:

echo -n "mypassword" | kubectl create secret generic demo-secret \
  --dry-run=client --from-literal=password=/dev/stdin -o yaml \
  | kubeseal > sealed.yaml

Then apply:

kubectl apply -f sealed.yaml
kubectl get secret demo-secret

Shows:

• Encrypted CRD → real secret generated

---

8. Simple Demo CRD: "Hello" CRD (From Kubebuilder Test)

Install the sample operator:

helm repo add kubebuilder https://example-fake/kubebuilder

But since you want existing charts, this is lower priority.

---

Best 3 For Quick Demo (Beginner-Friendly)

CRD	Time to demo	Visual effect
cert-manager	⭐⭐⭐⭐ (2–3 mins)	Automatically creates TLS cert
ArgoCD	⭐⭐⭐⭐ (3 mins)	Deploys an app via CRD
KEDA	⭐⭐⭐⭐ (3 mins)	Autoscaling happens automatically

---

How cert-manager controller is installed

When you install cert-manager with Helm:

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --set installCRDs=true

Helm installs two types of components:

---

1️⃣ CRDs (API layer)

These define the custom API types:

• Certificate

• Issuer

• ClusterIssuer

• CertificateRequest

• Order

• Challenge

Stored inside: apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition

These tell Kubernetes:

“Hey, there is a new resource type called Certificate. Users can create it.”

But CRDs alone do nothing. They are just schemas stored in the API server.

---

2️⃣ Controller Deployments (Code layer)

After CRDs are installed, Helm installs CERT-MANAGER CONTROLLERS:

You will see 3 Pods:

kubectl -n cert-manager get pods

Example output:

cert-manager-5b87d9c984-x2r79
cert-manager-cainjector-6f54db7c9c-hj2lf
cert-manager-webhook-79d4695454-vbp4m

These are the controllers (actual code):

✅ cert-manager

• Watches Certificate, Issuer, ClusterIssuer

• Issues certificates

• Updates status fields

• Creates Secrets with TLS keys

• Talks to ACME / CA endpoints

✅ cert-manager-cainjector

• Injects CA bundles into CRDs and webhook configs

✅ cert-manager-webhook

• Validates and mutates Certificate API objects

• Ensures CRD fields are correct before storing

• Adds defaults

All these are Go binaries packaged inside containers.

Each pod runs Go programs from cert-manager source code:

GitHub: https://github.com/cert-manager/cert-manager (don’t click; just referencing context)

---

So what actually happens when you apply a Certificate CRD?

Example:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: demo-cert
spec:
  secretName: demo-tls
  issuerRef:
    name: selfsigned
  dnsNames:
    - demo.com

⏳ Controller flow:

1. Certificate object stored in etcd

2. cert-manager controller loop sees it

3. Controller understands:

   • Need to generate keypair

   • Need sign certificate

4. Controller creates:

   • Secret (key + certificate)

5. Updates CRD status:

   status:
     conditions:
       - type: Ready
         status: "True"
     notAfter: 2026-01-01T00:00:00Z

---

Why you thought "controller must be code"? — you're right.

CRDs = API types Controllers = Code implementing behavior

You always need both for custom APIs:

Component	Purpose
CRD	Defines the schema of the resource
Controller	Implements the logic (create certs, update status, etc.)

Cert-manager's helm chart installs both.

---

Where does the actual controller code run?

Inside Kubernetes:

• As Deployments

• As Pods

• Using service accounts

• Using RBAC

• Watching the CRDs

• Taking actions

Example Deployment:

kubectl -n cert-manager get deployment cert-manager -o yaml

You will see something like:

spec:
  template:
    spec:
      containers:
        - name: cert-manager
          image: quay.io/jetstack/cert-manager-controller:v1.14.0

This image contains the Go controller code.

---

Summary (Simple)

Component	What it is	Installed by Helm?
CRDs	API definitions (schemas)	✔️ Yes
Controllers	Go code running in Pods	✔️ Yes
Webhook	Validates CRD objects	✔️ Yes
CA injector	Injects CA bundles	✔️ Yes

Helm installs everything needed to make cert-manager work.

---

Great — this is the core understanding of Kubernetes architecture. Let me explain it clearly and simply.

---

1. Does a Kubernetes API resource exist only if a controller exists?

✔️ Yes and No — depends on what you mean by “exist.”

🔹 YES — if you want behavior

A Kubernetes resource definition alone (CRD or built-in resource) does nothing unless there is a controller watching it.

Example:

apiVersion: apps/v1
kind: Deployment

Deployment has meaning only because Kubernetes has:

• deployment controller

• replicaset controller

• pod controller

• kubelet

Without controllers, a Deployment YAML would just sit in etcd doing nothing.

🔹 NO — the object itself can exist without a controller

You can create a CRD without writing a controller.

It will exist in the API server:

kubectl get mycrd
kubectl get myresource

But:

• No Pods will be created

• No logic will run

• No reconciliation happens

• No status updates

It is like creating a database table with no application using it.

---

2. Why Kubernetes created CRDs? (Why resort to CRDs?)

This is the most important design decision in Kubernetes history.

💡 Reason #1 — Kubernetes needed to be EXTENSIBLE

Originally Kubernetes had only built-in types:

• Pods

• Services

• Deployments

• Jobs

But companies wanted:

• Kafka clusters

• Redis clusters

• Ingress controllers

• Cert management

• GitOps

• Security policies

• Cloud resource management (RDS, S3, etc.)

It was impossible to include all these as built-in Kubernetes objects.

So Kubernetes added CRDs so ANYONE can create new APIs inside Kubernetes.

---

Reason #2 — To implement “Kubernetes as a Platform”

CRDs allow Kubernetes to become a platform for building platforms.

Example:

• ArgoCD brings Application resource → GitOps platform

• cert-manager brings Certificate resource → TLS automation platform

• Crossplane brings RDSInstance/Bucket → Cloud provisioning platform

• Istio brings VirtualService → Service mesh platform

Kubernetes becomes the foundation layer.

---

Reason #3 — Unified API for everything

Instead of exposing multiple systems with different APIs:

• AWS API

• Terraform

• Jenkins pipelines

• Database provisioning scripts

CRDs allow everything to be controlled via one consistent interface:

kubectl apply -f <any-resource>.yaml
kubectl get <any-type>
kubectl describe <any-type>

This is extremely powerful.

---

Reason #4 — Operators (Automation on top of CRDs)

CRD + Controller = Operator pattern.

Operators can automate:

• Day-2 operations

• Upgrades

• Failovers

• Scaling

• Backups

• Validation

• Drift correction

This allows companies to encode human operational knowledge into software.

Instead of running scripts manually.

---

Reason #5 — Decoupling API Server from Controllers

The API server does NOT need to know what the controller does.

Flow:

1. CRD registers new API

2. Controller watches that new API

3. Controller manages Kubernetes or external resources

4. Kubernetes itself stays minimal and clean

This avoids coupling business logic into Kubernetes core.

---

Reason #6 — CRDs allow Kubernetes to evolve without changing the core code

No need to:

• fork Kubernetes

• recompile the API server

• modify built-in resources

Just define new APIs dynamically.

Example:

If you want your own object:

apiVersion: business.io/v1
kind: Customer

You can add it without touching Kubernetes source.

---

Summary — Why CRDs?

Reason	Explanation
Extensibility	Add new APIs without modifying Kubernetes itself
Platform on top of platform	Build GitOps, Mesh, DB operators
Consistency	Everything managed via kubectl & YAML
Automation	Operators implement real logic
Decoupling	API server stays generic, controllers do the work
Ecosystem growth	Thousands of operators exist today

---

3. Final Answer (Simple Version)

✔️ Kubernetes resources need controllers to “do” anything

Objects are only data. Controllers give them life.