Kubernetes Architecture

1. Why Kubernetes?

Running containers on a single host is easy. Running hundreds of microservices across dozens of machines is not. Kubernetes (K8s) solves the operational problems of distributed container orchestration:

• Scheduling - place containers on nodes with available CPU, memory, and affinity rules.
• Self-healing - restart failed containers, replace unhealthy Pods, reschedule when nodes die.
• Scaling - scale replicas up or down manually or automatically (HPA).
• Service discovery & load balancing - stable DNS names and virtual IPs for ephemeral Pods.
• Declarative config - describe desired state in YAML; controllers reconcile reality to match.
• Rolling updates & rollbacks - zero-downtime deploys with version history.
• Secrets & config management - inject configuration without baking it into images.
• Storage orchestration - attach cloud or network storage to any Pod.

2. Cluster Overview

A Kubernetes cluster consists of at least one control plane and one or more worker nodes. The control plane makes global decisions (scheduling, API, state storage). Worker nodes run your workloads.

┌──────────────────────────────────────────────────────────────────────┐
│                        CONTROL PLANE                                   │
│  ┌─────────────┐  ┌─────────────┐  ┌──────────────┐  ┌────────────┐  │
│  │ API Server  │  │  Scheduler  │  │  Controller  │  │   etcd     │  │
│  │  (REST)     │  │             │  │   Manager    │  │ (key-value)│  │
│  └──────┬──────┘  └──────┬──────┘  └──────┬───────┘  └─────┬──────┘  │
│         └────────────────┴────────────────┴─────────────────┘         │
│                    all components talk to API server                   │
└──────────────────────────────────────────────────────────────────────┘
         │                           │                           │
         ▼                           ▼                           ▼
┌─────────────────┐       ┌─────────────────┐       ┌─────────────────┐
│  Worker Node 1  │       │  Worker Node 2  │       │  Worker Node N  │
│  kubelet        │       │  kubelet        │       │  kubelet        │
│  kube-proxy     │       │  kube-proxy     │       │  kube-proxy     │
│  container rt   │       │  container rt   │       │  container rt   │
│  CNI plugin     │       │  CNI plugin     │       │  CNI plugin     │
│  ┌────┐ ┌────┐  │       │  ┌────┐ ┌────┐  │       │  ┌────┐         │
│  │Pod │ │Pod │  │       │  │Pod │ │Pod │  │       │  │Pod │ ...     │
│  └────┘ └────┘  │       │  └────┘ └────┘  │       │  └────┘         │
└─────────────────┘       └─────────────────┘       └─────────────────┘

High-availability control planes run multiple API server instances behind a load balancer, with etcd deployed as a 3- or 5-node quorum cluster. Worker nodes are cattle - add or remove them without manual reconfiguration.

Namespaces

Logical partitions inside a cluster. Resources in different namespaces can share the same physical nodes but are isolated by RBAC and naming.

kubectl get namespaces
# default, kube-system, kube-public, kube-node-lease  -  plus your own

kubectl create namespace production
kubectl get pods -n production

3. Control Plane Components

Every control plane component is a process (or Pod in managed clusters) that watches or writes cluster state through the API server. etcd is the sole source of truth for persisted state.

API Server (kube-apiserver)

The front door to the cluster. All clients (kubectl, kubelet, controllers, scheduler) communicate exclusively via the REST API. It validates requests, applies admission webhooks, persists objects to etcd, and serves watch streams.

• Authenticates requests (certs, bearer tokens, OIDC)
• Authorizes via RBAC, Node, or webhook modes
• Admits or mutates objects through admission controllers
• Exposes /api/v1, /apis/apps/v1, etc. grouped by API group

# Direct API call (requires valid credentials)
kubectl proxy &
curl http://localhost:8001/api/v1/namespaces/default/pods

# Raw request with verbose output
kubectl get pods -v=8

etcd

A consistent, distributed key-value store holding all cluster data: Pod specs, Secrets (encrypted at rest), ConfigMaps, RBAC policies, and more. Only the API server reads and writes etcd directly.

# etcd data layout (simplified)
/registry/pods/default/nginx-abc123
/registry/deployments/production/api
/registry/secrets/kube-system/bootstrap-token-xyz

Scheduler (kube-scheduler)

Watches for newly created Pods with spec.nodeName unset and selects a suitable node based on:

• Resource requests (CPU, memory, ephemeral storage)
• Node selectors, affinity, anti-affinity, taints/tolerations
• Pod topology spread constraints
• Custom scheduler plugins (scheduling framework)

apiVersion: v1
kind: Pod
metadata:
  name: gpu-worker
spec:
  nodeSelector:
    accelerator: nvidia-tesla-v100
  tolerations:
    - key: "gpu"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
  containers:
    - name: trainer
      image: ml-trainer:latest
      resources:
        requests:
          nvidia.com/gpu: 1

Controller Manager (kube-controller-manager)

Runs controller loops that watch API objects and drive the cluster toward desired state. Each controller is responsible for one resource type or relationship.

Cloud-specific controllers (e.g. cloud node lifecycle, route controllers) run in the cloud-controller-manager on cloud-hosted clusters.

4. Worker Node Components

Worker nodes host Pods. Each node runs a fixed set of agents that register with the API server and maintain local runtime state.

kubelet

The primary node agent. Registers the node, watches PodSpecs assigned to its node, and instructs the container runtime to start/stop containers. Reports node and Pod status back to the API server.

• Mounts volumes (ConfigMaps, Secrets, PVCs)
• Runs liveness, readiness, and startup probes
• Enforces Pod resource limits via cgroups
• Pulls images using configured credentials

# Check kubelet health and node status
kubectl get nodes -o wide
kubectl describe node worker-1

# kubelet logs (on the node itself)
journalctl -u kubelet -f

kube-proxy

Programs network rules on each node so Services work. Implements ClusterIP, NodePort, and LoadBalancer semantics via iptables, IPVS, or eBPF (depending on mode).

# Traffic flow for ClusterIP Service "api" on port 80
Client Pod  →  DNS resolves api.default.svc  →  virtual IP 10.96.0.10:80
             →  kube-proxy DNAT  →  one of Pod IPs :8080 (round-robin)

Container Runtime

Kubernetes talks to runtimes through the Container Runtime Interface (CRI). Supported runtimes:

# On a node  -  list running containers via crictl
sudo crictl ps
sudo crictl inspect <container-id>
sudo crictl logs <container-id>

CNI (Container Network Interface)

When kubelet creates a Pod sandbox, it calls the CNI plugin to assign an IP, configure routes, and set up the network namespace. See Networking & Services for CNI details.

┌─────────────────────────────────────────┐
│  Pod network namespace                  │
│  eth0  10.244.2.15/24                   │
│       │                                 │
│  veth pair ──► bridge cni0 on host      │
│              ──► overlay / BGP routing  │
└─────────────────────────────────────────┘

5. kubectl Mastery

kubectl is the CLI for the Kubernetes API. It reads kubeconfig (~/.kube/config) for cluster URL, credentials, and current context.

Essential commands

# Context and cluster info
kubectl config get-contexts
kubectl config use-context prod-eks
kubectl cluster-info

# CRUD operations
kubectl apply -f deployment.yaml      # declarative (preferred)
kubectl create deployment nginx --image=nginx:1.25
kubectl get pods -o wide -w           # watch mode
kubectl describe pod nginx-abc123
kubectl delete pod nginx-abc123 --grace-period=30

# Imperative edits
kubectl scale deployment api --replicas=5
kubectl set image deployment/api api=myregistry/api:v2.1
kubectl rollout status deployment/api
kubectl rollout undo deployment/api

Output formats

Debugging shortcuts

kubectl logs pod/api-xyz -c api --previous    # crashed container logs
kubectl exec -it pod/api-xyz -c api -- sh     # shell into container
kubectl port-forward svc/api 8080:80          # local access
kubectl top pods -n production                # metrics (needs metrics-server)
kubectl api-resources                         # list all resource types
kubectl explain pod.spec.containers           # OpenAPI docs in terminal

Generating YAML

kubectl create deployment nginx --image=nginx --dry-run=client -o yaml > deploy.yaml
kubectl run debug --image=busybox --restart=Never --dry-run=client -o yaml

6. API Resources & Objects

Everything in Kubernetes is an API object with apiVersion, kind, metadata, spec (desired state), and status (observed state).

Core object anatomy

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: production
  labels:
    app: api
    team: platform
  annotations:
    deployment.kubernetes.io/revision: "3"
spec:          # what you want
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: myregistry/api:1.2.0
status:        # what the cluster reports (read-only via kubectl)
  replicas: 3
  availableReplicas: 3
  conditions: [...]

Common API groups

Labels and selectors

Labels are key/value pairs on objects. Selectors connect resources: a Service selects Pods; a Deployment selects ReplicaSets; a ReplicaSet selects Pods.

metadata:
  labels:
    app: api
    env: production
    version: v2

# Service routes to Pods with matching labels
spec:
  selector:
    app: api
    env: production

Owner references and garbage collection

Child objects (Pods owned by ReplicaSets) have ownerReferences. Deleting a Deployment cascades to its ReplicaSets and Pods. Set orphanDependents or use --cascade=orphan to keep children.

kubectl delete deployment api --cascade=orphan   # keep ReplicaSets/Pods

7. The Reconciliation Loop

Kubernetes is built on the control loop pattern: observe current state, compare to desired state, act to reduce the difference, repeat.

         ┌──────────────────────────────────────────┐
         │           DESIRED STATE (spec)              │
         │   Deployment: replicas: 3, image: v2     │
         └──────────────────┬───────────────────────┘
                            │
    ┌───────────────────────▼───────────────────────┐
    │              API SERVER / etcd                 │
    └───────────────────────┬───────────────────────┘
                            │ watch
         ┌──────────────────┼──────────────────┐
         ▼                  ▼                  ▼
   Deployment ctrl    ReplicaSet ctrl     Scheduler
         │                  │                  │
         │ creates/         │ creates/         │ assigns
         │ updates RS       │ deletes Pods     │ nodeName
         ▼                  ▼                  ▼
   ReplicaSet          Pods (actual)      Worker Node
                            │
                     kubelet runs containers
                            │
                     status reported back ──► loop repeats

Example: scaling a Deployment

1. You run kubectl scale deployment api --replicas=5
2. API server updates Deployment spec in etcd
3. Deployment controller sees spec.replicas=5, current RS has 3 Pods
4. ReplicaSet controller creates 2 new Pods
5. Scheduler assigns them to nodes; kubelet starts containers
6. kubelet reports Pod status; Deployment status shows 5/5 available

Level-triggered vs edge-triggered

Kubernetes controllers are level-triggered: they reconcile the full desired state on every loop, not just the last change. If a Pod is deleted manually, the ReplicaSet controller recreates it without a new event. This makes the system resilient to missed events or restarts.

Finalizers

Objects with metadata.finalizers cannot be deleted until each finalizer is removed by its controller. Used for graceful cleanup (e.g. removing LoadBalancer IPs, volume detachment).

kubectl get pod stuck-terminating -o yaml | grep finalizers -A 3
# Remove finalizer only when you understand the risk (break-glass)
kubectl patch pod stuck-terminating -p '{"metadata":{"finalizers":[]}}' --type=merge

8. Cluster Setup Options

How you run Kubernetes depends on environment, team size, and operational appetite.

kind quick start

# Install: go install sigs.k8s.io/kind@latest
kind create cluster --name dev --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker
EOF

kubectl cluster-info --context kind-dev
kubectl get nodes

minikube quick start

minikube start --cpus=4 --memory=8192 --driver=docker
minikube addons enable ingress
minikube addons enable metrics-server
kubectl get pods -A

Production checklist

• HA control plane (managed or 3+ control plane nodes with kubeadm)
• etcd backups and tested restore procedure
• Node auto-repair and auto-scaling (cluster autoscaler)
• RBAC, Pod Security Standards, and network policies
• Separate node pools for system vs workload vs GPU workloads
• Upgrade strategy (surge upgrades, blue/green node pools)

9. Cheat Sheet

Full cheat sheet: kubectl Cheat Sheet