Kubernetes Networking & Services

1. The Kubernetes Networking Model

Kubernetes enforces four fundamental networking requirements:

1. Every Pod gets its own IP address and can communicate with every other Pod without NAT.
2. Nodes can communicate with all Pods without NAT.
3. The IP a Pod sees itself as is the same IP others see it as.
4. Services provide stable virtual IPs that load-balance across Pod backends.

This flat, routable Pod network is implemented by a CNI (Container Network Interface) plugin on each node. Common choices: Calico, Cilium, Flannel, AWS VPC CNI, Azure CNI.

┌─────────────────────────────────────────────────────────────────┐
│  Node 10.0.1.5                                                  │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐          │
│  │ Pod 10.244.1.3│  │ Pod 10.244.1.4│  │ Pod 10.244.1.5│          │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘          │
│         └─────────────────┴─────────────────┘                    │
│                           │ bridge / overlay (CNI)              │
│                    kubelet + container runtime                  │
└─────────────────────────────────────────────────────────────────┘
         Pod-to-Pod traffic routes across nodes via CNI overlay or
         cloud VPC routing  -  no port mapping required.

2. Services

A Service is a stable network endpoint that selects Pods by label and load-balances traffic to them. kube-proxy programs iptables or IPVS rules on every node to implement the Service virtual IP (ClusterIP).

Service types

ClusterIP Service

apiVersion: v1
kind: Service
metadata:
  name: api
  namespace: production
spec:
  type: ClusterIP
  selector:
    app: api
    tier: backend
  ports:
    - name: http
      port: 80          # Service port (what clients connect to)
      targetPort: 8080  # Container port (where app listens)
      protocol: TCP
  sessionAffinity: None  # or ClientIP for sticky sessions

NodePort Service

apiVersion: v1
kind: Service
metadata:
  name: api-nodeport
spec:
  type: NodePort
  selector:
    app: api
  ports:
    - port: 80
      targetPort: 8080
      nodePort: 30080   # must be 30000–32767; auto-assigned if omitted

LoadBalancer Service

apiVersion: v1
kind: Service
metadata:
  name: api-lb
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
spec:
  type: LoadBalancer
  selector:
    app: api
  ports:
    - port: 443
      targetPort: 8443

Headless Service (StatefulSet pattern)

apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  clusterIP: None   # headless  -  no virtual IP
  selector:
    app: postgres
  ports:
    - port: 5432
# DNS returns individual Pod A records:
# postgres-0.postgres.default.svc.cluster.local
# postgres-1.postgres.default.svc.cluster.local

Endpoints and EndpointSlices

When you create a Service, Kubernetes automatically creates Endpoints (or EndpointSlices in newer clusters) listing the IP:port of every matching Pod. If no Pods match the selector, the Endpoints object is empty and the Service has no backends.

kubectl get endpoints api -o wide
kubectl get endpointslices -l kubernetes.io/service-name=api
kubectl describe svc api

3. Ingress & Gateway API

Ingress exposes HTTP/HTTPS routes from outside the cluster to Services. It requires an Ingress Controller (nginx-ingress, Traefik, AWS ALB Ingress Controller, etc.) to actually program the load balancer or reverse proxy.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - api.example.com
      secretName: api-tls
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /v1
            pathType: Prefix
            backend:
              service:
                name: api
                port:
                  number: 80
    - host: admin.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: admin-ui
                port:
                  number: 80

pathType values

• Prefix - matches path prefix (most common)
• Exact - exact path match only
• ImplementationSpecific - behaviour depends on the controller

Gateway API (successor to Ingress)

The Gateway API is a newer, more expressive standard with separate Gateway, HTTPRoute, and GRPCRoute resources. It supports advanced traffic splitting, header-based routing, and cross-namespace references. Use it on new clusters when your CNI/ingress stack supports it.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-route
spec:
  parentRefs:
    - name: main-gateway
  hostnames:
    - api.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v1
      backendRefs:
        - name: api
          port: 80
          weight: 90
        - name: api-canary
          port: 80
          weight: 10

4. Cluster DNS

Every cluster runs CoreDNS (or kube-dns) as a cluster add-on. Pods are configured with DNS settings so they can resolve Services by name.

# DNS naming convention:
# <service>.<namespace>.svc.cluster.local
# Short names work within the same namespace:
#   curl http://api:80
# Cross-namespace:
#   curl http://api.production.svc.cluster.local

# Pod DNS (StatefulSet):
# <pod-name>.<headless-service>.<namespace>.svc.cluster.local

# Debug DNS from a debug pod
kubectl run -it --rm debug --image=busybox:1.36 --restart=Never -- sh
nslookup api.production.svc.cluster.local
wget -qO- http://api.production:80/health

# Check CoreDNS
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

Custom DNS policies per Pod:

spec:
  dnsPolicy: ClusterFirst   # default  -  cluster DNS, then upstream
  # dnsPolicy: Default      # inherit node DNS
  # dnsPolicy: None         # use dnsConfig explicitly
  dnsConfig:
    nameservers:
      - 1.1.1.1
    searches:
      - custom.svc.cluster.local
    options:
      - name: ndots
        value: "2"

5. Network Policies

By default, all Pods can talk to all Pods. NetworkPolicy resources restrict ingress and egress traffic using label selectors - like a firewall for Pods.

Deny all ingress, allow only from frontend

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-ingress-only
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080

Namespace-scoped policy

  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
        - podSelector:
            matchLabels:
              app: prometheus

Egress restriction (allow DNS + database only)

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-egress-restrict
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Egress
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    - to:
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

6. CNI Plugins

# Check CNI on a node
ls /etc/cni/net.d/
cat /etc/cni/net.d/*.conflist

# Cilium connectivity test
cilium connectivity test

7. Service Meshes

A service mesh adds a sidecar proxy (Envoy in Istio/Linkerd) to every Pod, providing:

• mTLS between all services (zero-trust networking)
• Advanced traffic management (canary, circuit breaking, retries, timeouts)
• Observability (distributed tracing, golden metrics per service)
• L7 routing and authorization policies

# Istio  -  inject sidecar into namespace
kubectl label namespace production istio-injection=enabled

# Canary with Istio VirtualService (90/10 split)
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: api
spec:
  hosts:
    - api
  http:
    - route:
        - destination:
            host: api
            subset: stable
          weight: 90
        - destination:
            host: api
            subset: canary
          weight: 10

8. Troubleshooting

# Service has no endpoints?
kubectl get pods -l app=api          # do labels match Service selector?
kubectl describe svc api             # check selector and endpoints

# Pod can't reach another Pod?
kubectl run -it --rm netshoot --image=nicolaka/netshoot --restart=Never -- bash
curl -v telnet://api.production:80
traceroute 10.244.1.5

# Check kube-proxy
kubectl get pods -n kube-system -l k8s-app=kube-proxy
kubectl logs -n kube-system -l k8s-app=kube-proxy --tail=20

# Ingress not routing?
kubectl describe ingress app-ingress
kubectl get ingressclass
kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx

# NetworkPolicy blocking traffic?
kubectl describe networkpolicy -n production
# Temporarily remove policies to isolate the issue (dev only)

9. Cheat Sheet

Full cheat sheet: kubectl Cheat Sheet