Helm & Observability

1. Helm Basics

Helm is the package manager for Kubernetes. A chart is a collection of templated YAML manifests. A release is a deployed instance of a chart in a cluster.

# Install Helm
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# Add a chart repository
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Search charts
helm search repo nginx
helm search hub postgres

2. Chart Anatomy

mychart/
├── Chart.yaml          # chart metadata (name, version, dependencies)
├── values.yaml         # default configuration values
├── charts/             # dependent charts (subcharts)
├── templates/          # Kubernetes manifest templates
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── ingress.yaml
│   ├── _helpers.tpl    # reusable template functions
│   └── NOTES.txt       # post-install instructions
└── .helmignore

Chart.yaml

apiVersion: v2
name: myapp
description: My application Helm chart
type: application
version: 1.2.0        # chart version
appVersion: "2.3.1"   # application version
dependencies:
  - name: postgresql
    version: "12.x.x"
    repository: https://charts.bitnami.com/bitnami
    condition: postgresql.enabled

3. Templates & Values

Helm uses Go templates with Sprig functions. Values from values.yaml (and overrides) are injected into templates.

values.yaml

replicaCount: 3

image:
  repository: mycompany/api
  tag: "2.3.1"
  pullPolicy: IfNotPresent

service:
  type: ClusterIP
  port: 80

ingress:
  enabled: true
  host: api.example.com
  tls: true

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 512Mi

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilization: 70

deployment.yaml template

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "myapp.fullname" . }}
  labels:
    {{- include "myapp.labels" . | nindent 4 }}
spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}
  selector:
    matchLabels:
      {{- include "myapp.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "myapp.selectorLabels" . | nindent 8 }}
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - containerPort: 8080
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080

Common template functions

{{ .Values.replicaCount }}           # access value
{{ .Release.Name }}                 # release name
{{ .Release.Namespace }}            # target namespace
{{ include "myapp.fullname" . }}    # call named template
{{ .Values.ingress.enabled | ternary "enabled" "disabled" }}
{{- if .Values.ingress.enabled }}   # conditional block
{{- range .Values.env }}            # loop
{{- toYaml .Values.resources | nindent 12 }}  # indent YAML block

4. Helm Workflow

# Install a release
helm install my-api bitnami/nginx \
  --namespace production \
  --create-namespace \
  --set replicaCount=3 \
  --set service.type=LoadBalancer

# Install with values file
helm install my-api ./mychart -f values-prod.yaml -f values-secrets.yaml

# Upgrade (deploy new version)
helm upgrade my-api ./mychart --set image.tag=2.4.0

# Upgrade or install (idempotent)
helm upgrade --install my-api ./mychart -f values-prod.yaml

# Rollback
helm rollback my-api 2        # rollback to revision 2
helm history my-api

# Uninstall
helm uninstall my-api -n production

# Dry-run (render templates without applying)
helm template my-api ./mychart -f values-prod.yaml
helm install my-api ./mychart --dry-run --debug

# Lint chart
helm lint ./mychart

# Package chart for distribution
helm package ./mychart
helm push mychart-1.2.0.tgz oci://registry.example.com/charts

Helm hooks

Run Jobs at specific lifecycle points:

metadata:
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "1"
    "helm.sh/hook-delete-policy": before-hook-creation

5. Prometheus & Metrics

Prometheus scrapes metrics from Pods and stores them as time-series data. Install via kube-prometheus-stack Helm chart for a full monitoring stack.

helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace

Expose metrics from your app

# Annotate Pod/Service for auto-discovery
metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
    prometheus.io/path: "/metrics"

ServiceMonitor (Prometheus Operator)

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: api-metrics
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: api
  namespaceSelector:
    matchNames:
      - production
  endpoints:
    - port: http
      interval: 30s
      path: /metrics

Key metrics to monitor

PromQL examples

# CPU usage as % of request
sum(rate(container_cpu_usage_seconds_total{namespace="production"}[5m])) by (pod)
  / sum(kube_pod_container_resource_requests{resource="cpu"}) by (pod) * 100

# HTTP error rate
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m])) * 100

# Pods not ready
kube_pod_status_ready{condition="false", namespace="production"}

6. Grafana Dashboards

Grafana visualises Prometheus (and other) data sources. The kube-prometheus-stack includes pre-built dashboards for cluster, node, and workload monitoring.

# Port-forward Grafana (default admin/prom-operator)
kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80

# Import community dashboards from grafana.com/dashboards
# Popular IDs: 315 (Kubernetes cluster), 6417 (Node Exporter), 12006 (K8s resources)

Dashboard-as-code with Grafana Operator or provisioning files:

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboard-api
  labels:
    grafana_dashboard: "1"
data:
  api-dashboard.json: |
    { "title": "API Service", ... }

7. Logging

Kubernetes does not have a built-in log aggregation system. Common stacks:

# View Pod logs
kubectl logs deployment/api -n production
kubectl logs -f pod/api-7d8f9c-abc12 -c app --tail=100
kubectl logs --previous pod/api-7d8f9c-abc12   # crashed container

# Logs from all Pods matching label
kubectl logs -l app=api -n production --tail=50

# Install Loki stack via Helm
helm install loki grafana/loki-stack \
  --namespace monitoring \
  --set promtail.enabled=true \
  --set grafana.enabled=true

Structured logging best practices

• Log to stdout/stderr (not files) - Kubernetes captures container streams
• Use JSON format for machine parsing
• Include correlation IDs for request tracing
• Set log levels appropriately (INFO in prod, DEBUG in dev)

8. Distributed Tracing

Distributed tracing tracks requests across microservices. OpenTelemetry is the standard instrumentation API.

# OpenTelemetry Collector deployment (receives traces, exports to backend)
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel-collector
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
    exporters:
      otlp/jaeger:
        endpoint: jaeger-collector:4317
    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [otlp/jaeger]

Instrument apps with OpenTelemetry SDKs - auto-instrumentation available for Python, Node.js, Java, Go.

9. Resource Quotas & Limits

ResourceQuota limits total resource consumption per namespace. LimitRange sets defaults and constraints per Pod/container.

ResourceQuota

apiVersion: v1
kind: ResourceQuota
metadata:
  name: production-quota
  namespace: production
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi
    pods: "50"
    persistentvolumeclaims: "10"
    services.loadbalancers: "2"

LimitRange

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: production
spec:
  limits:
    - type: Container
      default:
        cpu: 500m
        memory: 512Mi
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      max:
        cpu: "2"
        memory: 4Gi
      min:
        cpu: 50m
        memory: 64Mi

# Check quota usage
kubectl describe resourcequota -n production
kubectl describe limitrange -n production

# PriorityClasses for critical workloads
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "Critical production services"

10. Cheat Sheet

Full cheat sheets: kubectl · YAML Recipes