Interview: Helm & Observability

Helm is the package manager for Kubernetes. A chart is a versioned package of templated YAML manifests (Chart.yaml, values.yaml, templates/). A release is a deployed instance of a chart in a cluster - you can install the same chart multiple times with different release names. Helm 3 stores release state as Secrets in the target namespace (no Tiller). Use helm upgrade --install for idempotent deployments.

Templates in templates/ are Go templates with Sprig functions. values.yaml provides defaults; overrides come from -f values-prod.yaml, --set key=value, or --set-file. Access values with {{ .Values.replicaCount }}, release metadata with {{ .Release.Name }}, and reusable snippets via {{ include "mychart.fullname" . }}. Preview rendered output with helm template before applying.

helm install my-api ./mychart -f values-prod.yaml creates revision 1. helm upgrade my-api ./mychart --set image.tag=2.4.0 creates revision 2. helm history my-api lists revisions; helm rollback my-api 1 reverts. Use helm upgrade --install in CI/CD for idempotency. Dry-run with helm install --dry-run --debug to inspect rendered manifests and catch template errors early.

Helm hooks run Kubernetes Jobs or other resources at lifecycle points: pre-install, post-install, pre-upgrade, pre-delete, etc. Annotate with helm.sh/hook, helm.sh/hook-weight (execution order), and helm.sh/hook-delete-policy. Common uses: database migrations before upgrade, smoke tests after install, backup before delete. Hooks run outside the normal release manifest ordering.

Prometheus pulls (scrapes) metrics from HTTP endpoints on an interval. In K8s, use the kube-prometheus-stack Helm chart for Prometheus, Alertmanager, and node-exporter. Expose app metrics at /metrics and discover targets via ServiceMonitor CRDs (Prometheus Operator) or Pod annotations (prometheus.io/scrape). Key categories: cluster health (node CPU, pending Pods), workload RED metrics (rate, errors, duration), and resource utilization vs requests/limits.

A ServiceMonitor is a Prometheus Operator CRD that declaratively defines which Services to scrape (selector, namespace, port, path, interval). Annotation-based discovery (prometheus.io/scrape: "true") is simpler but less flexible. ServiceMonitors support TLS, bearer tokens, relabeling, and multi-namespace selection - preferred in production Prometheus Operator setups installed via Helm.

Install Grafana alongside Prometheus via kube-prometheus-stack or the Loki stack. Pre-built dashboards cover cluster, node, and workload views. Import community dashboards from grafana.com (IDs 315, 6417, 12006). Manage dashboards as code with ConfigMaps labeled grafana_dashboard: "1" or the Grafana Operator. Set up alerts in Grafana or Alertmanager for SLO burn rates, Pod crash loops, and PVC capacity thresholds.

Kubernetes captures container stdout/stderr per Pod but does not aggregate cluster-wide. Common stacks: EFK (Elasticsearch, Fluent Bit, Kibana), Loki (label-based, pairs with Grafana), or cloud-native (CloudWatch, Cloud Logging). Deploy log agents as DaemonSets. Best practices: log JSON to stdout, include correlation IDs, avoid file-based logging inside containers, and set retention policies per environment.

Distributed tracing tracks requests across microservices with trace IDs and spans. OpenTelemetry is the vendor-neutral instrumentation standard (SDK + collector). Export traces to Jaeger, Grafana Tempo, or Zipkin. Deploy an OpenTelemetry Collector as a DaemonSet or sidecar to receive, batch, and forward telemetry. Correlate traces with logs (trace ID in log lines) and metrics (exemplars) for faster incident response.

A ResourceQuota caps total resources a namespace can consume (CPU, memory, Pod count, PVC count, Services). A LimitRange sets defaults and min/max per Pod or container in a namespace. Use quotas on multi-tenant clusters to prevent one team from exhausting cluster capacity. Combine with requests and limits on every container; without requests, quotas cannot enforce fairly.

Metrics: request rate, error rate (5xx), p95/p99 latency, Pod restarts, CPU/memory vs requests, HPA current/desired replicas. Alerts: error rate > 1% for 5 min, latency p99 > SLO threshold, Pods not Ready, deployment replica mismatch, certificate expiry. Use the RED method (Rate, Errors, Duration) for services and USE method (Utilization, Saturation, Errors) for infrastructure. Route alerts through Alertmanager with severity tiers and runbooks.

Run helm status my-release and helm history my-release for revision state. Inspect rendered templates: helm get manifest my-release. Check Kubernetes events: kubectl get events. For hook failures, find hook Jobs with kubectl get jobs. Use helm rollback if the new revision is broken. Lint before deploy: helm lint ./mychart. In CI, always run helm template and kubeconform or helm unittest to catch errors pre-deploy.