Deployment & Serving
Getting a model into production reliably.
What are the main serving frameworks for LLMs and how do they differ?
- vLLM - open-source, known for PagedAttention and continuous batching, very strong throughput for open-weight models, widely used default.
- TGI (Text Generation Inference, Hugging Face) - production-ready serving with quantization support, good HF ecosystem integration.
- NVIDIA Triton / TensorRT-LLM - highly optimized for NVIDIA hardware, strong for teams needing maximum raw throughput and willing to invest in tuning.
- Managed APIs (OpenAI, Anthropic, Bedrock, Vertex) - no infra to manage at all, pay-per-token, best when you don't need a self-hosted open-weight model.
When should you self-host a model vs use a managed API?
Self-hosting makes sense when you have high, predictable volume (cost amortization beats per-token API pricing), need data to never leave your infrastructure (compliance/privacy), need fine-grained control (custom fine-tunes, latency tuning), or need to avoid vendor rate limits. Managed APIs make sense for lower/unpredictable volume, faster time-to-market, access to frontier-quality models you couldn't feasibly host yourself, and avoiding the operational burden of GPU infra, scaling, and monitoring.
How do you think about autoscaling for LLM inference workloads?
LLM inference has an unusually slow "cold start" (loading multi-gigabyte weights onto a GPU can take tens of seconds to minutes), so naive request-based autoscaling (like typical web services) causes severe latency spikes during scale-up. Common mitigations: keep a warm minimum pool of instances, predictive/scheduled scaling based on known traffic patterns, and separating scaling decisions from per-request signals by using queue depth or GPU utilization as the scaling trigger instead.
How do you version and roll out changes to prompts, models, or fine-tunes safely?
- Version prompts and configs like code (git), not as strings scattered through application logic.
- Use canary/shadow deployments - route a small % of traffic to the new version and compare metrics before full rollout.
- Keep automated evaluation suites that run on every prompt/model change, not just at initial launch.
- Maintain the ability to instantly roll back to the previous known-good prompt/model version.
What is a model gateway/router and why do production systems use one?
A gateway sits between application code and multiple LLM providers/models, handling routing (e.g. cheap model for easy queries, powerful model for hard ones), fallback (retry on a different provider if one is down/rate-limited), unified logging/cost tracking, and caching - decoupling application code from any single provider's API and enabling multi-model strategies without rewriting call sites.
How do you handle rate limits and reliability when calling third-party LLM APIs at scale?
Use exponential backoff with jitter on retries, request queuing to smooth burst traffic, multiple API keys/provider fallback for redundancy, circuit breakers to fail fast during outages rather than compounding latency, and caching identical/near-identical requests to reduce call volume in the first place.
How do you deploy LLMs on Kubernetes?
Package the serving runtime (vLLM, TGI) in containers with GPU node selectors, use HPA on queue depth or GPU utilization (not raw RPS), mount model weights from object storage or pre-baked images, and put a gateway in front for auth and routing. Use readiness probes that wait until weights are loaded to avoid cold-start traffic during rollouts.