Deployment

Deployment Topology

Production LLM systems typically split into layers:

• Client / API gateway - auth, rate limits, request validation
• Application orchestration - RAG, tools, agents, prompt assembly
• Model serving - GPU workers running vLLM, TGI, or TRT-LLM
• Supporting services - vector DB, feature store, logging, eval pipelines

Keeping orchestration stateless lets you scale app servers independently from GPU pools sized to token demand.

Refer: vLLM docs · TGI docs

Serving Runtimes

GPU server frameworks

vLLM and TensorRT-LLM target high-throughput datacenter GPUs with continuous batching and optimized kernels. Text Generation Inference (TGI) offers Hugging Face ecosystem integration. Choose based on model support, quantization formats, and SLA requirements.

# Kubernetes deployment snippet — GPU node pool for inference
resources:
  limits:
    nvidia.com/gpu: 1
  requests:
    cpu: "4"
    memory: "16Gi"
readinessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 120  # model load time

CPU and edge

llama.cpp and ONNX Runtime serve quantized small models on CPU, Apple Silicon, or embedded devices where GPUs are unavailable.

Managed APIs

OpenAI, Anthropic, Bedrock, and Azure OpenAI outsource serving entirely. You integrate HTTP APIs and manage quotas, fallbacks, and data policies.

Scaling Patterns

• Horizontal replica scaling - more GPU pods behind a load balancer
• Model parallelism - tensor parallel groups for models too large for one GPU
• Multi-model routing - separate pools for 7B vs 70B workloads
• Autoscaling on queue depth - scale on tokens/sec or pending requests, not CPU
• Geographic distribution - place inference near users and data residency zones

API Gateways and Compatibility

Gateways unify access across multiple backends: OpenAI-compatible proxies (LiteLLM, Portkey), canary routing between model versions, API key management per tenant, and centralized budget caps. OpenAI-compatible APIs reduce vendor lock-in by letting clients swap base URLs.

Implement idempotency keys for paid operations, request IDs propagated through logs, and structured error codes for rate limits vs model errors vs timeouts.

Reliability and SLOs

Define SLOs for availability, p95 latency, and error rate. Patterns include:

• Timeouts with partial streaming responses where UX allows
• Fallback to smaller or cached models on primary failure
• Circuit breakers when GPU pools overheat or OOM
• Graceful degradation (retrieve-only mode without generation)

GPU OOM from oversized contexts is operational reality. Enforce max input tokens at the gateway and return clear errors before saturating workers.

Security and Compliance

Deploy inside VPCs for sensitive data, encrypt traffic with TLS, rotate API keys, and audit which models process which data classes. For regulated industries, log retention policies must balance debuggability with privacy (PII redaction in logs).

Supply chain: verify model weight checksums, scan containers, and pin dependency versions on serving images.

Release Management

Ship model versions like microservices: shadow traffic, A/B tests, instant rollback to prior checkpoint, and compatibility tests against prompt templates and tool schemas. Document changelog per model (context length, tokenizer, fine-tune data cutoff) for support teams.