Design Questions Up Front
Before drawing boxes, clarify constraints: latency target, data residency, budget per request, offline vs online, human-in-the-loop requirements, and acceptable error modes. A copilot for developers tolerates different failure semantics than automated loan decisioning.
Choose build vs buy early: managed LLM APIs accelerate MVP; owned inference wins at scale, privacy, or deep customization.
Reference Architecture
A typical enterprise LLM product includes:
- Edge: CDN, WAF, API gateway with auth and rate limits
- App tier: stateless orchestrators assembling prompts, calling retrieval and tools
- Model tier: GPU inference pools or external LLM APIs with fallback routes
- Data tier: vector index, document store, OLTP for user state, object storage for raw files
- Observability: tracing, eval jobs, feedback ingestion, cost dashboards
Async job queues handle heavy work (ingestion, summarization batches) off the interactive path.
RAG-Centric Q&A System
User query flows through authentication, query rewriting, hybrid retrieval with ACL filters, reranking, context packing, LLM generation with citations, output moderation, and response logging. Ingestion runs separately: parse, chunk, embed, index on document change events.
Refer: Azure OpenAI chat baseline architecture
Scale retrieval and generation independently. A spike in users may require more app replicas; a spike in context length requires more GPU memory, not necessarily more CPUs.
Agentic Workflow System
Agents add a state machine or graph runtime, tool registry with permissions, memory store, and iteration caps. Design explicit states: plan, execute tool, wait for human approval, complete, fail. Persist checkpoint state so long tasks survive process restarts.
Refer: LangGraph state machines
Sandbox tool execution (containers, ephemeral VMs) for untrusted code. Separate read tools from write tools at the network policy layer.
Multi-Model Routing
Classifier or rules route requests to small, large, or specialist models. Cache frequent answers. Escalate when confidence is low or user tier demands premium quality. Maintain consistent tool and JSON schemas across models so routing is transparent to downstream code.
Canary new models on a traffic slice with automatic rollback when eval metrics or error rates breach thresholds.
Reliability Patterns
- Timeouts at every external call with partial results where UX allows
- Idempotency keys for billable or side-effecting operations
- Bulkheads isolating ingestion from query serving
- Feature flags for prompts, models, and retrieval sources
- Degraded mode: search-only without generation during outages
Capacity plan on tokens per second and KV cache footprint, not traditional QPS alone.
Security and Compliance
Map data flows for SOC2, HIPAA, or GDPR as applicable. Encrypt at rest and in transit, minimize prompt retention, support regional deployment, and audit admin access to indexes containing sensitive documents. Threat model includes prompt injection via retrieved content and exfiltration via agent tools.
Evolution Over Time
Ship an MVP with manual eval and a single model. Add retrieval when hallucination blocks adoption. Add routing and caching when cost bites. Add agents only when workflows truly need multi-step tool use. Each layer increases ops burden; justify it with measured user value, not architecture fashion.