Small Language Models
Efficiency, distillation, and deployment patterns for sub-billion to few-billion parameter models.
When should you choose an SLM over a frontier LLM?
- Latency-sensitive apps (on-device assistants, real-time autocomplete).
- High volume, narrow domain tasks where a fine-tuned 1-3B model matches a 70B general model.
- Privacy/offline requirements (mobile, embedded, air-gapped).
- Cost caps where per-token API pricing does not scale.
Use frontier LLMs when you need broad reasoning, rare knowledge, or complex multi-step tasks without heavy customization.
What SLM sizes are common in production today?
Rough tiers: <1B for mobile/edge (Phi, Gemma 2B), 1-8B for server-side efficient inference (Mistral 7B, Llama 3 8B), 8-15B as a sweet spot for quality vs cost on a single GPU. Below 1B, quality drops sharply for open-ended chat but can work for classification, extraction, and routing.
How do you make an SLM perform like a larger model on a specific task?
- Distillation from a teacher LLM on task-specific outputs.
- Targeted SFT/LoRA on high-quality domain data.
- RAG to supply knowledge instead of memorizing it in weights.
- Prompting + tools so the SLM orchestrates rather than reasons from scratch.
- Model cascading: SLM handles easy queries, escalates hard ones to a larger model.
What is knowledge distillation in the SLM context?
A large teacher model generates labels or soft probability distributions on inputs; the smaller student is trained to match those outputs. This transfers reasoning patterns and style more efficiently than training only on hard labels. Common for building compact models that retain much of a 70B teacher's behavior on a narrow task.
What quantization levels work well for SLMs on consumer hardware?
4-bit (GPTQ, AWQ, GGUF Q4_K_M) often gives acceptable chat quality for 7B models on 8-16GB VRAM. 8-bit is safer for critical tasks. On CPU/mobile, GGUF with aggressive quantization (Q4, Q5) is standard. Always eval on your task after quantization; math and code degrade more than summarization.
SLM on-device vs cloud SLM serving: tradeoffs?
On-device: zero network latency, offline, data stays local, but limited RAM, thermal throttling, and harder updates. Cloud SLM: easier updates, batching, GPU acceleration, centralized logging, but network dependency and privacy considerations. Hybrid patterns run a tiny on-device model for drafts/routing and cloud for heavy generation.
What is model routing and how does it pair SLMs with LLMs?
A classifier or heuristic scores incoming requests (complexity, domain, safety) and routes simple ones to a cheap SLM and complex ones to a frontier LLM. This can cut cost 5-10x while preserving quality on hard queries. The router itself can be a tiny model or embedding similarity to known "hard" query clusters.
How do you evaluate whether an SLM is good enough to ship?
Compare against your current LLM baseline on a task-specific eval set (not just MMLU). Measure quality, latency p50/p99, tokens/sec, memory, and cost per 1K requests. SLMs often fail on edge cases; explicitly test failure modes and define escalation to a larger model when confidence is low.