Prompt Compression

What Is Prompt Compression?

Prompt compression reduces the number of tokens sent to a large language model while preserving the information needed for a correct answer. Unlike summarization by the same LLM you are about to call, compression usually runs through a smaller model, classifier, or structure-aware algorithm before the expensive inference step.

Targets include:

• Long system prompts and few-shot examples (redundant phrasing, low-information demonstrations)
• RAG retrieved chunks (overlapping passages, boilerplate headers)
• Agent tool outputs (JSON blobs, logs, search hits, file dumps)
• Conversation history (stale turns that still fit in window but waste budget)

Compression is distinct from context window extension (e.g. sliding windows, memory stores). It makes existing context denser. It is complementary to KV-cache optimization, quantization, and smaller models.

Refer: Prompt Compression Survey · LLMLingua paper

Why It Matters

• Cost - API pricing is per token; input tokens dominate RAG and agent workloads.
• Latency - prefill time scales with prompt length; shorter prompts mean faster time-to-first-token (TTFT).
• Context limits - even 128k windows fill up in agent loops; compression buys headroom for reasoning and tools.
• Signal-to-noise - models attend better when redundant tokens are removed (related to lost-in-the-middle effects in bloated context).
• Downstream model choice - aggressive compression can let a cheaper model handle tasks that previously needed a frontier model.

How Compression Works (General)

Methods fall into several families:

Most production stacks compress before the target LLM call and treat the compressor as a separate service with its own CPU/GPU budget.

When to Use Compression

Good fits

• RAG with many retrieved chunks (10+ passages, repeated doc templates)
• Few-shot prompts with long demonstrations
• Agents that dump full JSON API responses, CI logs, or repo search results into context
• High-volume chat with long histories where older turns are low value
• Cost-sensitive batch jobs (classification, extraction) with repetitive instructions

Poor fits

• Short prompts already near minimum tokens
• Tasks requiring verbatim quotes, legal text, or exact code diff fidelity without retrieval fallback
• Highly entangled reasoning where every token carries nuance (some math proofs, subtle policy interpretation)
• When you have not validated compression on your domain (medical, financial, locale-specific jargon)

When to combine tools

Use LLMLingua for natural-language prompt bodies (instructions + demonstrations + question). Use Headroom for agent pipelines heavy on tool outputs, logs, JSON, and code. Many agent stacks run Headroom on tool results and LLMLingua on the final assembled prompt.

LLMLingua (Microsoft Research)

LLMLingua is a research-backed prompt compression framework. It works with black-box LLMs because compression uses an external small language model; only the shortened text is sent to GPT-4, Claude, etc.

LLMLingua (v1)

Coarse-to-fine pipeline on prompts structured as {Instruction, Demonstrations, Question}:

1. Budget controller - allocates compression budget across prompt parts (few-shot examples compressed more aggressively than core instructions).
2. Demonstration-level filtering - small LM perplexity scores sentences; low-surprise (predictable) content is dropped first.
3. Iterative token-level compression - remaining segments compressed token-by-token using conditional perplexity, preserving interdependent phrases better than one-shot filtering.
4. Distribution alignment - optional tuning so compressed prompts align with target LLM behavior.

Reported results: up to 20x compression with small accuracy loss on benchmarks like GSM8K, BBH, ShareGPT. Best when prompts contain redundant natural language and long ICL examples.

LongLLMLingua

Extends LLMLingua for long-context RAG. Query-aware compression reorders and prioritizes chunks so question-relevant evidence sits where the target LLM attends reliably, mitigating lost-in-the-middle. Use when retrieved context spans thousands of tokens and the user question is known at compression time.

LLMLingua-2

Task-agnostic compression formulated as token classification. A bidirectional encoder (e.g. XLM-RoBERTa-sized) trained via distillation from GPT-4 labels which tokens to keep. Typically 3-6x faster than v1 and stronger on out-of-domain text. Recommended default for new projects unless you specifically need LongLLMLingua query routing.

Key features

• Configurable compression rate (target fraction of tokens to keep)
• force_tokens to preserve punctuation, newlines, or domain delimiters
• drop_consecutive to collapse repeated tokens after compression
• Works without white-box access to the downstream LLM
• Open-source Python package: pip install llmlingua

How to use LLMLingua

pip install llmlingua

from llmlingua import PromptCompressor

# LLMLingua-2 (recommended starting point)
compressor = PromptCompressor(
    model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
    use_llmlingua2=True,
)

long_prompt = """
Instruction: Answer using the examples below.
...
[thousands of tokens of few-shot examples and context]
...
Question: What is the refund policy for enterprise tier?
"""

result = compressor.compress_prompt(
    long_prompt,
    rate=0.5,                    # keep ~50% of tokens
    force_tokens=["\n", ".", "?"],
    drop_consecutive=True,
)

compressed = result["compressed_prompt"]
print(result["origin_tokens"], "->", result["compressed_tokens"])

LongLLMLingua example (RAG)

compressor = PromptCompressor(
    model_name="NousResearch/Llama-2-7b-hf",
    use_llmlingua2=False,
)

compressed = compressor.compress_prompt(
    context=retrieved_documents,   # long multi-doc string
    question=user_query,
    rate=0.55,
    condition_in_question="after_condition",
    reorder_context="sort",
    dynamic_context_compression_ratio=0.3,
)

LLMLingua pros and cons

Headroom

Headroom is an open-source context compression layer for AI agents. It targets tool outputs, logs, RAG chunks, files, and conversation history before they reach the LLM. Typical savings: 60-95% on JSON/search results, 15-70% on code-heavy agent traces, with a focus on preserving errors, stack traces, and structurally important fields.

Refer: Headroom GitHub · Headroom docs

What makes Headroom different

• Content-aware routing - detects JSON, code, logs, search results, HTML, plain text and picks a specialized compressor.
• CCR (Compress-Cache-Retrieve) - aggressive compression with originals stored; LLM gets a headroom_retrieve tool to fetch full content on demand. Nothing is permanently destroyed.
• CacheAligner - stabilizes static prefixes so provider prompt caching (e.g. Anthropic cached reads) still hits while dynamic tail is compressed.
• Live-zone compression - compresses newest tool results and user turns without rewriting the cache-hot system prompt and tool definitions.
• Agent-first integrations - library, HTTP proxy, MCP server, one-command wraps for Claude Code, Cursor, Codex, etc.

Compression pipeline

1. CacheAligner - separates dynamic tokens (dates, session vars) from static system prompt prefix.
2. ContentRouter - routes each message block to a compressor:

IntelligentContext (optional) scores messages by recency, relevance, and error signals to drop lowest-value history when still over budget.

Integration modes

1. Python library

pip install headroom-ai

from headroom import compress

messages = [
    {"role": "system", "content": "You are a debugging assistant."},
    {"role": "user", "content": "Why did deploy fail?"},
    {"role": "tool", "content": huge_ci_log_string},
]

optimized, stats = compress(messages)
print(stats.tokens_before, stats.tokens_after, stats.compression_ratio)

2. Transparent proxy

pip install "headroom-ai[proxy]"
headroom proxy --port 8787 --mode token

# Point your OpenAI/Anthropic client base URL to localhost:8787
# Requests are compressed automatically before forwarding

--mode token maximizes compression. --mode cache freezes prior turns to preserve provider prefix-cache bytes on long conversations.

3. Compression-only HTTP API

# POST http://localhost:8787/v1/compress
# Body: {"messages": [...]}
# Returns tokens_before, tokens_after, compressed messages

4. MCP server

Tools: headroom_compress, headroom_retrieve, headroom_stats for any MCP-compatible agent client.

5. Agent wrap

headroom wrap claude   # one-command wrapper for supported agents
headroom unwrap claude

Headroom + RAG example

from headroom import compress

rag_chunks = load_retrieved_chunks()  # 40k tokens of HTML + metadata
tool_message = format_chunks_as_tool_output(rag_chunks)

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": user_question},
    {"role": "tool", "content": tool_message},
]

compressed_messages, stats = compress(
    messages,
    enable_ccr=True,   # store originals for headroom_retrieve
)

response = llm.invoke(compressed_messages)
# Model can call headroom_retrieve if it needs a full passage

Headroom pros and cons

LLMLingua vs Headroom

They are complementary, not mutually exclusive.

Production Patterns

RAG Q&A API

1. Retrieve top-k chunks (hybrid search + rerank).
2. Run LongLLMLingua or LLMLingua-2 on assembled context with the user question.
3. Optionally run Headroom if chunks include HTML tables, JSON metadata, or logs.
4. Call target LLM with compressed context; log compression ratio and faithfulness scores.

Coding agent

1. Wrap agent with Headroom proxy or compress() on each tool result.
2. Enable CCR so the model can retrieve full files when diffs reference missing lines.
3. Use --mode cache on long sessions to protect prefix-cache discounts.

High-volume extraction

1. Fixed instruction prefix (cached at provider).
2. Compress variable few-shot block with LLMLingua-2 at rate tuned on eval set.
3. Batch requests; track $/successful extraction.

Evaluation Checklist

• Build a golden set (50-200 examples) with task-specific success criteria.
• Sweep compression rates (0.3, 0.5, 0.7) and measure accuracy, not just tokens.
• Track latency end-to-end: compression + LLM; compression can add 50-500ms.
• Log failure modes: dropped numbers, wrong entity, missed tool argument.
• A/B in shadow mode before enabling compression for all traffic.
• For Headroom CCR, test that retrieve tool is invoked when answers need verbatim detail.

Summary: What, Why, How, When