Every token in your LLM prompt costs money. Most of them don't need to be there. Prompt compression removes redundant tokens from inputs before they reach the model. You cut input token costs without switching models or touching your application logic.

The technique works because natural language is deeply redundant. A RAG retrieval that returns five 500-word passages to answer a simple question is sending thousands of tokens the model doesn't need. A system prompt written conversationally contains filler, repetition, and padding that carries no information. Prompt compression identifies which tokens are doing work and drops the rest. It is one of the five core techniques in our guide to LLM cost optimization.

Loupe by LeanLM measures what AI workloads cost in energy per request. This post covers prompt compression techniques for production workloads.

Research from Microsoft shows compression ratios of up to 20× with under 2% quality degradation on benchmarks including GSM8K, BBH, and ShareGPT (Jiang et al., LLMLingua, 2023). At 5× compression on a workload spending $20,000/month on input tokens, that's $16,000 saved monthly with no model changes.

Definition

Prompt compression is the removal of redundant, low-information tokens from LLM inputs before inference. A compression model scores each token's informativeness and drops those below a threshold. The tokens that carry essential meaning stay, and the token count falls. The compressed prompt produces equivalent output from the target LLM at a fraction of the input cost.

Prompt compression before-and-after diagram: an original prompt of 847 tokens costing $0.013 per call is reduced by LLMLingua to 94 tokens costing $0.0014 per call, a 9× compression with under 2% quality degradation
LLMLingua cuts this prompt from 847 to 94 tokens. That's a 9× reduction, and input cost drops by the same factor with under 2% quality loss.

Why Prompts Are Redundant

Most production LLM inputs are longer than they need to be. This isn't a design flaw. It comes with how LLM applications get built:

RAG retrieval over-fetches context. Retrieval systems return top-k passages to cover the answer. Retrieved passages are rarely 100% relevant to the query because they're ranked by semantic similarity, with no regard for information density. A typical RAG retrieval for a focused question returns 2,000–5,000 tokens of context where 400–800 tokens contain the actual answer.

System prompts are written like documentation. Engineers write system prompts for clarity and correctness, and token efficiency comes last. Explanations, caveats, examples, and formatting guidance all help prompt engineering, and all run longer than the model needs.

Few-shot examples are padded. Few-shot examples are chosen to be representative and readable. Representative examples include common phrasing patterns, which means they contain natural language redundancy by design.

Conversation history accumulates. Multi-turn applications send the entire conversation history with each request. Early turns are often less relevant to the current question than recent ones, but they get included anyway.

20×
maximum compression ratio demonstrated by LLMLingua-2 on RAG workloads with under 2% quality degradation

How Prompt Compression Works

At its core, prompt compression inserts an intelligent filtering stage between your application and the target LLM API. Instead of sending raw, bloated prompts directly to expensive frontier models, a lightweight compression engine evaluates which tokens or passages contribute genuine semantic information to the target task, prunes the non-essential content, and forwards a compact, high-density prompt.

Because prompt pricing is strictly linear per token ($/MTok), cutting token volume by 4× reduces your input bill by exactly 75% on every call. Furthermore, because transformer attention is O(N²) (or O(N) with chunked prefill), reducing prompt length dramatically cuts initial prefill latency and time-to-first-token (TTFT).

The 4 Modern Prompt Compression Techniques (Compared)

Modern prompt compression has evolved far beyond rudimentary stopword stripping. Today, production architectures fall into four distinct technical paradigms, each operating at a different granularity and trade-off point:

Token-level

1. Perplexity & Entropy Pruning (LLMLingua, LLMLingua-2)

Uses an auxiliary language model to score the conditional informativeness of each token. The original LLMLingua (Microsoft, 2023) used a causal model (Llama-2-7B or Falcon-7B) to compute perplexity: tokens with low perplexity (predictable syntax, repetitive articles) are dropped, while high-perplexity tokens (dense factual terms) are retained. LLMLingua-2 (Microsoft, 2024) improved on this by reformulating compression as an extractive token classification task using a bidirectional encoder (XLM-RoBERTa), making compression 3–6× faster with less task degradation.

LLMLingua-2 (Pan et al., 2024): up to 20× compression with under 2% quality loss; sub-30ms latency on GPU. arxiv:2403.12968 ↗

Attention-guided

2. Attention-Guided Saliency Pruning

Calculates cross-attention and self-attention weights between query tokens and retrieved context tokens inside an embedding model or intermediate transformer layer. Tokens that fail to receive attention from the user's question are flagged as uninformative and pruned before final generation. This guarantees that preserved tokens are directly linked to the user's specific intent, resolving the needle-in-a-haystack problem in long-context documents.

LongLLMLingua (Jiang et al., 2023): question-aware context pruning boosts retrieval QA accuracy by 21.4% while cutting tokens 4×. arxiv:2310.06839 ↗

Chunk-level

3. Semantic Chunk & Sentence Filtering (Selective Context)

Rather than pruning isolated tokens (which can occasionally create disfluent sentence fragments), semantic chunk filtering evaluates entire sentences or paragraphs. Using informative content measures (such as sentence-level perplexity or cross-encoder similarity), it discards whole passages that do not shift the predictive distribution. Because complete grammatical sentences are preserved, this technique prevents syntactic degradation in brittle downstream tasks.

Selective Context (Li et al., 2023): achieves up to 50% token reduction with zero loss of syntactic coherence. arxiv:2310.06201 ↗

Representation

4. Learned Soft Prompts & Gist Tokens

Compresses arbitrary natural language instructions into a small set of continuous virtual token embeddings (“gist tokens”). Instead of passing a 1,000-token system prompt, the prompt is encoded into 5–10 virtual token representations that condition the model via prefix tuning. While offering extreme compression ratios (20×–50×), this technique requires white-box model weights and prefix-tuning training, making it suitable for self-hosted models (vLLM, HuggingFace) rather than black-box API endpoints.

Gist Tokens (Mu et al., Stanford, 2023): compresses prompt instructions up to 26× with minimal downstream performance drop. arxiv:2304.08467 ↗

Technique Primary Mechanism Typical Ratio Compressor Latency Compressor VRAM Quality Retention Best Workload
Token Entropy Pruning (LLMLingua-2) Extractive token classification or conditional perplexity scoring via encoder 2×–5× (up to 20×) 20–40ms (GPU)
80–140ms (CPU)
~1.2 GB (XLM-RoBERTa) >98% on QA & summarization; degraded on code/math RAG context, conversational history, document QA
Attention-Guided Saliency Pruning Self/cross-attention weight ranking between query and context tokens 2×–4× 50–120ms (GPU) 4–8 GB (requires attention weights) High on needle-in-haystack QA; sensitive to long contexts Multi-turn chat, multi-document retrieval
Semantic Chunk & Sentence Filtering Sentence-level perplexity or cross-encoder passage re-ranking 2×–6× 5–25ms (GPU / CPU) <500 MB (bi-encoder or BM25) 100% syntactic coherence; preserves full grammatical sentences Long enterprise docs, compliance, structured policies
Learned Soft Prompts & Gist Tokens Compresses prompts into learned continuous virtual vector embeddings 10×–50× <10ms (cached prefix forward pass) Native model VRAM Very high on fixed tasks; zero generalization to unseen prompts White-box self-hosted models, high-volume fixed system prompts

Empirical comparison across LLMLingua-2 (Pan et al., 2024), Selective Context (Li et al., 2023), and Gist Tokens (Mu et al., 2023). Latency reflects batch size 1 on an NVIDIA A10G or 8-core modern x86 CPU via ONNX Runtime.

What Compresses Well vs. What Doesn't

Prompt compression is not universally applicable. The savings depend heavily on your prompt structure and task type. Understanding where it works prevents misapplication.

High-compression candidates (5–20× possible):

Low-compression candidates (2× or less, proceed carefully):

The safest starting point is RAG context. Retrievals are reliably redundant relative to the query, compression ratios are highest, and the quality impact is lowest because you're removing context the model wasn't using anyway.

Prompt Compression vs. Prompt Caching: Stacking for 95%+ Savings

Engineers often ask whether prompt compression is obsolete now that frontier providers (OpenAI, Anthropic, Google) offer native prompt caching. The answer is that they address fundamentally different bottlenecks—and when combined, their cost savings multiply.

Prompt caching stores the pre-computed Key-Value (KV) activations of an identical prefix on the model provider's GPU cluster. When subsequent requests share that exact prefix, the provider skips the prefill compute phase and discounts input token prices by 80% to 90% (e.g., Anthropic Claude prompt cache hits receive a 90% discount, paying $0.30/MTok instead of $3.00/MTok on Sonnet). However, the prompt itself remains full size. If your prompt contains 8,000 tokens of noisy RAG context, you still transmit 8,000 tokens over the wire, consume 8,000 tokens of precious KV cache capacity on the server, and pay full price on cache misses.

Prompt compression physically prunes tokens from the payload before the request is transmitted. It reduces bandwidth, eliminates irrelevant distractor context, and shrinks the memory footprint of the request.

Because prompt caching discounts the rate per token while prompt compression reduces the number of tokens, the two techniques stack multiplicatively:

The Multiplicative Savings Equation

Consider an enterprise agent handling 100,000 requests/day with a 2,000-token system prompt and 8,000 tokens of retrieved documentation (10,000 total input tokens per call) using a frontier model priced at $3.00 / MTok input:

  • Baseline (Uncompressed, Uncached): 10,000 tokens × $3.00/MTok = $0.0300 per call ($3,000/day).
  • Prompt Caching Alone (90% hit discount on 8k static tokens): 2,000 dynamic tokens @ $3.00/MTok ($0.0060) + 8,000 cached tokens @ $0.30/MTok ($0.0024) = $0.0084 per call (72% total savings).
  • Prompt Compression Alone (4× compression to 2,500 total tokens): 2,500 tokens × $3.00/MTok = $0.0075 per call (75% total savings).
  • Stacked Compression + Caching: The 2,000-token system prompt is compressed to 500 tokens and cached; retrieved context is compressed from 8,000 to 2,000 tokens. The combined request is 2,500 tokens. On cache hits, effective input cost drops to $0.00075 per call—a 97.5% net cost reduction ($75/day vs. $3,000/day baseline). You can calculate your exact token mix using our LLM cost calculator.

Furthermore, compressed prompts occupy 75% less KV cache VRAM on the inference cluster (whether in commercial APIs or on self-hosted vLLM engines). Smaller KV cache footprints directly increase serving throughput, reduce cache eviction rates under heavy multi-tenant concurrency, and decrease the probability of cache thrashing.

Production Latency & Hardware: When Does Compression Pay Off?

Adding a prompt compression step introduces an additional model call into your request pipeline. Whether this improves or degrades end-to-end user latency depends on the Latency Budget Equation:

Net Latency = Compressor Inference Latency + Network Transfer Time + LLM Prefill Latency + LLM Decode Latency

In standard LLM inference, time to first token (TTFT) is dominated by network transfer time and the target LLM's prefill phase. On frontier models (e.g., GPT-5.4, Claude 3.7 Sonnet, Gemini 2.5 Pro), prefilling 15,000 tokens can take 400ms to 1,200ms depending on API load. If prompt compression cuts that context to 3,000 tokens, the LLM prefill phase drops to 100–250ms, saving 300–950ms of server-side computation.

The decisive factor is the compressor model's execution time:

Generation-based compressors (LLMLingua-1) used causal decoder LLMs (such as Llama-2-7B or Falcon-7B) to calculate conditional perplexity autoregressively. This added 200–500ms of GPU latency and required 14GB+ of GPU VRAM, making it impractical for interactive chat or low-latency SLAs.

Extractive classification compressors (LLMLingua-2) replaced causal perplexity with a lightweight bidirectional encoder (such as XLM-RoBERTa-large, 560M parameters). Instead of predicting the next token, the encoder performs single-pass binary classification (keep vs. drop) across the entire sequence simultaneously. On an NVIDIA A10G or L4 GPU, LLMLingua-2 processes 4,000 tokens in 22ms to 38ms. Even on an 8-core CPU instance using ONNX Runtime with INT8 quantization, compression takes under 110ms.

Net Negative TTFT: When processing prompts larger than 4,000 tokens with an extractive encoder compressor, the 25ms compressor overhead is smaller than the 200ms+ saved in LLM prefill and WAN transfer time. The user experiences a faster initial response while paying 60–80% less for the inference call.

Production Guardrails & Failure Modes

While prompt compression is highly effective for prose, deploying it blindly in production introduces specific failure modes:

Research Foundations

LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
Jiang et al., Microsoft, 2023, arxiv:2310.05736
20×
max compression
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
Pan et al., Microsoft, 2024, arxiv:2403.12968
<2%
quality drop
Selective Context: Filtering Irrelevant Context for Long-Document LLMs
Li et al., 2023, arxiv:2310.06201
50%
token reduction
Learning to Compress Prompts with Gist Tokens
Mu et al., Stanford & OpenAI, 2023, arxiv:2304.08467
26×
token reduction

Deploying Prompt Compression in Production

Adding prompt compression to an existing LLM application is an infrastructure change, not an application change. The compressor sits between your application and the LLM API. It receives the full prompt and returns a compressed version before forwarding.

  1. Identify your compression targets: instrument your LLM calls to measure average prompt length by component: system prompt, retrieved context, conversation history, user query. The longest components with the most structural redundancy are your first targets.
  2. Set compression budget: start conservative (2–3× compression ratio on context). Run compressed vs. uncompressed on a sample of your production queries and measure quality on your actual output metrics, not just benchmark proxies.
  3. Compress selectively: compress RAG context and conversation history aggressively, compress system prompts conservatively, and leave the user's actual query alone.
  4. Monitor quality per task type: sensitivity to compression varies by task. Track quality metrics (downstream task accuracy, user ratings, etc.) per task type as well as in aggregate.
  5. Combine with caching: compress the system prompt once, then cache the compressed version. Compress RAG context per query. You get compression savings on every call and caching savings on repeated prefixes.
  6. Measure the caching side too: prefix caching in vLLM and Ollama, measured cut time to first token from 7.8s to 0.15s on a 32B model with a 32,000-token shared prompt.

Once compression is in place, layer in LLM model routing to send the compressed inputs to the cheapest capable model. For high-volume task types, consider LoRA fine-tuning where a smaller adapted model can replace the API call entirely.

"Prompt compression is the only LLM cost optimization that requires no model changes, no routing infrastructure, and no quality tradeoff calibration. You just make the input smaller. The savings are immediate and the risk is low."

Running coding agents on multi-file repositories? Sign up for a $10 EcoCoder credit or inspect our SlopCodeBench brownfield results.

Frequently Asked Questions

What is prompt compression?

Prompt compression removes redundant, low-information tokens from LLM inputs before sending them to the model. By stripping filler words, repeated context, and low-importance tokens, it reduces the token count of inputs by 2–20× while preserving the information the model needs to produce a correct response.

How does LLMLingua work?

LLMLingua uses a small language model (typically Llama or Falcon) to score the perplexity of each token in the prompt. Low-perplexity tokens (the predictable, redundant ones) are dropped. High-perplexity tokens, which carry essential information, are preserved. The result is a compressed prompt that keeps the key information at a fraction of the token count.

Does prompt compression work with all LLMs?

Yes. Prompt compression is model-agnostic. It runs before the prompt reaches the target LLM, so it works with GPT-5.5, Claude, Gemini, or any other model. The compressor is a separate small model used only for token scoring. It is not the model handling inference.

What is the difference between prompt compression and prompt caching?

Prompt caching (available in Claude, GPT-5.x, and Gemini) stores a processed version of a repeated prefix to avoid re-processing it on each call. Prompt compression reduces the token count of the prompt itself before it reaches the model. They are complementary: compress first, then cache the compressed prefix.

What are the best prompt compression techniques?

The four primary prompt compression techniques are: 1) Token-level perplexity and entropy pruning (such as LLMLingua and LLMLingua-2), which scores and drops low-information tokens using a small encoder or causal model; 2) Attention-guided saliency pruning, which removes tokens that receive minimal cross-attention weight from the query; 3) Semantic chunk and sentence filtering (such as Selective Context), which drops entire irrelevant paragraphs or sentences to preserve grammatical integrity; and 4) Learned soft prompts or gist tokens, which compress hundreds of prompt tokens into a small set of learned continuous virtual vectors.

Does prompt compression increase latency?

Prompt compression adds a small compression stage (typically 20–40ms on GPU or 80–120ms on CPU using lightweight models like LLMLingua-2 XLM-RoBERTa). However, by reducing input tokens by 2–5×, it significantly cuts both WAN transmission payload size and server-side LLM prefill latency (which can take 200–800ms on large models). On prompts longer than 3,000–5,000 tokens, prompt compression typically yields a net reduction in overall time-to-first-token (TTFT).

How much cost does prompt compression save?

Prompt compression saves on input token costs directly, which can be significant for workloads with long system prompts, RAG context, or few-shot examples. LLMLingua achieves 2–20× compression with under 2% quality degradation. At 4× average compression on input tokens, a workload spending $10,000/month on inputs would save $7,500/month with no model switching required.

What types of prompts compress well?

Prompts that compress well include: RAG-retrieved context (often 60–80% redundant relative to the query), long system prompts with repetitive instructions, few-shot examples with verbose formatting, and document summarization inputs. Prompts that compress poorly include: short queries (nothing to remove), code with precise syntax, and mathematical problems where every token matters.