Self-Hosting an LLM: Is It Actually Cheaper Than the API?

Self-hosting an open LLM can undercut frontier API list prices, but only when the GPU stays busy, and rarely against hosted open-weight APIs serving the same model. At high-concurrency batching (October 2026 rates), a single H100 serving gpt-oss-120b gets raw GPU cost down to roughly $0.10 per million tokens, but hosted endpoints serve the identical model for $0.60 per million output tokens (and as little as $0.17), so the margin at perfect utilization is a few ×. The hourly-rate arithmetic suggests 100×. Run a DeepSeek-class 671B MoE yourself and hosted pricing beats your raw GPU cost at every utilization level. At 10% utilization every self-hosted number is 10× worse, and the all-in cost with engineering runs 3–5× the raw GPU bill. The break-even: self-hosting beats frontier API pricing at sustained volume, but it beats a hosted open-weight API running the same model only near saturation. Published break-even estimates disagree by about 50× because they hide these assumptions.

So "is self-hosting cheaper?" is really a utilization question wearing a hardware costume. This guide walks the actual 2026 cost math, the GPU and hardware tiers, the self-host-vs-API break-even, and when it's worth it. It's the build-vs-buy companion to per-call LLM cost optimization. Before you optimize an API bill, check whether you should be paying it at all.

Loupe by LeanLM (not affiliated with Google's LearnLM educational AI) measures what AI workloads cost in energy per request. Note: "LLM" here means Large Language Model, not the Master of Laws degree. The two share an acronym, and the degree pollutes a lot of cost searches.

The Real Cost Math: Why Utilization Is Everything

Start with the optimistic number, because it's the one that gets quoted. Take an H100 at $2.89/hr on-demand (RunPod Secure Cloud H100 PCIe, unchanged as of September 27, 2026), serving gpt-oss-120b (the current single-GPU workhorse, a 120B MoE that fits one 80GB card at its native MXFP4 precision) at a benchmarked aggregate throughput of ~8,000 tokens/second per GPU on a realistic mixed workload (GPUStack, vLLM). That's about 28.8 million tokens per hour, so the raw GPU cost is roughly $0.10 per million tokens at full saturation. Next to GPT-5.5 at $30 per million output tokens, that looks like a 300× saving. That single comparison is how most "just self-host it" arguments get built. The comparison that matters: Groq, Fireworks, and Together all serve the identical gpt-oss-120b for $0.60 per million output tokens, and DeepInfra's base endpoint lists $0.17, so at perfect utilization your raw margin over just calling the hosted model is about 6×, before you've paid anyone to run it. (You'll also see wilder self-hosting multiples in the wild; they usually come from dividing an hourly rate by a single-stream token count, or a straight arithmetic slip.)

And three assumptions behind even that number almost never hold. First, saturation: a GPU you rent by the hour bills the same whether it's serving 8,000 tokens/second or zero, so at 10% utilization the effective cost per token is 10× higher. Second, batching: aggregate throughputs come from serving many requests concurrently. NVIDIA's benchmark for a dense 70B (Llama 3.3, 2× H100 FP8) runs 51.7 tokens/second at concurrency 1 but 920 at concurrency 250, an ~18× spread, so break-even math built on single-stream numbers is wrong before it starts. Third, ops: one mid-level MLOps engineer typically covers only 4–6 GPUs, and LeanLM's working multiplier (consistent with third-party total-cost-of-ownership analyses) puts the all-in cost at 3–5× the raw GPU rental. A peer-reviewed 2026 costing study makes the utilization point concrete: the same Mixtral 8x7B deployment on an H100 costs $15.25 per million output tokens at 1 request/second but $0.87 at 25 requests/second. That's a 17× swing from traffic shape alone.

Effective raw GPU cost per million tokens, by utilization (October 2026)

Raw GPU rental only. Multiply by the 3–5× ops factor for all-in cost. Derived from RunPod on-demand rates (H100 PCIe Secure Cloud $2.89/hr, H200 Community Cloud $3.59/hr, both unchanged as of September 27, 2026) and DeepInfra list prices (pulled July 22, 2026), GPUStack's vLLM benchmark for gpt-oss-120b on H100, and the vLLM project's production figure of 2,200 tok/s per H200 for 671B-class DeepSeek (8-GPU minimum; per-GPU shown). Throughputs are aggregate served tokens on batched workloads. Read the last column carefully: for the 671B MoE, the hosted API beats your raw GPU cost at every utilization level, and even for gpt-oss-120b, all-in cost with ops beats the $0.60 hosted price only above roughly 50–80% sustained utilization.

This is the same lesson as long-context cost management, one level up: the cost you don't see (idle GPU-hours, re-sent context) dwarfs the cost you do. Keeping the GPU you already have busy beats shopping for a cheaper one.

What GPUs actually cost (October 2026)

As of July 22, 2026, except the H100, H200, and A100 on-demand cells, which are as of October 27, 2026: RunPod Secure Cloud rates from runpod.io/pricing, Community Cloud rates from RunPod's public GPU-type API, and Lambda rates from Lambda's on-demand price list (per GPU-hour; Lambda's H100 SXM rate runs from $3.99 on an 8-GPU instance to $4.29 on a single GPU, and its only A100 80GB option is the 8-GPU SXM instance). spot/marketplace prices (Vast.ai, Spheron) move minute-to-minute, so treat them as a floor, not a plannable rate. AWS sells H100/A100 only in 8-GPU instances, so the per-GPU figure is arithmetic, not a rentable unit. Prices have been falling through 2025–2026, so re-check before budgeting.

GPU and Hardware Requirements: What You Actually Need

Which GPU you need is set by one thing first: whether the model's weights plus its KV cache fit in VRAM (see how much VRAM a local LLM needs for measured numbers by size, quant and context). And in 2026 that calculation has a twist, because the workhorse open models are now mixture-of-experts (MoE): you pay VRAM for the model's total parameters but earn throughput on its much smaller active parameters. Qwen3.6-35B-A3B holds 35B parameters in memory and computes with 3B per token; DeepSeek's 671B-class models compute with ~37B. That asymmetry (the memory cost of a big model with the compute speed of a small one) is why hosted providers with pooled traffic serve these models so cheaply, and why your half-idle GPU can't match them. A sizing guide for what people actually self-host in mid-2026:

Weights only; add KV-cache headroom, which grows with context length and concurrency. The KV cache is what pushes a dense 70B off a single 80GB card. Cluster-scale models (DeepSeek-V4-Pro, 1.6T params) are omitted: if it takes 8× B200 to hold, "self-host to save money" is not the question. As of October 2026.

Throughput then determines how many requests that GPU can actually serve, and the batched-vs-single-stream gap is where most break-even math dies. On a single H100 with vLLM, gpt-oss-120b delivers roughly 8,000 aggregate tokens/second on a realistic mixed workload (GPUStack's published benchmark), while an interactive single user sees more like 500–600 tokens/second, a 13–16× spread between the number your capacity plan needs and the number a demo shows you. For the big MoEs, the vLLM project's production benchmark sustains 2,200 tokens/second per H200 serving 671B-class DeepSeek across an 8-GPU group. Whichever model you run: aggregate benchmarked throughput divided into your real demand is your utilization. Utilization is the whole game.

Quantization is the second lever, and in 2026 it's table stakes: gpt-oss ships MXFP4 natively, and serving a model in FP8 instead of BF16 measures out to roughly 24% lower cost per million tokens (Baseten, TensorRT-LLM on H100) while retaining ~99.5% of benchmark accuracy (Red Hat's evaluation across Llama and Mixtral families), and the memory savings often enable larger batches on top. One hardware caveat: H100s run MXFP4 through software emulation; Blackwell-generation cards run it natively, which is why one B200 roughly matches two H100s on gpt-oss-120b. For a consumer-card data point at a smaller size, see our measured Qwen 3.8 27B speed and VRAM on one 32GB AMD R9700.

Inference Software Stack: What Software Reduces Cost per Token?

The hardware you rent sets your maximum cost ceiling, but your inference serving software dictates your effective cost floor. An unoptimized serving setup running vanilla PyTorch or naive Hugging Face pipelines suffers from severe KV-cache fragmentation (frequently stranding 60–80% of VRAM), redundant prefill compute on static prompts, and serial request stalls. In 2026, four distinct software runtime optimizations determine whether your self-hosted setup achieves its theoretical economics or bleeds margin:

1. PagedAttention & Chunked Prefill (vLLM v0.31+)

In traditional serving architectures, memory for the key-value (KV) cache is allocated contiguously based on the model's maximum sequence length (such as 32k or 128k tokens). Because most user requests generate far fewer tokens, up to 70% of GPU memory is wasted in internal and external fragmentation. PagedAttention solves this by partitioning the KV cache into non-contiguous physical memory blocks, mirroring virtual memory paging in operating systems. Fragmentation plummets below 4%, allowing a single H100 to support 2–4× larger concurrent batch sizes. Paired with chunked prefill (interleaving compute-heavy prompt evaluations with memory-bandwidth-bound token generations), vLLM prevents long prompt prefills from starving ongoing token generation streams, stabilizing tail latencies and eliminating queue dropouts.

2. Automatic Prefix Caching (APC)

In real-world applications—especially system prompts, multi-step agent tool definitions, RAG context injection, and few-shot classification—70–90% of prompt tokens are identical across sequential requests. Standard inference engines recompute the KV cache for the full prompt on every single API call. Automatic Prefix Caching maintains an in-memory hash tree of evaluated tokens. When incoming requests share an identical prefix, the runtime reuses the cached KV tensors directly from GPU VRAM, bypassing the prefill compute phase entirely. On production workloads with heavy system prompts, prefix caching cuts prefill compute and time-to-first-token (TTFT) by 70–90%, effectively multiplying the request throughput of an existing GPU cluster without purchasing additional hardware.

3. Tree-Structured KV Caching (SGLang & RadixAttention)

While linear prefix caching works well for simple prompts, complex workflows (such as multi-turn conversational agents, chain-of-thought branching, and structured JSON extraction) produce non-linear, branching KV dependencies. SGLang organizes the KV cache into a radix tree. When an agent explores multiple parallel reasoning paths, retries failed tool calls, or conducts multi-turn user dialogues, RadixAttention retains intermediate KV states across turns without eviction or recomputation. Combined with compressed finite-state machine (FSM) grammar execution for guaranteed JSON schemas, SGLang achieves 3–5× higher throughput on agentic and structured extraction workloads compared to baseline serving engines.

4. Speculative Decoding & Kernel-Level Tuning (TensorRT-LLM & Marlin)

Because autoregressive decoding is memory-bandwidth-bound (transferring billions of parameters just to generate one token), raw compute cores remain largely underutilized during output generation. Speculative decoding addresses this bottleneck by using a small, high-speed draft model (such as a 1B–2B parameter SLM or speculative draft heads) to generate multiple candidate tokens in parallel, which the primary target model then verifies in a single forward pass. On NVIDIA Hopper and Blackwell cards, TensorRT-LLM combines hardware-fused FP8 GEMM kernels with native speculative decoding to deliver 1.8–2.4× higher output token throughput per GPU-watt. Paired with Marlin kernels for ultra-fast mixed-precision INT4/FP8 dequantization, memory bandwidth bottlenecks are minimized, unlocking maximum saturation on dense models.

Feature availability and performance characteristics as of October 2026. For high-volume multi-tenant serving, choosing between vLLM, SGLang, and TensorRT-LLM depends primarily on whether your workload is linear multi-tenant traffic (vLLM), branching agentic/JSON workflows (SGLang), or dedicated Hopper/Blackwell hardware saturation (TensorRT-LLM).

Self-Hosted vs API: The Honest Break-Even

To compare fairly you need the other side of the ledger: current API list prices. As of 2026, per million tokens (input / output):

List prices as of October 27, 2026 (GPT-6 at its short-context tier, up to 272K input tokens); confirm against each provider and note the cheap tier is what self-hosting actually competes with. See our LLM effective cost table for cache- and batch-adjusted real prices, or run your own token mix through the cost calculator to get the API side of your break-even in dollars. Watch the Haiku ambiguity: Haiku 4.5 is $1/$5; the older Haiku 3 was $0.25/$1.25.

But the closed-model table above is only half the yardstick, and it's the flattering half. The comparison self-hosting actually has to win is against hosted open-weight APIs serving the exact model you'd deploy:

Pulled from provider docs and DeepInfra's live model API on July 22, 2026. The gpt-oss-120b price is unusually solid: Groq, Fireworks, and Together publish the identical $0.15/$0.60. These prices are the benchmark: a provider serving the same open model at pooled, near-saturated utilization.

The three break-evens (there is no single number)

Put the two tables together with the utilization math above and "when does self-hosting win?" splits into three different questions with three different answers:

• Vs a frontier closed API ($10–50 per million output): self-hosting an open model wins early: even modest utilization of a $0.10-per-million-at-saturation rig undercuts frontier list prices, with the 3–5× ops multiplier included. But this comparison smuggles in the hard assumption: that the open model actually matches the frontier model on your task. If it does, the cheaper move is usually the next row down, not a GPU.
• Vs a hosted open-weight API, same model (the yardstick that matters): for gpt-oss-120b, your raw GPU cost beats the $0.60 hosted price only above ~17% sustained utilization, and once the 3–5× ops factor is loaded in, you need roughly 50–80% sustained utilization just to tie. Against DeepInfra's $0.17 base price, an all-in win is out of reach for almost everyone.
• Vs hosted for the big MoEs (DeepSeek-class): hosted wins at every utilization level: $0.38 per million output versus your $0.45 per million raw GPU cost at perfect saturation, before ops. The economics of pooled utilization plus expert-parallel serving belong to whoever aggregates the most traffic, and that isn't a single-tenant cluster.

Published break-even estimates for self-hosting span roughly 50×, from "more than ~8,000 conversations/day" at one end to "only above hundreds of millions of tokens/day" at the other. The spread comes entirely from differing assumptions about utilization and hidden ops cost. A single number won't help here. Use the tiered rule of thumb LeanLM works from, which tracks where third-party TCO analyses converge:

• Under ~$50K/year of LLM spend → use the API. Your volume won't keep a GPU busy, and one engineer to run inference costs more than the bill you're trying to cut.
• ~$50K–$500K/year → go hybrid. Keep most traffic on the API; self-host only your single highest-volume, most predictable task where a fixed GPU genuinely stays saturated.
• Over ~$500K/year of steady volume → a well-utilized GPU cluster (often with LoRA fine-tuning) usually wins, provided you can keep utilization high and staff the ops.

A useful sanity check from the utilization math above: a small model needs roughly 50% sustained GPU utilization just to beat a cheap hosted model on cost. If you can't confidently project that, the API is almost certainly cheaper all-in. And the target keeps moving: API prices fell on the order of 80% across 2025–2026, which pushes the break-even toward ever-higher volume.

At What Token Volume & GPU Utilization Does Self-Hosting Win?

Because self-hosting exchanges variable per-token pricing for a fixed hourly hardware lease, calculating your break-even point requires solving for two tightly linked variables: sustained GPU utilization and minimum daily token volume.

The mathematical relationship governing utilization-adjusted inference cost is straightforward:

When you solve this equation across real-world serving runtimes and hardware tiers, the minimum daily token volume required to justify self-hosting becomes stark:

Break-even calculations assume mixed 3:1 input/output token distributions. Hourly rates based on RunPod on-demand GPU pricing as of October 2026. All-in operational costs reflect an industry standard 4× multiplier over raw compute to capture MLOps salaries, orchestration, and staging redundancy.

The "Average Traffic" Trap and Diurnal Spikes

A fatal mistake in self-hosting capacity planning is calculating break-even against average daily token volume. In production, enterprise AI workloads follow an aggressive diurnal pattern: traffic peaks between 9:00 AM and 5:00 PM local time, with an 8:1 peak-to-trough ratio. If your application averages 100 million tokens per day, your peak throughput might demand 5,500 tokens/second at 2:00 PM, while dropping to 400 tokens/second at 3:00 AM.

Because self-hosted GPUs cannot scale to zero instantly without incurring 45–120 second container and model-weight reload latencies, you must provision hardware to meet peak demand plus a safety margin for tail-latency SLAs. A cluster sized for peak load will run at 10–15% utilization for more than 14 hours every day. Unless your workload can be dumped into asynchronous batch queues (such as nightly document indexing or offline ETL), diurnal traffic variance effectively triples your real cost per token, destroying the financial case for dedicated hardware.

The Operational Rule of Thumb: Unless you generate at least 100 million tokens per day of steady, predictable traffic (or have non-negotiable compliance boundaries), managed APIs or intelligent multi-model routing will deliver a lower cost per token and zero operational overhead.

When Self-Hosting Is Worth It, and When It Isn't

Cost is only one axis. Self-hosting can be the right call even when it's not strictly cheaper, and the wrong call even when the napkin math looks good.

"The fastest way to lose money on AI infrastructure is to buy a GPU before you've measured your traffic. Idle hours don't show up in the benchmark. They show up on the invoice."

Best Self-Hosted Models: SLM vs LLM

The most commonly self-hosted open-weight families in mid-2026 are gpt-oss (20B and 120B; the 120B fitting a single H100 is what made it the default), Qwen3.5/3.6 (the MoE A3B variants for serving, small dense models at the edge), and DeepSeek V3/R1-class for teams with multi-GPU budgets; Llama 4 and the dense Llama 3.x models are still widely deployed but are now the previous generation. Often the bigger decision is size. Family matters less. A small model (gpt-oss-20b, a small dense Qwen, or Phi-class) fits on cheap hardware, serves far more requests per GPU, and frequently matches a frontier model on a specific narrow task while costing a fraction to run.

That's the quiet reason self-hosting sometimes wins. An open 120B doesn't beat a frontier model head-to-head, but you don't need frontier capability for classification, extraction, or templated responses. Pair a right-sized small model with the model-routing instinct (cheapest capable model per task) and the economics improve far more than chasing a lower GPU rate ever will.

The same logic applies to coding models. On one 32GB card, a stock 7B model with a test-feedback retry ladder passed 87.8% of EvalPlus's hidden HumanEval tests, and a 132-sample fine-tune that took 3.08 minutes cut tokens per task 17.3% at the same blind score within noise. The numbers, with the legend, are in Qwen2.5-Coder-7B hits 87.8% HumanEval+ on one 32GB GPU.

The Hidden-Cost Ledger: What the GPU Invoice Doesn't Show

The 3–5× all-in multiplier isn't hand-waving. It itemizes. The line items below are LeanLM's planning estimates for a typical production deployment; before comparing any self-host number to an API price, walk the ledger:

• Engineering time: commonly estimated at 10–20 hours/month of MLOps/DevOps work per deployment ($750–3,000/month at typical rates), and one mid-level engineer realistically covers only 4–6 production GPUs. For most teams this line alone exceeds the GPU bill.
• Model churn: open-weight leaders turn over every few months (this page's model list would have looked different in January). Each swap means re-provisioning, re-quantizing, and re-validating quality on your traffic.
• Redundancy and failover: a hosted API's SLA is priced in; your second node isn't. Capacity for zero-downtime deploys and spot-instance eviction is idle capacity you pay for.
• Idle time itself: the biggest line, covered above: at 25% utilization, 75% of your GPU spend buys nothing.
• Electricity and facilities (owned hardware): a 4090-class box under sustained load draws ~600W, about $52/month at $0.12/kWh running 24/7. That's the facilities bill, and it's the wrong number to watch. The one that matters is joules per request, because on a saturated GPU that's capacity. On our own runs, holding the model fixed and changing only the output constraint moved per-request energy from 73.1 J to 24.3 J. That's two thirds of the serving capacity back, on hardware you have already paid for. See measured energy per request across model sizes and workloads, or the raw measurements.

Where Loupe fits

We measure energy per request on a stated hardware boundary. The measurements are here.

Want this measured on your own workload? Scope a measurement ($5k to $15k, about two weeks).

Frequently Asked Questions

How much does it cost to host your own LLM?

It depends almost entirely on GPU utilization. As of July 2026, an NVIDIA H100 rents for about $2.89/hr on-demand (RunPod Secure Cloud, H100 PCIe, unchanged as of September 27, 2026; ~$1–1.50 spot, ~$6.90/GPU-hr on AWS which only sells 8-GPU nodes). At full saturation serving gpt-oss-120b that works out to roughly $0.10 per million tokens of raw compute. But a GPU sitting idle still bills the full hourly rate, so at 10% utilization the per-token cost is 10× higher, and once you add the engineering time to run the cluster, the realistic all-in cost is about 3–5× the raw GPU rental.

Is self-hosting an LLM worth it?

For most teams under roughly $50K/year of LLM spend, no. Commercial APIs are cheaper and far less work once you count engineering, redundancy, and idle GPU time. Self-hosting starts to win at high, steady volume that keeps GPUs busy (commonly cited thresholds range from about 8,000 conversations/day to hundreds of millions of tokens/day, depending on assumptions), or when data residency, latency control, or a fine-tuned model make it a strategic rather than purely cost decision.

When is self-hosting an LLM cheaper than an API?

Self-hosting is cheaper only when a GPU stays heavily utilized. Below the break-even, the idle hours you pay for erase the per-token advantage. Published break-even estimates vary by roughly 50× because they assume different utilization and ops overhead, so the answer is a range: API for low or spiky volume, a hybrid (API plus self-host for the highest-volume task) in the middle, and a well-utilized GPU cluster only for sustained heavy load.

What GPU do you need to self-host an LLM?

It is set by the model's memory footprint: total parameters for MoE models, even though only a fraction are active per token. gpt-oss-20b (~13GB at MXFP4) runs on an L4 or RTX 4090-class card. gpt-oss-120b (~63GB) and Qwen3.6-35B-A3B in FP8 (~35GB) each fit a single H100 or A100 80GB. A dense 70B in FP8 needs two H100s in practice, because the weights alone fill one card and leave no KV-cache headroom. DeepSeek 671B-class MoEs need 8× H100 or H200. Always add KV-cache headroom, which grows with context length and concurrency.

What is the best self-hosted LLM model?

As of mid-2026 the most widely self-hosted open-weight families are gpt-oss (20B/120B; the 120B fits a single H100, which made it the default serving workhorse), Qwen3.5/3.6 (MoE A3B variants), and DeepSeek V3/R1-class for multi-GPU budgets; Llama 4 and Llama 3.x remain common but previous-generation. For narrow tasks, a small model such as gpt-oss-20b, a small dense Qwen, or Phi-class is often the right answer. It fits on cheap hardware, serves more requests per GPU, and matches a frontier model on the specific task while costing a fraction to run.

Can self-hosting beat DeepInfra, Groq, or Together per-token prices?

Rarely, and for large MoE models effectively never. Those providers serve the same open-weight models at pooled, near-saturated utilization: gpt-oss-120b at $0.60 per million output tokens (as low as $0.17), DeepSeek-V3.2 at $0.38. Self-hosting gpt-oss-120b beats $0.60 in raw GPU cost only above roughly 17% sustained utilization, and all-in (with the 3–5× ops factor) only above roughly 50–80%. For a 671B-class MoE, hosted pricing beats your raw GPU cost at every utilization level. Self-hosting wins on control, compliance, and fine-tuned models. Beating hosted open-weight prices is rarely the reason.

Does quantization actually lower self-hosted serving cost?

Yes, measurably. Serving in FP8 instead of BF16 benchmarks at roughly 24% lower cost per million tokens with about 99.5% benchmark-accuracy retention across Llama- and Mixtral-family evaluations, and the memory savings often enable larger batches on top. gpt-oss ships MXFP4 natively (~63GB for the 120B), though H100s run MXFP4 via software emulation, while Blackwell cards run it natively, which is why one B200 roughly matches two H100s on that model.

Does self-hosting an LLM save money for a small team?

Rarely. Small teams usually have variable, bursty traffic that leaves GPUs idle, and a single mid-level MLOps engineer can cost more per year than a large API bill. Commercial APIs also keep cutting prices (roughly 80% over 2025–2026), which pushes the break-even higher. Start on an API, instrument your real token volume and traffic pattern, and revisit self-hosting only when sustained utilization would actually keep a GPU busy.

At what GPU utilization does self-hosting an LLM win over API pricing?

It depends on the model class and the comparator. Against a frontier closed API ($10–50/1M output), self-hosting wins even at 15–25% utilization. But against hosted open-weight APIs running the identical model (e.g., $0.60/1M output for gpt-oss-120b or $0.38/1M for DeepSeek 671B), self-hosting raw GPU hardware requires roughly 17% utilization on a single H100 to break even on compute alone, and 50–80% sustained utilization once fully loaded with a 3–5× ops factor (engineering, monitoring, redundancy). For massive 671B-class MoEs, hosted pooled endpoints beat self-hosted raw GPU cost at 100% saturation because multi-tenant providers aggregate continuous batching across thousands of concurrent streams.

At what token volume is self-hosting open-weight LLMs cheaper than a managed API?

For a single-GPU workhorse model like gpt-oss-120b on a dedicated H100 ($2.89/hr = ~$2,080/month raw, ~$7,500/month all-in with ops), the raw compute break-even against hosted open-weight endpoints ($0.60/1M output) requires at least 3.5 billion output tokens per month (~115 million tokens per day). Factoring in full MLOps overhead, you need over 12.5 billion tokens per month (~415 million tokens per day) of steady, predictable traffic to justify dedicated infrastructure over managed APIs. Below 100 million tokens per day, managed APIs or hybrid routing are almost always cheaper.

What software helps reduce cost per token for self-hosted LLMs?

The standard 2026 production stack combines: 1) vLLM (v0.31+) with PagedAttention and automatic prefix caching (saving 70–90% of prefill compute on repeated system prompts and few-shot examples); 2) SGLang with RadixAttention for multi-turn conversational agents and complex structured JSON extraction (reusing KV caches across tree-structured dialogue for 3–5× throughput gains); 3) TensorRT-LLM for kernel-level FP8 GEMM optimization and speculative decoding (1.8–2.4× lower latency and higher tokens/watt on NVIDIA Hopper/Blackwell); and 4) memory-quantization formats like AutoAWQ, AutoFP8, and Marlin kernels that cut VRAM footprint by 50–75%, allowing 2× larger batch sizes on the same hardware.