Self-Hosting an LLM: Is It Actually Cheaper Than the API?

Self-hosting an open LLM can undercut frontier API list prices — but only when the GPU stays busy, and rarely against hosted open-weight APIs serving the same model. At high-concurrency batching (July 2026 rates), a single H100 serving gpt-oss-120b gets raw GPU cost down to roughly $0.10 per million tokens — but hosted endpoints serve the identical model for $0.60 per million output tokens (and as little as $0.17), so the margin at perfect utilization is a few ×, not the 100× the hourly-rate arithmetic suggests. Run a DeepSeek-class 671B MoE yourself and hosted pricing beats your raw GPU cost at every utilization level. At 10% utilization every self-hosted number is 10× worse, and the all-in cost with engineering runs 3–5× the raw GPU bill. The honest break-even: self-hosting beats frontier API pricing at sustained volume, but it beats a hosted open-weight API running the same model only near-saturation — published break-even estimates disagree by about 50× precisely because they hide these assumptions.

So "is self-hosting cheaper?" is really a utilization question wearing a hardware costume. This guide walks the actual 2026 cost math, the GPU and hardware tiers, the self-host-vs-API break-even, and the honest answer to when it's worth it. It's the build-vs-buy companion to per-call LLM cost optimization — before you optimize an API bill, it's worth knowing whether you should be paying it at all.

LeanLM (not affiliated with Google's LearnLM educational AI) is an LLM cost optimization platform. Note: "LLM" here means Large Language Model — not the Master of Laws degree, which shares the acronym and pollutes a lot of cost searches.

The Real Cost Math: Why Utilization Is Everything

Start with the optimistic number, because it's the one that gets quoted. Take an H100 at $2.89/hr on-demand (RunPod, July 2026), serving gpt-oss-120b — the current single-GPU workhorse, a 120B MoE that fits one 80GB card at its native MXFP4 precision — at a benchmarked aggregate throughput of ~8,000 tokens/second per GPU on a realistic mixed workload (GPUStack, vLLM). That's about 28.8 million tokens per hour, so the raw GPU cost is roughly $0.10 per million tokens at full saturation. Next to GPT-5.5 at $30 per million output tokens, that looks like a 300× saving — and that single comparison is how most "just self-host it" arguments are built. The comparison that actually matters: Groq, Fireworks, and Together all serve the identical gpt-oss-120b for $0.60 per million output tokens, and DeepInfra's base endpoint lists $0.17 — so at perfect utilization your raw margin over just calling the hosted model is about 6×, before you've paid anyone to run it. (You'll also see wilder self-hosting multiples in the wild; they usually come from dividing an hourly rate by a single-stream token count, or a straight arithmetic slip.)

And three assumptions behind even that number almost never hold. First, saturation: a GPU you rent by the hour bills the same whether it's serving 8,000 tokens/second or zero, so at 10% utilization the effective cost per token is 10× higher. Second, batching: aggregate throughputs come from serving many requests concurrently — NVIDIA's benchmark for a dense 70B (Llama 3.3, 2× H100 FP8) runs 51.7 tokens/second at concurrency 1 but 920 at concurrency 250, an ~18× spread — so break-even math built on single-stream numbers is wrong before it starts. Third, ops: one mid-level MLOps engineer typically covers only 4–6 GPUs, and LeanLM's working multiplier — consistent with independent total-cost-of-ownership analyses — puts the all-in cost at 3–5× the raw GPU rental. A peer-reviewed 2026 costing study makes the utilization point concrete: the same Mixtral 8x7B deployment on an H100 costs $15.25 per million output tokens at 1 request/second but $0.87 at 25 requests/second — a 17× swing from traffic shape alone.

Effective raw GPU cost per million tokens, by utilization (July 2026)

Raw GPU rental only — multiply by the 3–5× ops factor for all-in cost. Derived from RunPod on-demand rates and DeepInfra list prices (both pulled July 22, 2026), GPUStack's vLLM benchmark for gpt-oss-120b on H100, and the vLLM project's production figure of 2,200 tok/s per H200 for 671B-class DeepSeek (8-GPU minimum; per-GPU shown). Throughputs are aggregate served tokens on batched workloads. Read the last column carefully: for the 671B MoE, the hosted API beats your raw GPU cost at every utilization level — and even for gpt-oss-120b, all-in cost with ops beats the $0.60 hosted price only above roughly 50–80% sustained utilization.

This is the same lesson as long-context cost management, one level up: the cost you don't see (idle GPU-hours, re-sent context) dwarfs the cost you do. The lever isn't a cheaper GPU — it's keeping the one you have busy.

What GPUs actually cost (July 2026)

As of July 22, 2026. RunPod on-demand figures verified against the live pricing page that day; spot/marketplace prices (Vast.ai, Spheron) move minute-to-minute, so treat them as a floor, not a plannable rate. AWS sells H100/A100 only in 8-GPU instances, so the per-GPU figure is arithmetic, not a rentable unit. Prices have been falling through 2025–2026 — re-verify before budgeting.

GPU and Hardware Requirements: What You Actually Need

Which GPU you need is set by one thing first: whether the model's weights plus its KV cache fit in VRAM. And in 2026 that calculation has a twist, because the workhorse open models are now mixture-of-experts (MoE): you pay VRAM for the model's total parameters but earn throughput on its much smaller active parameters. Qwen3.6-35B-A3B holds 35B parameters in memory and computes with 3B per token; DeepSeek's 671B-class models compute with ~37B. That asymmetry — memory cost of a big model, compute speed of a small one — is exactly why hosted providers with pooled traffic serve these models so cheaply, and why your half-idle GPU can't match them. A sizing guide for what people actually self-host in mid-2026:

Weights only; add KV-cache headroom, which grows with context length and concurrency — it's the KV cache, not the weights, that pushes a dense 70B off a single 80GB card. Cluster-scale models (DeepSeek-V4-Pro, 1.6T params) are omitted: if it takes 8× B200 to hold, "self-host to save money" is not the question. As of July 2026.

Throughput then determines how many requests that GPU can actually serve — and the batched-vs-single-stream gap is where most break-even math dies. On a single H100 with vLLM, gpt-oss-120b delivers roughly 8,000 aggregate tokens/second on a realistic mixed workload (GPUStack's published benchmark), while an interactive single user sees more like 500–600 tokens/second — a 13–16× spread between the number your capacity plan needs and the number a demo shows you. For the big MoEs, the vLLM project's production benchmark sustains 2,200 tokens/second per H200 serving 671B-class DeepSeek across an 8-GPU group. Whichever model you run: aggregate benchmarked throughput divided into your real demand is your utilization — and utilization is the whole game.

Quantization is the second lever, and in 2026 it's table stakes rather than exotic: gpt-oss ships MXFP4 natively, and serving a model in FP8 instead of BF16 measures out to roughly 24% lower cost per million tokens (Baseten, TensorRT-LLM on H100) while retaining ~99.5% of benchmark accuracy (Red Hat's evaluation across Llama and Mixtral families) — with the memory savings often enabling larger batches on top. One hardware caveat: H100s run MXFP4 through software emulation; Blackwell-generation cards run it natively, which is why one B200 roughly matches two H100s on gpt-oss-120b.

Self-Hosted vs API: The Honest Break-Even

To compare fairly you need the other side of the ledger: current API list prices. As of 2026, per million tokens (input / output):

List prices as of July 2026; confirm against each provider and note the cheap tier is what self-hosting actually competes with. See our LLM effective cost table for cache- and batch-adjusted real prices, or run your own token mix through the cost calculator to get the API side of your break-even in dollars. Watch the Haiku ambiguity: Haiku 4.5 is $1/$5; the older Haiku 3 was $0.25/$1.25.

But the closed-model table above is only half the yardstick — and it's the flattering half. The comparison self-hosting actually has to win is against hosted open-weight APIs serving the exact model you'd deploy:

Pulled from provider docs and DeepInfra's live model API on July 22, 2026. The gpt-oss-120b price is unusually solid — Groq, Fireworks, and Together publish the identical $0.15/$0.60. These prices are the honest benchmark: a provider serving the same open model at pooled, near-saturated utilization.

The three break-evens (there is no single number)

Put the two tables together with the utilization math above and "when does self-hosting win?" splits into three different questions with three different answers:

• Vs a frontier closed API ($12–30 per million output): self-hosting an open model wins early — even modest utilization of a $0.10-per-million-at-saturation rig undercuts frontier list prices, with the 3–5× ops multiplier included. But this comparison smuggles in the hard assumption: that the open model actually matches the frontier model on your task. If it does, the cheaper move is usually the next row down, not a GPU.
• Vs a hosted open-weight API, same model (the real yardstick): for gpt-oss-120b, your raw GPU cost beats the $0.60 hosted price only above ~17% sustained utilization — and once the 3–5× ops factor is loaded in, you need roughly 50–80% sustained utilization just to tie. Against DeepInfra's $0.17 base price, an all-in win is out of reach for almost everyone.
• Vs hosted for the big MoEs (DeepSeek-class): hosted wins at every utilization level — $0.38 per million output versus your $0.45 per million raw GPU cost at perfect saturation, before ops. The economics of pooled utilization plus expert-parallel serving belong to whoever aggregates the most traffic, and that isn't a single-tenant cluster.

Here's where honesty matters more than a tidy chart. Published break-even estimates for self-hosting span roughly 50× — from "more than ~8,000 conversations/day" at one end to "only above hundreds of millions of tokens/day" at the other. That spread isn't sloppiness; it's entirely driven by differing assumptions about utilization and hidden ops cost. The useful framing isn't a single number, it's the tiered rule of thumb LeanLM uses, consistent with where independent TCO analyses converge:

• Under ~$50K/year of LLM spend → use the API. Your volume won't keep a GPU busy, and one engineer to run inference costs more than the bill you're trying to cut.
• ~$50K–$500K/year → go hybrid. Keep most traffic on the API; self-host only your single highest-volume, most predictable task where a fixed GPU genuinely stays saturated.
• Over ~$500K/year of steady volume → a well-utilized GPU cluster (often with LoRA fine-tuning) usually wins — provided you can keep utilization high and staff the ops.

A useful sanity check from the utilization math above: a small model needs roughly 50% sustained GPU utilization just to beat a cheap hosted model on cost. If you can't confidently project that, the API is almost certainly cheaper all-in. And the target keeps moving — API prices fell on the order of 80% across 2025–2026, which pushes the break-even toward ever-higher volume.

When Self-Hosting Is Worth It — and When It Isn't

Cost is only one axis. Self-hosting can be the right call even when it's not strictly cheaper, and the wrong call even when the napkin math looks good.

"The fastest way to lose money on AI infrastructure is to buy a GPU before you've measured your traffic. Idle hours don't show up in the benchmark — they show up on the invoice."

Best Self-Hosted Models: SLM vs LLM

The most commonly self-hosted open-weight families in mid-2026 are gpt-oss (20B and 120B — the 120B fitting a single H100 is what made it the default), Qwen3.5/3.6 (the MoE A3B variants for serving, small dense models at the edge), and DeepSeek V3/R1-class for teams with multi-GPU budgets; Llama 4 and the dense Llama 3.x models are still widely deployed but are now the previous generation. The highest-leverage decision, though, is often size, not family. A small model — gpt-oss-20b, a small dense Qwen, or Phi-class — fits on cheap hardware, serves far more requests per GPU, and frequently matches a frontier model on a specific narrow task while costing a fraction to run.

That's the quiet reason self-hosting sometimes wins: not because an open 120B beats a frontier model head-to-head, but because you don't need frontier capability for classification, extraction, or templated responses. Pair a right-sized small model with the model-routing instinct — cheapest capable model per task — and the economics improve far more than chasing a lower GPU rate ever will.

The Hidden-Cost Ledger: What the GPU Invoice Doesn't Show

The 3–5× all-in multiplier isn't hand-waving — it itemizes. The line items below are LeanLM's planning estimates for a typical production deployment; before comparing any self-host number to an API price, walk the ledger:

• Engineering time — commonly estimated at 10–20 hours/month of MLOps/DevOps work per deployment ($750–3,000/month at typical rates), and one mid-level engineer realistically covers only 4–6 production GPUs. For most teams this line alone exceeds the GPU bill.
• Model churn — open-weight leaders turn over every few months (this page's model list would have looked different in January). Each swap means re-provisioning, re-quantizing, and re-validating quality on your traffic.
• Redundancy and failover — a hosted API's SLA is priced in; your second node isn't. Capacity for zero-downtime deploys and spot-instance eviction is idle capacity you pay for.
• Idle time itself — the biggest line, covered above: at 25% utilization, 75% of your GPU spend buys nothing.
• Electricity and facilities (owned hardware) — a 4090-class box under sustained load draws ~600W, about $52/month at $0.12/kWh running 24/7. Small next to cloud rental, real for a rack.

How LeanLM Models the Self-Host Decision

LeanLM treats self-host-vs-API as a measurement problem, not a debate:

1. Instrument your real traffic — capture monthly input/output tokens and the hour-by-hour load shape, because the traffic pattern decides utilization
2. Size the model to hardware — map your model and quantization to the cheapest GPU tier that fits with KV-cache headroom
3. Model utilization honestly — divide demand by batched throughput, including idle hours, instead of assuming full saturation
4. Load in the hidden costs — apply the 3–5× ops multiplier and any compliance or redundancy overhead
5. Compare and recommend — put fully loaded self-host cost next to all-in API price, and surface the hybrid split when it beats both

The output is a number you can take to a budget review: self-host, stay on the API, or run a hybrid — with the break-even shown on your data, not a vendor's.

Frequently Asked Questions

How much does it cost to host your own LLM?

It depends almost entirely on GPU utilization. As of July 2026, an NVIDIA H100 rents for about $2.89/hr on-demand (RunPod; ~$1–1.50 spot, ~$6.90/GPU-hr on AWS which only sells 8-GPU nodes). At full saturation serving gpt-oss-120b that works out to roughly $0.10 per million tokens of raw compute. But a GPU sitting idle still bills the full hourly rate, so at 10% utilization the per-token cost is 10× higher — and once you add the engineering time to run the cluster, the realistic all-in cost is about 3–5× the raw GPU rental.

Is self-hosting an LLM worth it?

For most teams under roughly $50K/year of LLM spend, no — commercial APIs are cheaper and far less work once you count engineering, redundancy, and idle GPU time. Self-hosting starts to win at high, steady volume that keeps GPUs busy (commonly cited thresholds range from about 8,000 conversations/day to hundreds of millions of tokens/day, depending on assumptions), or when data residency, latency control, or a fine-tuned model make it a strategic rather than purely cost decision.

When is self-hosting an LLM cheaper than an API?

Self-hosting is cheaper only when a GPU stays heavily utilized. Below the break-even, the idle hours you pay for erase the per-token advantage. Published break-even estimates vary by roughly 50× because they assume different utilization and ops overhead, so the honest answer is a range: API for low or spiky volume, a hybrid (API plus self-host for the highest-volume task) in the middle, and a well-utilized GPU cluster only for sustained heavy load.

What GPU do you need to self-host an LLM?

It is set by the model's memory footprint — total parameters for MoE models, even though only a fraction are active per token. gpt-oss-20b (~13GB at MXFP4) runs on an L4 or RTX 4090-class card. gpt-oss-120b (~63GB) and Qwen3.6-35B-A3B in FP8 (~35GB) each fit a single H100 or A100 80GB. A dense 70B in FP8 needs two H100s in practice (weights alone fill one card, leaving no KV-cache headroom). DeepSeek 671B-class MoEs need 8× H100 or H200. Always add KV-cache headroom, which grows with context length and concurrency.

What is the best self-hosted LLM model?

As of mid-2026 the most widely self-hosted open-weight families are gpt-oss (20B/120B — the 120B fits a single H100, which made it the default serving workhorse), Qwen3.5/3.6 (MoE A3B variants), and DeepSeek V3/R1-class for multi-GPU budgets; Llama 4 and Llama 3.x remain common but previous-generation. For narrow tasks, a small model such as gpt-oss-20b, a small dense Qwen, or Phi-class is often the right answer — it fits on cheap hardware, serves more requests per GPU, and matches a frontier model on the specific task while costing a fraction to run.

Can self-hosting beat DeepInfra, Groq, or Together per-token prices?

Rarely, and for large MoE models effectively never. Those providers serve the same open-weight models at pooled, near-saturated utilization: gpt-oss-120b at $0.60 per million output tokens (as low as $0.17), DeepSeek-V3.2 at $0.38. Self-hosting gpt-oss-120b beats $0.60 in raw GPU cost only above roughly 17% sustained utilization, and all-in (with the 3–5× ops factor) only above roughly 50–80%. For a 671B-class MoE, hosted pricing beats your raw GPU cost at every utilization level. Self-hosting wins on control, compliance, and fine-tuned models — beating hosted open-weight prices is usually not the reason.

Does quantization actually lower self-hosted serving cost?

Yes, measurably. Serving in FP8 instead of BF16 benchmarks at roughly 24% lower cost per million tokens with about 99.5% benchmark-accuracy retention across Llama- and Mixtral-family evaluations, and the memory savings often enable larger batches on top. gpt-oss ships MXFP4 natively (~63GB for the 120B), though H100s run MXFP4 via software emulation — Blackwell cards run it natively, which is why one B200 roughly matches two H100s on that model.

Does self-hosting an LLM save money for a small team?

Rarely. Small teams usually have variable, bursty traffic that leaves GPUs idle, and a single mid-level MLOps engineer can cost more per year than a large API bill. Commercial APIs also keep cutting prices (roughly 80% over 2025–2026), which pushes the break-even higher. Start on an API, instrument your real token volume and traffic pattern, and revisit self-hosting only when sustained utilization would actually keep a GPU busy.