How Much VRAM Do You Need for a Local LLM? Measured

A 7B model at Q4 needs about 5 GB of VRAM at a 4,000-token context and 6.8 GB at 32,000, measured in Ollama.

A 32B at Q4 needs 20 to 27 GB and a 72B at Q4 needs 46 to 55 GB.

Context length adds memory on top of the weights: 1.8 GB for a 7B and 9 GB for a 72B going from 4k to 32k.

LeanLM (not affiliated with Google's LearnLM educational AI) measures what AI workloads cost to serve. Every number below comes from our own runs on a stated GPU.

How much VRAM does a local LLM need?

Between 5 GB and 82 GB in our tests: a 7B needs 5 to 10 GB, a 32B 20 to 41 GB and a 72B 46 to 82 GB, depending on the quant and the context length. We ran Qwen2.5 Instruct at 7B, 32B and 72B, each at Q4_K_M and Q8_0 (4-bit and 8-bit quantized weights), at a 4,096 and a 32,768-token context, in Ollama on one NVIDIA RTX PRO 6000 (96 GB):

Peak VRAM is nvidia-smi memory.used, the maximum over the cell, and includes the CUDA context; all GB on this page are binary (GiB, as nvidia-smi reports). Decode tok/s and J per short task (idle-subtracted GPU energy) are means over 20 HumanEval prompts at temperature 0. Time to first token (TTFT) is from one long prompt filling about 70% of the 32k context. One NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB), Ollama with OLLAMA_FLASH_ATTENTION=1, every cell 100% on the GPU. Full data: how-much-vram-local-llm-measured.csv. Speed and energy are from the 4k runs; the 32k runs were within a few percent (see the CSV).

Which models fit 8GB, 16GB, 24GB and 32GB cards?

A 16 GB card runs any 7B we tested, a 24 GB card runs a 32B at Q4 only with a short context, and a 72B needs 48 GB or more. The fits below are derived from the measured peaks above plus 1 GB of headroom for the OS and other processes; they weren't run on those specific cards.

Derived from the measured peaks above plus 1 GB of headroom, not measured directly on these card sizes. Use the two tables as a VRAM calculator: find the model size and quant, then read the peak at your context length. Weights-only calculators leave out the growth of the KV cache (the attention keys and values the model stores for every token in the context) shown in the next section.

The rig runs in your VPC, or your engineers run it and we interpret the output.

How much memory does context length add?

Context length adds memory through the KV cache, and how much depends on the model's layers and attention heads. The quant doesn't change it. Going from 4k to 32k tokens after load added 1.8 GB for the 7B (Q4) and 1.7 GB (Q8), 7.2 GB for the 32B (both quants), and 9.0 GB for the 72B (Q4) and 8.8 GB (Q8). The KV cache grows with context length, and it's about the same size at a given context whether the weights are Q4 or Q8.

The measured growth matches the standard KV cache formula within 0.3 GB:

KV cache per token = 2 (K and V) x layers x KV heads x head dim x 2 bytes (f16)

For Qwen2.5: the 7B (28 layers, 4 KV heads, head dim 128) works out to 56 KiB per token, about 1.5 GB for the 28,672 extra tokens between 4k and 32k. The 32B (64 layers, 8 KV heads) works out to 256 KiB per token, about 7.0 GB. The 72B (80 layers, 8 KV heads) works out to 320 KiB per token, about 8.8 GB. Those are derived from the formula; the measured growth (1.8, 7.2 and 9.0 GB) landed within 0.3 GB of each.

Worked VRAM sizing examples

To size a card without guessing, start from the measured 4k figure (which already includes the model weights and the CUDA runtime context), then add the KV cache for your target context length:

• 8 GB card (laptop or desktop GPU, 16k coding context). Start with Qwen2.5 7B Q4_K_M at 4k: 5.1 GB measured. Raising context from 4,096 to 16,384 adds 12,288 tokens at 56 KiB per token, which is 0.67 GiB of KV cache, putting expected GPU memory near 5.8 GiB. In our 507-task coding run at num_ctx 16384, Qwen2.5-Coder 7B measured 5.9 GiB median GPU memory, leaving about 2 GiB for the OS display server on an 8 GB card.
• 16 GB card (desktop workstation, 16k to 32k context). You have two options that stay 100% in VRAM: a 7B at Q8_0 with a 32k context (9.7 GB measured peak, leaving over 6 GB free), or the 20B mixture-of-experts coding model gpt-oss:20b at its native MXFP4 precision with a 16k context (13.0 GiB measured median GPU memory, leaving 3 GiB free). A 32B dense model at Q4_K_M needs 19.9 GB even at a 4k context, so it will not fit in 16 GB.
• 24 GB card (RTX 3090, RTX 4090 or L4, 16k vs 32k context). At a 4k to 16k context, 27B to 32B models at Q4_K_M fit comfortably: Qwen2.5 32B peaked at 19.9 GB at 4k, Qwen 3.8 27B used 19.1 GiB at 16k, and Qwen3-Coder 30B used 19.5 GiB at 16k. Extending a 32B dense model to a 32k context adds 7.2 GB of f16 KV cache (256 KiB per token across 28,672 tokens), pushing peak VRAM to 27.2 GB and spilling off a 24 GB card. On a 24 GB card, cap num_ctx at 16,384 for 27B to 32B models, or step up to a 32 GB card for 32k to 128k contexts. When serving repeated system prompts on the same card, prefix caching reuses KV blocks across requests; see measured prompt caching in vLLM and Ollama.

Q4 or Q8: which quant should you run?

Q8 needs more memory than Q4, runs slower and uses more energy per task, in every pair we measured. The 7B ran at 75 tok/s at Q8 against 106 tok/s at Q4 and used 1,180 J against 887 J per short task. The 32B ran at 20 tok/s against 39 tok/s and used 4,720 J against 2,690 J. The 72B ran at 16 tok/s against 20 tok/s and used 6,514 J against 5,894 J. We didn't measure any accuracy difference between the two quants in this test, so this comparison is speed, memory and energy only.

Best local LLM for coding on 16GB or 24GB VRAM

Whether a 27B or 30B coding model fits a 24 GB card depends on the context window you allocate. At a 16,384-token coding context on one NVIDIA RTX PRO 6000 (96 GB), all five local coding models we tested across 507 EvalPlus tasks (164 HumanEval+ and 343 MBPP+, up to 3 attempts per task) stayed under 20 GiB of GPU memory:

Measured across 507 EvalPlus tasks (164 HumanEval+ and 343 MBPP+) with up to 3 attempts per task (blind retry on a failed visible test), num_ctx 16384, num_predict 4096, on one NVIDIA RTX PRO 6000 (96 GB), 2026-09-29. The 128k VRAM peaks (23.8 GB and 25.8 GB) are from our single-attempt run on one 32 GB AMD Radeon AI PRO R9700 (Ollama/Vulkan, 2026-09-28). Raw data: best-local-llm-for-coding-measured.csv.

By card tier, the coding choices break down cleanly:

• 8 GB VRAM: Run qwen2.5-coder:7b. It used 5.9 GiB at a 16k context, passed 81.1% of the 507 tasks (and 87.8% of HumanEval+ in our 7B retry-ladder test), and finished at a median 1.3 seconds and 137 J per task.
• 16 GB VRAM: Run gpt-oss:20b. At 13.0 GiB of GPU memory it fits a 16 GB card with 3 GiB to spare and tied for the highest score in the entire test (84.8%, 430 of 507) at 849 J per task; see the five-model local coding benchmark.
• 24 GB vs 32 GB VRAM: At a 16k context, both Qwen 3.8 27B (19.1 GiB) and Qwen3-Coder 30B (19.5 GiB) fit on a 24 GB card. Once you allocate a 128k context, Qwen 3.8 27B peaked at 23.8 GB and Qwen3-Coder 30B peaked at 25.8 GB on our 32 GB AMD R9700, so long-context coding belongs on a 32 GB card. See the full Qwen 3.8 27B coding benchmark on the R9700 for single-attempt and hard-task scores, and what self-hosting costs once a model fits before buying hardware.

Can a local LLM run on system RAM or CPU offload when VRAM runs out?

Yes, Ollama and llama.cpp will automatically keep the layers that fit on the GPU and spill the remaining layers into system RAM for the CPU to execute, rather than crashing with an out-of-memory error. The problem is speed. Autoregressive token generation reads every active model weight once for every single output token, so decode speed is governed by memory bandwidth.

A workstation or desktop GPU moves 500 to 1,800 GB/s across its GDDR6 or GDDR7 memory bus. Dual-channel desktop DDR5 system RAM moves roughly 50 to 90 GB/s, and any activation traffic crossing a PCIe 4.0 x16 slot is capped near 25 to 32 GB/s. As soon as a model spills a third or half of its layers into system RAM, every generated token has to wait on those CPU-resident layers at system-RAM bandwidth. We kept every cell in our tests 100% on the GPU and didn't benchmark partial CPU offload directly, because the 507-task coding table above shows a better alternative: step down one model size so the entire model and its KV cache stay in VRAM. Across 507 coding tasks, dropping from the 19.1 GiB Qwen 3.8 27B to the 13.0 GiB gpt-oss:20b lost zero pass rate (both scored 84.8%) while cutting median task time from 6.8 seconds to 4.3 seconds, and dropping all the way to the 5.9 GiB qwen2.5-coder:7b still held 81.1% at a median 1.3 seconds per task.

What hasn't been measured yet?

This test covers Ollama on one 96 GB workstation card at two context lengths (4k and 32k for the Qwen2.5 size-and-quant grid, plus 16k for the five-model coding run) and one 32 GB AMD R9700 at 128k context. On the Qwen2.5 grid, requesting num_ctx 131,072 was capped by Ollama at Qwen2.5's default of 32,768, so memory stayed at the 32k figure and the long prompt was truncated to 16,386 tokens. By the KV cache formula, a 7B at 128k would add about 5.3 GB more KV cache over 32k; that's derived, not measured. It also doesn't cover decode speed on consumer 8 GB or 16 GB cards (the speeds in the first table are from a 96 GB workstation card; consumer cards with less memory bandwidth will decode slower, though the VRAM footprints carry over), partial CPU offload speeds, other runtimes (llama.cpp directly, vLLM), accuracy differences between Q4 and Q8, or multi-GPU tensor-parallel splits.

Frequently Asked Questions

How much VRAM do I need to run a 7B model?

About 5 GB at Q4 with a 4,000-token context and 6.8 GB at 32,000 tokens, measured in Ollama. At Q8 it's 8.5 to 9.7 GB.

Can I run a 32B model on 24GB VRAM?

At Q4 with a short context, yes: it peaked at 19.9 GB at 4,000 tokens. At 32,000 tokens it needed 27.2 GB, which is more than a 24 GB card has.

How much VRAM does a 70B model need?

The closest size we tested is Qwen2.5 72B. A 72B at Q4 needed 46.2 GB at a 4,000-token context and 54.7 GB at 32,000. At Q8 it needed 73.4 to 81.6 GB.

Is 16GB VRAM enough for a local LLM?

For a 7B model, yes, at Q4 or Q8 and up to a 32,000-token context. A 32B model needs about 20 GB even at Q4 with a short context.

How much does context length add to VRAM?

It depends on the model's layers and KV heads. The quant barely changes it. Going from 4,000 to 32,000 tokens added 1.8 GB for Qwen2.5 7B, 7.2 GB for 32B and 9 GB for 72B.

Is Q8 worth it over Q4?

It costs more memory, speed and energy: a 32B at Q8 ran at 20 tokens per second against 39 at Q4 and used 75% more energy per task. We haven't measured the accuracy difference.

Can a local LLM run on system RAM if I don't have enough VRAM?

Yes, runtimes like Ollama and llama.cpp will spill layers that don't fit in VRAM onto system RAM and the CPU, but token generation slows down sharply because decode is memory-bandwidth bound. Desktop DDR5 system RAM delivers roughly 50 to 90 GB/s of bandwidth, against 500 to 1,800 GB/s on modern GPUs. In our coding runs, a model that fits 100% in VRAM such as Qwen2.5-Coder 7B (5.9 GiB, 81.1% pass rate) or gpt-oss 20B (13.0 GiB, 84.8% pass rate) finished tasks in 1.3 to 4.3 seconds.

What local LLM can I run with 8GB of VRAM?

A 7B model at 4-bit quantization (Q4_K_M). In Ollama, Qwen2.5 7B at Q4_K_M peaked at 5.1 GB of VRAM at a 4,096-token context and 6.8 GB at a 32,768-token context, and Qwen2.5-Coder 7B used a median 5.9 GiB at a 16,384-token context while passing 81.1% of 507 coding tasks. A 7B at Q8_0 (8.5 to 9.7 GB) exceeds an 8 GB card.

Methods

• Hardware. RunPod, one NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB), measured 2026-09-28 (size/quant/context grid) and 2026-09-29 (507-task five-model coding run); plus one AMD Radeon AI PRO R9700 (32 GB, Ollama/Vulkan, 2026-09-28) for the 128k context peaks.
• Software. Ollama with OLLAMA_FLASH_ATTENTION=1, KV cache f16 (default).
• Models. Size/quant grid: Qwen2.5 Instruct 7B, 32B and 72B from the Ollama library, each at q4_K_M and q8_0. Coding run: qwen2.5-coder:7b, gpt-oss:20b, qwen3.8:latest (27B), glm-4.7-flash:latest and qwen3-coder:30b.
• Context. num_ctx 4,096 and 32,768 on the size/quant grid; num_ctx 16,384 (num_predict 4,096) on the 507-task coding run; 128k allocated on the R9700 run.
• Per-cell procedure. Size/quant grid: 20 HumanEval prompts (temperature 0, up to 512 output tokens) for decode speed and energy, plus one long prompt filling about 70% of the context (2,836 tokens at 4k, 23,102 at 32k) for time to first token. Coding run: 507 EvalPlus tasks (164 HumanEval+ and 343 MBPP+), up to 3 attempts per task with blind retry on a failed visible test.
• Memory method. nvidia-smi memory.used, peak over the cell on the size/quant grid and median per task on the 507-task coding run, includes the CUDA context. Every cell ran 100% on the GPU.
• Energy method. GPU board power at 5 Hz, idle power (84 to 92 W on the RTX PRO 6000) subtracted.
• 128k cap. Requesting num_ctx 131,072 on Qwen2.5 was capped by Ollama at Qwen2.5's default of 32,768; memory stayed at the 32k figure and the long prompt was truncated to 16,386 tokens, so 128k wasn't measured on the Qwen2.5 grid.
• Data. Full CSVs: how-much-vram-local-llm-measured.csv, best-local-llm-for-coding-measured.csv.
• Date. Measured 2026-09-28 and 2026-09-29.