# LLM GPU requirements: how much VRAM open models need

> VRAM in BF16, FP8 and 4-bit, KV cache and the cheapest servers for Llama, Qwen3, Mistral and gpt-oss models.

Source: https://overgpu.com/models  
Updated: 2026-11-03  
Publisher: OverGPU (https://overgpu.com/)

## GPU requirements of open LLMs

Weights only for BF16, FP8 and 4-bit; the cheapest server serves the model in its usual precision with an 8k context and 8 concurrent requests. Also available as CSV: https://overgpu.com/data/llm-gpu-requirements.csv

| Model | Parameters | BF16 weights | FP8 weights | 4-bit weights | Cheapest OverGPU server |
| --- | --- | --- | --- | --- | --- |
| [Llama 3.1 8B](https://overgpu.com/models/llama-3-1-8b) | 8B | 16 GB | 8.0 GB | 4.0 GB | NVIDIA GeForce RTX 5090 ($449/month) |
| [Qwen3 14B](https://overgpu.com/models/qwen3-14b) | 14.8B | 30 GB | 15 GB | 7.4 GB | NVIDIA RTX PRO 6000 Blackwell ($1,390/month) |
| [Mistral Small 24B](https://overgpu.com/models/mistral-small-24b) | 23.6B | 47 GB | 24 GB | 12 GB | NVIDIA L40S ($890/month) |
| [Qwen3 32B](https://overgpu.com/models/qwen3-32b) | 32.8B | 66 GB | 33 GB | 16 GB | NVIDIA RTX PRO 6000 Blackwell ($1,390/month) |
| [Llama 3.3 70B](https://overgpu.com/models/llama-3-3-70b) | 70.6B | 141 GB | 71 GB | 35 GB | NVIDIA H200 ($3,090/month) |
| [gpt-oss-120b](https://overgpu.com/models/gpt-oss-120b) | 117B | 234 GB | 117 GB | 59 GB | NVIDIA RTX PRO 6000 Blackwell ($1,390/month) |
| [Qwen3 235B-A22B](https://overgpu.com/models/qwen3-235b-a22b) | 235B | 470 GB | 235 GB | 118 GB | 8× NVIDIA RTX PRO 6000 Blackwell ($10,490/month) |
| [Llama 3.1 405B](https://overgpu.com/models/llama-3-1-405b) | 405B | 810 GB | 405 GB | 203 GB | 8× NVIDIA RTX PRO 6000 Blackwell ($10,490/month) |

## How to estimate GPU requirements for any LLM

Three parts add up to the GPU memory a model needs to serve, and every page above computes them from the model's published architecture:

- **Weights:** Parameters × bytes per parameter: 2 bytes in BF16, 1 in FP8 and about 0.5 in 4-bit formats. A 70B model takes about 141 GB in BF16 and 71 GB in FP8.
- **KV cache:** 2 × layers × KV heads × head size × bytes per value, for every token in flight (see the [glossary](https://overgpu.com/glossary) for the terms). It grows with the context length and with the number of concurrent requests.
- **Runtime overhead:** Activations, CUDA graphs and allocator fragmentation: plan on about 10% of the weights plus 1 GB, and keep some GPU memory free.

For a model that is not listed, enter its parameters, layers and KV heads in the [VRAM calculator](https://overgpu.com/tools/vram-calculator); it runs the same formulas and lists every configuration that fits.

## Mixture-of-experts models: total parameters set the memory

A mixture-of-experts (MoE) model such as [Qwen3 235B-A22B](https://overgpu.com/models/qwen3-235b-a22b) or [gpt-oss-120b](https://overgpu.com/models/gpt-oss-120b) activates only a fraction of its parameters for each token, about 22B and 5.1B respectively, but every expert must sit in GPU memory. Size the weights from the total parameter count. The active count only sets the compute per token, which is why MoE models generate quickly once they fit.

## Inference vs fine-tuning requirements

Fine-tuning needs more memory than serving. QLoRA keeps the base weights in 4-bit and trains small adapters; LoRA trains adapters on top of frozen BF16 weights; a full fine-tune with AdamW in mixed precision keeps weights, gradients and optimizer state for every parameter, about 16 bytes per parameter before activations. For a 70B model our estimates are about 53 GB with QLoRA, 175 GB with LoRA and about 1.36 TB for a full fine-tune, sharded across an 8-GPU server with FSDP or ZeRO-3. The [70B fine-tuning guide](https://overgpu.com/blog/fine-tune-70b-model) walks through each method.

## Consumer vs data-center GPUs for LLMs

The [GeForce RTX 5090](https://overgpu.com/gpus/nvidia-rtx-5090) has 32 GB of GDDR7 at 1.8 TB/s: enough for 8B models in BF16, models up to about 32B at 4-bit with short contexts, and image generation, at the lowest monthly price. Data-center GPUs add capacity and bandwidth: 80 GB of HBM3 on the H100, 141 GB on the H200, 180 GB on the B200 and 288 GB on the B300 and MI355X, with NVLink or Infinity Fabric between the GPUs of an 8-GPU server so one model can span all eight. The [RTX PRO 6000 Blackwell](https://overgpu.com/gpus/nvidia-rtx-pro-6000) sits in between with 96 GB of GDDR7.

The rule of thumb: pick the smallest GPU whose memory holds the weights, the KV cache for your traffic and about 10% headroom, then compare monthly prices. The [best GPU for LLM inference](https://overgpu.com/blog/best-gpu-for-llm-inference) guide goes model size by model size.

## LLM GPU requirements FAQ

### How much VRAM do I need to run a 70B model?

About 141 GB for the weights in BF16, 71 GB in FP8 and 35 GB at 4-bit, plus the KV cache and runtime overhead. In FP8, one H200 with 141 GB serves a 70B model with 8 concurrent requests at an 8k context.

### Can I run an LLM on a single GPU?

Most models up to about 70B parameters run on one GPU in FP8 or 4-bit: 8B models on an RTX 5090, 32B models on an H100 and 70B models on an H200. Larger models such as Llama 3.1 405B or Qwen3 235B need an 8-GPU server.

### How much does the KV cache add?

It depends on the model, the context and the number of users. For Llama 3.3 70B, the BF16 KV cache takes about 0.33 MB per token, so 32 requests at 32k tokens need about 340 GB on top of the weights.

### Are these figures exact?

They are estimates from each model's published architecture with the same formulas as the VRAM calculator. Serving engines differ in overhead, so treat them as a sizing guide and check the free memory your engine reports at startup.