LLM GPU requirements: how much VRAM open models need

How to estimate GPU requirements for any LLM

Three parts add up to the GPU memory a model needs to serve, and every page above computes them from the model's published architecture:

• Weights Parameters × bytes per parameter: 2 bytes in BF16, 1 in FP8 and about 0.5 in 4-bit formats. A 70B model takes about 141 GB in BF16 and 71 GB in FP8.
• KV cache 2 × layers × KV heads × head size × bytes per value, for every token in flight (see the glossary for the terms). It grows with the context length and with the number of concurrent requests.
• Runtime overhead Activations, CUDA graphs and allocator fragmentation: plan on about 10% of the weights plus 1 GB, and keep some GPU memory free.

For a model that is not listed, enter its parameters, layers and KV heads in the VRAM calculator; it runs the same formulas and lists every configuration that fits.

Mixture-of-experts models: total parameters set the memory

A mixture-of-experts (MoE) model such as Qwen3 235B-A22B or gpt-oss-120b activates only a fraction of its parameters for each token, about 22B and 5.1B respectively, but every expert must sit in GPU memory. Size the weights from the total parameter count. The active count only sets the compute per token, which is why MoE models generate quickly once they fit.

Inference vs fine-tuning requirements

Fine-tuning needs more memory than serving. QLoRA keeps the base weights in 4-bit and trains small adapters; LoRA trains adapters on top of frozen BF16 weights; a full fine-tune with AdamW in mixed precision keeps weights, gradients and optimizer state for every parameter, about 16 bytes per parameter before activations. For a 70B model our estimates are about 53 GB with QLoRA, 175 GB with LoRA and about 1.36 TB for a full fine-tune, sharded across an 8-GPU server with FSDP or ZeRO-3. The 70B fine-tuning guide walks through each method.

Consumer vs data-center GPUs for LLMs

The GeForce RTX 5090 has 32 GB of GDDR7 at 1.8 TB/s: enough for 8B models in BF16, models up to about 32B at 4-bit with short contexts, and image generation, at the lowest monthly price. Data-center GPUs add capacity and bandwidth: 80 GB of HBM3 on the H100, 141 GB on the H200, 180 GB on the B200 and 288 GB on the B300 and MI355X, with NVLink or Infinity Fabric between the GPUs of an 8-GPU server so one model can span all eight. The RTX PRO 6000 Blackwell sits in between with 96 GB of GDDR7.

The rule of thumb: pick the smallest GPU whose memory holds the weights, the KV cache for your traffic and about 10% headroom, then compare monthly prices. The best GPU for LLM inference guide goes model size by model size.