LLM GPU requirements

gpt-oss-120b GPU requirements: VRAM, GPUs and servers

How much GPU memory gpt-oss-120b needs in BF16, FP8 and 4-bit, how its KV cache grows with context, what fine-tuning takes, and which servers run it, with monthly prices.

Parameters: 117B 5.1B active
Architecture: MoE 36 layers
KV heads: 8 × 64 dim
Context: 128k tokens

Open in the VRAM calculator

Cheapest servers that run it

MXFP4 (native 4-bit) weights · 8k context · 8 concurrent requests · about 68 GB

NVIDIA RTX PRO 6000 Blackwell In stock

96 GB GDDR7 · Single GPU

$1,390/mo
NVIDIA H100 SXM In stock

80 GB HBM3 · Single GPU

$2,190/mo
NVIDIA H200 In stock

141 GB HBM3e · Single GPU

$3,090/mo

Delivered in under 10 minutes after payment.

Inference memory

How much VRAM gpt-oss-120b needs

Weights take parameters × bytes per parameter. Serving adds the KV cache and about 10% runtime overhead; the totals below assume an 8k context with 8 concurrent requests and a BF16 KV cache.

gpt-oss-120b memory by precision and the cheapest server that fits
Precision	Weights	Total to serve	Cheapest server that fits
BF16 2 bytes per parameter	234 GB	261 GB	8× NVIDIA RTX PRO 6000 Blackwell $10,490/mo
FP8 1 byte per parameter	117 GB	132 GB	NVIDIA B200 $4,590/mo
MXFP4 / 4-bit 0.5 bytes per parameter	59 GB	68 GB	NVIDIA RTX PRO 6000 Blackwell $1,390/mo

Estimates with about 8% of GPU memory kept free. Quantized weights also keep some layers in higher precision, and frameworks differ in overhead; check the free memory your engine reports at startup. A model split across GPUs with tensor parallelism runs best on NVLink servers (HGX); the 8× RTX PRO 6000 server links its GPUs over PCIe.

GPUs and servers

Which GPUs can run gpt-oss-120b?

Yes. In MXFP4 (native 4-bit), a single NVIDIA H100 SXM (80 GB, $2,190 a month at OverGPU) runs gpt-oss-120b with an 8k context and 8 concurrent requests. The table checks every configuration we rent in each precision.

Whether each GPU configuration runs gpt-oss-120b in BF16, FP8 and 4-bit
Configuration	BF16	FP8	MXFP4
NVIDIA GeForce RTX 5090 32 GB · $449/mo	—	—	—
NVIDIA L40S 48 GB · $890/mo	—	—	—
NVIDIA H100 SXM 80 GB · $2,190/mo	—	—	Yes
NVIDIA RTX PRO 6000 Blackwell 96 GB · $1,390/mo	—	—	Yes
NVIDIA H200 141 GB · $3,090/mo	—	—	Yes
NVIDIA B200 180 GB · $4,590/mo	—	Yes	Yes
NVIDIA B300 288 GB · $5,190/mo	—	Yes	Yes
8× NVIDIA HGX H100 8 × 80 GB · $16,900/mo	Yes	Yes	Yes
8× NVIDIA RTX PRO 6000 Blackwell 8 × 96 GB · $10,490/mo	Yes	Yes	Yes
8× NVIDIA HGX H200 8 × 141 GB · $23,900/mo	Yes	Yes	Yes
8× NVIDIA HGX B200 8 × 180 GB · $34,900/mo	Yes	Yes	Yes
8× NVIDIA HGX B300 8 × 288 GB · $39,900/mo	Yes	Yes	Yes
8× AMD Instinct MI355X 8 × 288 GB · $19,900/mo	Yes	Yes	Yes

Yes: 8 concurrent requests with an 8k context each. 1 req: a single 8k request at a time. —: the weights, KV cache and runtime do not fit. Same formulas as the tables above, with about 8% of GPU memory kept free.

KV cache

gpt-oss-120b KV cache by context length

Each token in flight stores keys and values for 18 full-attention layers: 2 × 18 layers × 8 KV heads × 64 dimensions × 2 bytes = 36 KB per token in BF16. An FP8 KV cache halves these figures.

KV cache size by context length and number of concurrent requests
Context per request	1 request	8 requests	32 requests
4k tokens	151 MB	1.2 GB	4.8 GB
32k tokens	1.2 GB	9.7 GB	39 GB
128k tokens	4.8 GB	39 GB	155 GB

The sliding-window layers keep only the last 128 tokens, which adds a few megabytes per request and is left out above.

Fine-tuning

Memory to fine-tune gpt-oss-120b

Rough totals before tuning batch size and sequence length: QLoRA keeps 4-bit base weights, LoRA keeps BF16 weights frozen, and a full fine-tune with AdamW needs about 16 bytes per parameter plus activations, sharded across GPUs.

Fine-tuning memory by method and the cheapest server that fits
Method	Approximate memory	Cheapest server that fits
QLoRA (4-bit base)	88 GB	NVIDIA H200 $3,090/mo
LoRA (BF16 base)	290 GB	8× NVIDIA RTX PRO 6000 Blackwell $10,490/mo
Full fine-tune (AdamW, mixed precision)	2,246 GB	More than 8 GPUs: GB300 NVL72

Full fine-tuning assumes sharded training (FSDP or ZeRO-3) with activation checkpointing. See the fine-tuning guide for methods and settings.

About gpt-oss-120b

gpt-oss-120b is OpenAI’s open-weight Mixture-of-Experts model: about 117B parameters with roughly 5.1B active per token, 36 layers that alternate full attention and a 128-token sliding window, and Apache 2.0 weights. OpenAI releases the expert weights in MXFP4, a 4-bit format, and states that the model runs on a single 80 GB GPU.

Every figure on this page comes from the model’s published architecture and the same formulas as our VRAM calculator, where you can change the context length, concurrency and KV cache precision.

Model facts

Developer: OpenAI
License: Apache 2.0
Type: Mixture of Experts
Parameters: 117Babout 5.1B active per token
Layers: 36alternating full and sliding-window attention
Attention: 64 query heads, 8 KV headsgrouped-query attention, head dimension 64
Context window: 128k tokens

FAQ

gpt-oss-120b GPU FAQ

The short version. The help center has the details, and customers with an active server get answers by ticket, 24/7.

Visit the help center

How much VRAM does gpt-oss-120b need?

About 234 GB for the weights in BF16, 117 GB in FP8 and 59 GB in 4-bit, plus the KV cache: about 37 MB per 1,000 tokens in flight with a BF16 cache. Serving it in MXFP4 (native 4-bit) with an 8k context and 8 concurrent requests takes about 68 GB in total.

Can gpt-oss-120b run on a single GPU?

Yes. In MXFP4 (native 4-bit), a single NVIDIA RTX PRO 6000 Blackwell (96 GB) serves it with an 8k context and 8 concurrent requests. Longer contexts or more users need more memory for the KV cache, so check your case in the VRAM calculator.

What is the cheapest OverGPU server for gpt-oss-120b?

For MXFP4 (native 4-bit) serving with an 8k context and 8 concurrent requests, the NVIDIA RTX PRO 6000 Blackwell at $1,390 per month is the lowest-priced configuration that fits. For other contexts and concurrency, size it in the VRAM calculator.

How much memory do I need to fine-tune gpt-oss-120b?

Roughly 88 GB with QLoRA, 290 GB with LoRA on BF16 weights, and 2,246 GB for a full fine-tune with AdamW in mixed precision, before tuning batch size and sequence length. Full fine-tunes of large models are sharded across GPUs with FSDP or ZeRO-3.

Deploy a GPU server

Run gpt-oss-120b on your own GPUs.

Choose a configuration, create your account in the same form and pay the first month from your balance. Your server is delivered in under 10 minutes.

Delivered in under 10 minutes
Cancel any month
Pay with crypto, up to +50% bonus

Configure RTX PRO 6000 See all prices