# Mistral Small 24B GPU requirements: VRAM, GPUs and servers

> Mistral Small 24B needs about 47 GB of VRAM in BF16, 24 GB in FP8 and 12 GB in 4-bit, plus KV cache. GPUs and servers that run it, with prices.

Source: https://overgpu.com/models/mistral-small-24b  
Updated: 2026-11-03  
Publisher: OverGPU (https://overgpu.com/)

**In short:** Yes. In FP8, a single NVIDIA L40S (48 GB, $890 a month at OverGPU) runs Mistral Small 24B with an 8k context and 8 concurrent requests; one NVIDIA GeForce RTX 5090 (32 GB) holds a single 8k request.

## Model facts

- **Developer:** Mistral AI
- **License:** Apache 2.0
- **Type:** Dense
- **Parameters:** 23.6B
- **Layers:** 40
- **Attention:** 32 query heads, 8 KV heads, head dimension 128
- **Context window:** 128k tokens (3.1 and later)

## How much VRAM Mistral Small 24B needs

Weights take parameters × bytes per parameter; serving adds the KV cache and about 10% runtime overhead. Totals assume an 8k context with 8 concurrent requests and a BF16 KV cache.

| Precision | Weights | Total to serve | Cheapest OverGPU server that fits |
| --- | --- | --- | --- |
| BF16 | 47 GB | 64 GB | NVIDIA RTX PRO 6000 Blackwell ($1,390/month) |
| FP8 | 24 GB | 38 GB | NVIDIA L40S ($890/month) |
| 4-bit | 12 GB | 25 GB | NVIDIA GeForce RTX 5090 ($449/month) |

## Which GPUs can run Mistral Small 24B?

Yes = 8 concurrent requests with an 8k context each; 1 request = a single 8k request at a time; No = does not fit.

| Configuration | BF16 | FP8 | 4-bit |
| --- | --- | --- | --- |
| NVIDIA GeForce RTX 5090 (32 GB, $449/month) | No | 1 request | Yes |
| NVIDIA L40S (48 GB, $890/month) | No | Yes | Yes |
| NVIDIA H100 SXM (80 GB, $2,190/month) | Yes | Yes | Yes |
| NVIDIA RTX PRO 6000 Blackwell (96 GB, $1,390/month) | Yes | Yes | Yes |
| NVIDIA H200 (141 GB, $3,090/month) | Yes | Yes | Yes |
| NVIDIA B200 (180 GB, $4,590/month) | Yes | Yes | Yes |
| NVIDIA B300 (288 GB, $5,190/month) | Yes | Yes | Yes |
| 8× NVIDIA HGX H100 (8 × 80 GB, $16,900/month) | Yes | Yes | Yes |
| 8× NVIDIA RTX PRO 6000 Blackwell (8 × 96 GB, $10,490/month) | Yes | Yes | Yes |
| 8× NVIDIA HGX H200 (8 × 141 GB, $23,900/month) | Yes | Yes | Yes |
| 8× NVIDIA HGX B200 (8 × 180 GB, $34,900/month) | Yes | Yes | Yes |
| 8× NVIDIA HGX B300 (8 × 288 GB, $39,900/month) | Yes | Yes | Yes |
| 8× AMD Instinct MI355X (8 × 288 GB, $19,900/month) | Yes | Yes | Yes |

## Mistral Small 24B KV cache by context length

About 164 MB per 1,000 tokens in flight with a BF16 cache.

| Context per request | 1 request | 8 requests | 32 requests |
| --- | --- | --- | --- |
| 4k tokens | 671 MB | 5.4 GB | 21 GB |
| 32k tokens | 5.4 GB | 43 GB | 172 GB |
| 128k tokens | 21 GB | 172 GB | 687 GB |

## Memory to fine-tune Mistral Small 24B

| Method | Approximate memory | Cheapest OverGPU server that fits |
| --- | --- | --- |
| QLoRA (4-bit base) | 18 GB | NVIDIA GeForce RTX 5090 ($449/month) |
| LoRA (BF16 base) | 59 GB | NVIDIA RTX PRO 6000 Blackwell ($1,390/month) |
| Full fine-tune (AdamW, mixed precision) | 453 GB | 8× NVIDIA RTX PRO 6000 Blackwell ($10,490/month) |

## About Mistral Small 24B

Mistral Small 3.x is a 24B dense model family (about 23.6B parameters) with 40 layers and 8 KV heads, released under Apache 2.0; versions 3.1 and later accept a 128k-token context. It sits between the 14B and 32B classes: in BF16 its weights take about 47 GB, and FP8 halves that.

Every figure comes from the model's published architecture and the formulas of the VRAM calculator (https://overgpu.com/tools/vram-calculator).

## Frequently asked questions

### How much VRAM does Mistral Small 24B need?

About 47 GB for the weights in BF16, 24 GB in FP8 and 12 GB in 4-bit, plus the KV cache: about 164 MB per 1,000 tokens in flight with a BF16 cache. Serving it in FP8 with an 8k context and 8 concurrent requests takes about 38 GB in total.

### Can Mistral Small 24B run on a single GPU?

Yes. In FP8, a single NVIDIA L40S (48 GB) serves it with an 8k context and 8 concurrent requests. Longer contexts or more users need more memory for the KV cache, so check your case in the VRAM calculator.

### What is the cheapest OverGPU server for Mistral Small 24B?

For FP8 serving with an 8k context and 8 concurrent requests, the NVIDIA L40S at $890 per month is the lowest-priced configuration that fits. For other contexts and concurrency, size it in the VRAM calculator.

### How much memory do I need to fine-tune Mistral Small 24B?

Roughly 18 GB with QLoRA, 59 GB with LoRA on BF16 weights, and 453 GB for a full fine-tune with AdamW in mixed precision, before tuning batch size and sequence length. Full fine-tunes of large models are sharded across GPUs with FSDP or ZeRO-3.

## Related pages

- [LLM VRAM calculator: will your model fit on a GPU?](https://overgpu.com/tools/vram-calculator): Pick Llama, Qwen, Mistral or gpt-oss, set the precision and context length, and see weights, KV cache and which GPUs fit.
- [How much VRAM do I need for an LLM?](https://overgpu.com/blog/vram-for-llms): Quick answers for 7B, 14B, 32B, 70B and 405B models at BF16, FP8 and 4-bit, plus KV cache by context and the GPU that fits.
- [Best GPU for LLM inference in 2026: a sizing guide by model size](https://overgpu.com/blog/best-gpu-for-llm-inference): Use our 2026 sizing table, from RTX 5090 to 8× B300, to choose.
- [LLM GPU requirements](https://overgpu.com/models): VRAM, KV cache and the cheapest servers for Llama, Qwen3, Mistral and gpt-oss models.