How to fine-tune a 70B model: GPU requirements and workflow

You can fine-tune a 70B model on a single GPU or on a full 8-GPU server, and both answers are right: it depends on whether you update every weight, train a LoRA adapter on 16-bit weights, or train one on a 4-bit base with QLoRA. The memory needed ranges from about 40 GB to well over 1 TB. This guide explains where those numbers come from, which GPUs each method needs, and how to run the job from data preparation to serving. The examples use Llama 3.1 70B: 80 layers, a hidden size of 8,192 and 8 key-value heads.

70B fine-tuning GPU requirements at a glance

For budgeting, those configurations cost $2,190 per month for one H100, $3,090 for one H200, $4,590 for one B200, $5,190 for one B300, $16,900 for 8× H100, $23,900 for 8× H200, $34,900 for 8× B200 and $39,900 for 8× B300. A month also leaves room for several runs, which is how fine-tuning usually goes. The rest of this guide explains the numbers; for the general method behind them, see our guide to VRAM for LLMs.

Full fine-tuning: why it takes over 1 TB

Standard mixed-precision training with AdamW keeps five numbers per parameter:

Setups that keep gradients or communication buffers in FP32 land closer to 20 bytes per parameter, or 1.4 TB. Either way, the training state is over 1 TB before a single activation is stored.

No single GPU holds that, and plain data parallelism does not help, because every GPU keeps a full copy. You need sharding: PyTorch FSDP or DeepSpeed ZeRO stage 3 split weights, gradients and optimizer states across all GPUs and gather each layer's weights only while it computes. Across eight GPUs, each one holds 140 GB of state at 16 bytes per parameter, or 175 GB at 20. That arithmetic explains the hardware:

• 8× H100 (640 GB in total) cannot hold the state on one server.
• 8× H200 (141 GB each) holds 140 GB of state per GPU with nothing left for activations. It works only if you move the FP32 master weights and Adam moments to CPU memory (about 840 GB, which fits in the server's 2 TB of RAM) or shrink them with 8-bit optimizer states, if your sharding setup supports them. Offloading slows each step, because the optimizer runs on the CPU and state crosses PCIe.
• 8× B200 (180 GB each) leaves about 40 GB per GPU for activations and buffers at 16 bytes per parameter, which is workable with activation checkpointing. A setup closer to 20 bytes per parameter leaves almost nothing, and needs offloading here too.
• 8× B300 (288 GB each) leaves well over 100 GB per GPU, enough for longer sequences and larger micro-batches without tricks.

Two servers over a fast network are another route, but full fine-tuning of a 70B model is rarely the first thing to try. For most instruction-tuning and domain-adaptation jobs, start with LoRA, and move to a full fine-tune only when you have the data and evidence that the adapter is the bottleneck. When you do need it, our 8-GPU servers are bare metal, with NVLink between all eight GPUs.

Activation memory and sequence length

Activations are the second half of the bill, and the half that scales with your data. During the forward pass, every layer keeps intermediate tensors for the backward pass. For a Llama-style 70B model in BF16, that comes to roughly 25 MB per token across all 80 layers, so a single 8,192-token sequence would need around 200 GB on its GPU. Sharding does not reduce this, because each GPU stores the activations of its own micro-batch.

Activation checkpointing, also called gradient checkpointing, fixes it. Instead of keeping everything, each layer stores only its input, one vector of 8,192 BF16 values (16 KB) per token, and recomputes the rest during the backward pass. That is about 1.3 MB per token for 80 layers, or roughly 11 GB for one 8,192-token sequence, plus a few GB of working space for the layer being recomputed. The cost is roughly one extra forward pass of compute per step.

Two more items catch people out. The output logits over Llama 3.1's vocabulary of 128,256 tokens take about 4.2 GB in FP32 for one 8,192-token sequence, before their gradients; fused or chunked loss implementations avoid materializing them all at once. And sequence length is the main lever you control: pack short examples together, keep the micro-batch at one or two sequences, and use gradient accumulation to reach the effective batch size you want.

LoRA in BF16: the frozen weights set the floor

LoRA freezes the base model and trains small low-rank matrices next to chosen projection layers. The base weights still have to sit in GPU memory: about 140 GB in BF16. The adapters are tiny by comparison. A rank-16 LoRA on all seven projection matrices of each of the 80 layers adds about 207 million trainable parameters, roughly 0.3% of the model, or about 3.3 GB with their gradients and optimizer states.

Activations do not shrink much, because gradients still flow back through the frozen layers to reach the adapters in early layers, so keep activation checkpointing on. The total for one 8,192-token sequence is around 160 GB. That sets the hardware:

• 2× 80 GB is the floor: 160 GB in total, enough only with short sequences.
• 8× H100 or 8× H200 is comfortable. Sharded across eight GPUs, the frozen weights take 17.5 GB per GPU, which leaves room for long sequences and for data parallelism across all eight.
• One GPU also works, with no sharding to configure, at the speed of one GPU: a 180 GB B200 holds the job with about 20 GB to spare at 8,192 tokens, and a 288 GB NVIDIA B300 holds it comfortably.

With Hugging Face PEFT, a typical configuration for Llama-style models targets every linear projection:

from peft import LoraConfig

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    task_type="CAUSAL_LM",
)

The QLoRA authors found that applying adapters to all linear layers, not only to attention, was needed to match full fine-tuning in their experiments. A learning rate around 1e-4 is a common starting point for LoRA on large models, roughly ten times what full fine-tuning would use.

QLoRA: a 70B model on a single GPU

QLoRA stores the frozen base in the 4-bit NF4 format and trains 16-bit LoRA adapters on top, dequantizing weights on the fly. The 70B base then takes about 35 GB, or about 40 GB in practice once you add the quantization constants and the embedding and output layers, which stay in 16-bit. Add 3–4 GB for adapters and their optimizer states and roughly 14 GB of checkpointed activations for an 8,192-token sequence, and the job fits on one 80 GB NVIDIA H100. A 141 GB H200 gives you room for longer sequences or larger micro-batches.

The original QLoRA paper fine-tuned a 65B model on a single 48 GB GPU. Llama 3.1 70B is slightly larger and has a vocabulary four times as large, so plan on 80 GB. With Transformers and bitsandbytes, loading the base in 4-bit looks like this:

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-70B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)

The trade-offs: steps are slower than BF16 LoRA because of dequantization, and the adapter learns against a quantized model that differs slightly from the BF16 one you will probably serve. Evaluate after merging, not only during training.

How to fine-tune a 70B model: a practical workflow

The hardware is the easy part. Most failed fine-tunes fail on data or evaluation, not on memory.

1. Prepare the data. Format examples as conversations and render them with the model's own chat template (tokenizer.apply_chat_template in Transformers). Deduplicate, filter out low-quality examples, and remove personal data you should not train on. Hold out an evaluation split that is never trained on. Measure token lengths: the longest sequences you keep set your activation memory.
2. Pick a tool. Hugging Face TRL with PEFT provides trainers for supervised fine-tuning and preference tuning, with Accelerate, FSDP or DeepSpeed for multiple GPUs. Axolotl wraps the same building blocks in YAML configs. torchtune offers PyTorch-native recipes for full, LoRA and QLoRA fine-tuning. All three handle the methods above; pick the one whose config style your team will maintain.
3. Pin the environment. Use a container or a virtual environment with fixed versions of PyTorch, Transformers, PEFT and your trainer, so that a rerun next month behaves the same.
4. Smoke-test small. Run the same pipeline on an 8B model for a few hundred steps. It catches template, masking and data bugs before they cost you a long run.
5. Launch and watch. Start multi-GPU jobs with torchrun --nproc_per_node=8 train.py or your tool's own launcher, inside tmux so that an SSH disconnect does not kill the run. Watch training and evaluation loss, GPU memory and step time; a loss spike or a rising evaluation loss is a reason to stop early.

For the communication side on 8-GPU servers, see our multi-GPU and NCCL guide.

Checkpoints on NVMe

Checkpoints are large, and you want them on fast local disks:

• Full fine-tuning: a resumable checkpoint with optimizer states is the size of the training state, 1.1–1.4 TB. Write it sharded, with every rank saving its own part in parallel.
• Model weights only: about 140 GB in BF16, for evaluation and serving.
• LoRA adapter: about 0.4 GB at rank 16 in BF16, so you can keep many.

Our HGX servers carry 8× 3.84 TB of NVMe, room for datasets, the base model and several full checkpoints. The single H100 plan has 2 TB, plenty for QLoRA, where the BF16 base download is the largest file. Keep the last few checkpoints locally, copy the best one off the server, and remember that we do not back up local disks unless agreed in writing. The local NVMe storage guide has the details.

Evaluate, then serve

Evaluate against the base model with the same prompts, chat template and sampling settings:

• Loss on the held-out split, to catch overfitting.
• Task metrics on your own evaluation set: exact match, a rubric or pairwise preference.
• A regression set of general prompts, to catch lost capabilities such as instruction following or answers in other languages.
• A human read of a sample of outputs. Automated judges help, but they miss things.

For serving, you have two options. Merge the adapter into the base weights with merge_and_unload() in PEFT, then serve the merged model, quantized to FP8 if you want it on a single H200. Merging needs the base in BF16, about 140 GB, which fits in the 225 GB of RAM on our single H100 plan. Or keep the base model loaded and attach adapters at runtime: vLLM can serve several LoRA adapters on one base model. Our guide to self-hosting LLM inference with vLLM covers the server side. If you are planning a run, see our GPU servers for fine-tuning.

Frequently asked questions

How much GPU memory do I need to fine-tune a 70B model?

About 40–45 GB for QLoRA and about 145 GB for LoRA on a BF16 base, before activations. Full fine-tuning with AdamW needs 1.12–1.4 TB for weights, gradients and optimizer states alone.

Can I fine-tune a 70B model on a single GPU?

Yes, with QLoRA: the 4-bit base takes about 40 GB, so a single 80 GB H100 or 141 GB H200 has room for adapters and activations. LoRA on a BF16 base fits on a single 180 GB B200, or more comfortably on a 288 GB B300.

Can 8× H100 fully fine-tune a 70B model?

Not on one server with standard mixed-precision AdamW: the training state is over 1 TB, and 8× H100 has 640 GB. Use LoRA on 8× H100, or move to 8× B300, or 8× B200 with a lean setup, for full fine-tuning.

Is LoRA good enough compared with full fine-tuning?

For many instruction-tuning and domain-adaptation tasks, LoRA on all linear layers is a strong and much cheaper baseline. If you have the data and the budget, compare both on your own evaluation set before committing to a full fine-tune.

How large is a LoRA adapter for a 70B model?

A rank-16 adapter on all linear projections has about 207 million parameters, roughly 0.4 GB in BF16, so you can keep many versions and swap them at serving time.