GPU servers for LLM training on dedicated 8-GPU nodes

Choosing GPU servers for LLM training starts with memory, not FLOPS: weights, gradients and optimizer state have to fit across the GPUs before the first step runs, and activations come on top. We rent bare-metal NVIDIA HGX nodes with eight whole GPUs on NVLink, by the month, so the same machines, data and checkpoints stay with you from warm-up to the final evaluation.

How much GPU memory LLM training needs

With AdamW and mixed precision, each parameter costs roughly 16–20 bytes before activations: 2 for the BF16 weights, 2 for the gradients, 4 for an FP32 master copy and 8 for the two Adam moments, plus a few more if you keep gradients in FP32. Activations scale with micro-batch size, sequence length, hidden size and layer count. Activation checkpointing and FlashAttention keep them in check, at the cost of some recomputation.

FSDP or DeepSpeed ZeRO-3 shards this state across GPUs, so each GPU holds roughly the total divided by the size of the sharding group. When you are a little short, both can offload optimizer state to CPU memory, which is 2 TB of DDR5 on HGX H100, H200 and B200 nodes and 4 TB on HGX B300, at a cost in step time.

Choosing GPU servers for LLM training

Memory decides what fits; compute decides how long it takes. Training costs about 6 × parameters × tokens floating-point operations, so a 7B model trained on 1 trillion tokens needs about 4.2 × 10²² FLOPs. An HGX B200 node has a vendor-published dense BF16 peak of 18 PFLOPS (8 × 2.25). If your job sustains 40% of that, the run takes about 67 days. On an HGX H200 node, with 8 × 989 TFLOPS of dense BF16 per NVIDIA's specifications, the same run takes about five months. That is why most from-scratch pre-training runs on multi-node clusters, while post-training, continued pre-training on smaller token budgets and small-model work fit a single node and a monthly plan.

• HGX B300: 288 GB per GPU, 2.3 TB per node and eight 800G ConnectX-8 adapters. The roomiest NVIDIA node in our catalog, for 70B-class full training and long contexts.
• HGX B200: 180 GB per GPU, NVLink 5 at 1.8 TB/s per GPU, and a vendor-published 4.5 PFLOPS of dense FP8 per GPU for FP8 training with Transformer Engine.
• HGX H200: 141 GB per GPU and 1.13 TB per node on Hopper, for 8B to 32B training and for post-training.
• 8× AMD Instinct MI355X: 288 GB per GPU with ROCm, if your training stack already runs on AMD.

Sharding, NVLink and multi-node networking

Inside a node, the eight GPUs exchange data over NVLink: 900 GB/s per GPU on H100 and H200, 1.8 TB/s per GPU on B200 and B300, per NVIDIA's published specifications. That bandwidth is what makes tensor parallelism and fully sharded data parallelism efficient within one machine. A practical starting layout:

• Up to about 30B parameters: FSDP or ZeRO-3 across the eight GPUs, and plain data parallelism across nodes.
• 70B and above: tensor parallelism inside the node with Megatron-Core or PyTorch's built-in tensor parallelism, plus pipeline parallelism or FSDP across nodes.
• Mixture-of-experts: expert parallelism is all-to-all heavy, so keep expert groups inside one NVLink domain when you can.

Between nodes, each HGX server has eight 400G ConnectX-7 adapters, or eight 800G ConnectX-8 on HGX B300. To link several of your servers, ask for private networking in the order notes when you deploy them, with the node count and how your job communicates. Before a long run, check the topology with nvidia-smi topo -m and run nccl-tests inside and across nodes, as covered in our multi-GPU and NCCL guide.

Checkpoints, datasets and local NVMe

Each HGX node carries eight 3.84 TB NVMe drives, about 30 TB raw. A full checkpoint with optimizer state takes roughly 12–16 bytes per parameter, close to 1 TB for a 70B model, so keep the last few locally and write them sharded and in parallel with PyTorch Distributed Checkpoint. A tokenized dataset of 1 trillion tokens stored as 32-bit token IDs is about 4 TB, which still leaves room next to your checkpoints.

Local storage is not backed up by us unless agreed in writing, so copy milestone checkpoints off the server. Inbound traffic is not counted, and 8-GPU servers include 50 TB of outbound traffic per month with no egress fees, enough to ship a 1 TB checkpoint out every week. When a server leaves you, its disks are securely erased.

When monthly dedicated nodes fit, and when they do not

A monthly node fits when training keeps it busy: runs measured in weeks, a queue of ablations, or post-training loops where you iterate daily on the same data. In 2026, GPU capacity is tight and many providers prefer one-year contracts, so a node you hold month to month, and can cancel any month with no fee, is often the practical middle ground.

It does not fit a single three-day job followed by an idle month. An HGX B200 node at $34,900 a month works out to about $5.98 per GPU-hour when busy around the clock, but if you use it for only 72 hours that month, the effective rate is about $60 per GPU-hour, and hourly billing elsewhere will cost less. Read monthly vs hourly GPU pricing before you decide. At the other end, frontier-scale pre-training that needs hundreds of GPUs at once belongs on GB300 NVL72 racks, available from $7,990 per GPU per month on 12–36 month terms.

Getting started

1. Choose the configuration on the deployment page, set up the OS image, optional preinstalled software and your login (root password or SSH key), and create a free account or log in in the same form. Use the optional notes for our engineers to describe your model size, token budget and node count.
2. Check the configuration and price, which are shown before you pay.
3. Pay the first month from your prepaid balance, and top up in crypto if your balance is short; the IP and SSH details appear in your dashboard within 10 minutes of payment.
4. Start from our Ubuntu 22.04 or 24.04 image with NVIDIA drivers, CUDA, Docker and the NVIDIA Container Toolkit preinstalled, pull your training container, and validate NCCL before the first long run.

Planning adapters or single-GPU runs instead? See GPUs for LLM fine-tuning. For the hardware details of each node, browse our 8-GPU servers.