# NCCL tests on a multi-GPU server: all_reduce_perf step by step

> Run NCCL tests on a multi-GPU server. Check NVLink topology, build nccl-tests, read busbw from all_reduce_perf and launch torchrun jobs across 8 GPUs.

Source: https://overgpu.com/docs/multi-gpu-nccl  
Updated: 2026-10-22  
Publisher: OverGPU (https://overgpu.com/)

Running NCCL tests on a multi-GPU server is the quickest way to confirm that the GPUs, NVLink and the communication stack work together before you commit to a long training run. This page applies to our 8-GPU servers: check the topology, build NVIDIA's nccl-tests, run all_reduce_perf across eight GPUs, read busbw, then launch PyTorch jobs with torchrun. Multi-node notes come last. AMD servers use RCCL, covered in [ROCm setup](https://overgpu.com/docs/amd-rocm).

## Check the GPU topology

```
nvidia-smi topo -m
nvidia-smi nvlink --status
systemctl status nvidia-fabricmanager
```

In the `topo -m` matrix, `NV#` means two GPUs are connected by NVLink, with # the number of links. `PIX`, `PXB`, `PHB`, `NODE` and `SYS` are PCIe paths, from closest to farthest. The matrix also shows each GPU's CPU and NUMA affinity.

On HGX servers (H100, H200, B200, B300), every GPU pair should show an `NV` entry and Fabric Manager should be `active (running)`. The RTX PRO 6000 server has no NVLink: its GPUs talk over PCIe, so PCIe entries are expected there. Background in [NVLink vs PCIe](https://overgpu.com/blog/nvlink-vs-pcie).

## Build nccl-tests

1. Install the build tools and check for the NCCL development header: `sudo apt install -y git build-essential ls /usr/include/nccl.h`
2. If `nccl.h` is missing, install NCCL from NVIDIA's CUDA repository, set up as described in [driver and CUDA versions](https://overgpu.com/docs/nvidia-drivers-cuda). List the versions, then pick the newest one whose CUDA suffix (such as `+cuda12.8`) is not newer than the CUDA Version shown by nvidia-smi: `apt-cache madison libnccl2 | head sudo apt install -y libnccl2=VERSION libnccl-dev=VERSION`
3. Build the tests: `git clone https://github.com/NVIDIA/nccl-tests.git cd nccl-tests make -j"$(nproc)" CUDA_HOME=/usr/local/cuda` The binaries appear in `./build/`.

To leave the host untouched, build the same way inside an NGC PyTorch container, which already includes CUDA and NCCL.

## Run all_reduce_perf across 8 GPUs

```
./build/all_reduce_perf -b 8 -e 8G -f 2 -g 8
```

One process drives eight GPUs (`-g 8`), from 8 bytes (`-b`) to 8 GB (`-e`), doubling the size at each step (`-f 2`). Each line reports the time and two bandwidth figures, `algbw` and `busbw`, for out-of-place and in-place runs. Check that:

- the `#wrong` columns show 0 on every line, so the results are correct;
- busbw rises with message size and levels off at the largest sizes;
- the final "Avg bus bandwidth" line is stable from one run to the next.

Other collectives work the same way: `all_gather_perf`, `reduce_scatter_perf` and `sendrecv_perf`.

## What busbw means

`algbw` is the data size divided by the time. It changes with the number of GPUs, so it is hard to compare across setups. `busbw` applies a correction factor so the figure reflects the traffic each link actually carries. For all-reduce, the factor is 2 × (n − 1) / n, which is 1.75 with eight GPUs. That makes busbw comparable with the hardware's link bandwidth and between servers of the same model.

Vendor link speeds are peak values that a test does not reach, so judge consistency instead: compare runs, and compare with an identical server if you have one. A sharp drop at large sizes, a result far below a twin server, or a hang usually points to a topology, Fabric Manager or hardware problem. Send us the full output and the topology matrix in a [support ticket](https://overgpu.com/help/getting-support) from your dashboard.

## Debug with NCCL_DEBUG=INFO

```
NCCL_DEBUG=INFO ./build/all_reduce_perf -b 1G -e 1G -g 8 2>&1 \
  | tee nccl-debug.log
```

The log shows the NCCL version, the transport chosen between GPUs (P2P or NVLS over NVLink, SHM through host memory, NET over the network) and the rings and trees NCCL built. On HGX servers you should see NVLink transports. If you see SHM instead, check the topology and Fabric Manager. On the PCIe-based RTX PRO 6000 server, if the test hangs or reports errors, rerun it with `NCCL_P2P_DISABLE=1`. If that run works, peer-to-peer traffic is the problem: open a ticket from your dashboard and we can help investigate.

## Launch DDP or FSDP jobs with torchrun

torchrun starts one process per GPU and sets the variables that PyTorch's distributed package needs. In an environment with PyTorch, save this check as `allreduce_check.py`:

```
import os
import torch
import torch.distributed as dist

dist.init_process_group("nccl")
rank = int(os.environ["LOCAL_RANK"])
torch.cuda.set_device(rank)
x = torch.ones(1024, device="cuda") * (rank + 1)
dist.all_reduce(x)
print(f"rank {rank}: {x[0].item()}")
dist.destroy_process_group()
```

Run it on all eight GPUs:

```
torchrun --standalone --nproc_per_node=8 allreduce_check.py
```

Every rank should print `36.0`, the sum of 1 to 8. Your DDP or FSDP training script starts the same way. DDP keeps a full copy of the model on each GPU and averages gradients. FSDP shards parameters, gradients and optimizer state across GPUs, so you can train models that do not fit in one GPU's memory. Put `NCCL_DEBUG=INFO` in front of the command when a job stalls.

## Multi-node notes

- **Network.** Ask for [private networking](https://overgpu.com/products/private-networking) between your servers, in the order notes when you [deploy](https://overgpu.com/deploy) them or in a ticket from your dashboard, and use the private addresses for rendezvous and NCCL traffic.
- **Interface.** Find the private interface with `ip -br addr` and set `NCCL_SOCKET_IFNAME` to its name on every node, so NCCL does not pick the public interface.
- **Launch.** Run the same command on each node, changing only the rank. Here `10.0.0.1` stands for the first node's private address: `torchrun --nnodes=2 --nproc_per_node=8 --node_rank=0 \ --master_addr=10.0.0.1 --master_port=29500 train.py` Use `--node_rank=1` on the second node, and allow the port between the nodes if a firewall is active.
- **Fabric.** Full-speed GPU traffic between nodes needs an RDMA fabric (InfiniBand or RoCE) cabled and configured between your servers, which the private network does not provide. If one job needs more than eight GPUs exchanging data at NVLink speed, a [GB300 NVL72](https://overgpu.com/gpus/nvidia-gb300-nvl72) rack keeps all 72 GPUs in one NVLink domain.

## Related pages

- [8-GPU server rental: bare-metal NVIDIA HGX and AMD Instinct](https://overgpu.com/products/8-gpu-servers): NVIDIA HGX B300, B200, H200, H100, AMD MI355X or 8x RTX PRO 6000. 50 TB bandwidth, monthly terms, delivered in under 10 minutes.
- [Private network between GPU servers](https://overgpu.com/products/private-networking): Link your OverGPU servers on an isolated network for data pipelines and multi-node jobs. See how it works.
- [GPU servers for LLM training on dedicated 8-GPU nodes](https://overgpu.com/solutions/llm-training): Bare-metal HGX B300, B200 and H200 nodes with NVLink and 400–800G networking, rented monthly. See the memory math first.
- [nvidia-smi commands for monitoring GPU servers](https://overgpu.com/docs/gpu-monitoring): Live views, CSV logs, nvtop, DCGM exporter, ECC and Xid errors, plus rocm-smi equivalents.