# ROCm setup for AMD Instinct MI355X

> ROCm setup for AMD Instinct MI355X servers. Check GPUs with rocm-smi and amd-smi, set user groups, run PyTorch and vLLM in ROCm containers and test RCCL.

Source: https://overgpu.com/docs/amd-rocm  
Updated: 2026-10-22  
Publisher: OverGPU (https://overgpu.com/)

This ROCm setup for AMD Instinct MI355X servers takes you from the first login to PyTorch, vLLM and multi-GPU jobs. Our [8-GPU MI355X servers](https://overgpu.com/gpus/amd-mi355x) are bare metal and ship with a ROCm image, so the kernel driver and ROCm are already installed. Each GPU has 288 GB of HBM3e, per AMD's published specifications. Prefer containers and wheels built on ROCm 7.x: builds made for older ROCm releases may not include code for the MI355X, whose architecture name is `gfx950`.

## Check the GPUs with rocm-smi, amd-smi and rocminfo

```
rocm-smi
amd-smi list
amd-smi static
rocminfo | grep -E 'Marketing Name|gfx'
```

You should see eight GPUs, numbered 0 to 7, in `rocm-smi` and `amd-smi list`. `rocminfo` lists the CPUs and every GPU as agents, and each GPU should report `gfx950`. amd-smi is AMD's newer tool: it covers what rocm-smi does, and `amd-smi static` adds detailed information about each GPU. For ongoing checks, see [GPU monitoring](https://overgpu.com/docs/gpu-monitoring).

## Add users to the render and video groups

Root can use the GPUs directly. Other users need access to `/dev/kfd` and `/dev/dri`, which ROCm grants through the `render` and `video` groups. For each non-root user (here `bob`):

```
sudo usermod -aG render,video bob
# after bob logs out and back in
groups bob
```

The output should include `render` and `video`. Without them, ROCm programs find no GPUs or fail with permission errors on `/dev/kfd`.

## Run PyTorch with ROCm containers or wheels

The simplest path is AMD's PyTorch container, which bundles a matching ROCm user space. If Docker is not on your image, install it first (on Ubuntu, `sudo apt install -y docker.io`). The device flags are explained in [Docker GPU containers](https://overgpu.com/docs/docker-containers).

```
docker run -it --rm \
  --device=/dev/kfd --device=/dev/dri --group-add video \
  --ipc=host --security-opt seccomp=unconfined \
  rocm/pytorch:latest
```

Inside the container, check what PyTorch sees:

```
python3 -c "import torch; print(torch.version.hip, torch.cuda.device_count(),
  torch.cuda.get_device_name(0))"
```

You should see a HIP version, `8` and the GPU name. PyTorch on ROCm keeps the `torch.cuda` API, so most CUDA code runs unchanged. For reproducible builds, pin a specific tag from the rocm/pytorch repository instead of `latest`.

To install on the host instead, use a virtual environment and the ROCm wheel index that PyTorch's install selector lists for your ROCm version:

```
sudo apt install -y python3-venv
python3 -m venv ~/venvs/rocm
source ~/venvs/rocm/bin/activate
pip install torch --index-url https://download.pytorch.org/whl/rocmX.Y
```

Replace `rocmX.Y` with that version, and check that the PyTorch release supports the MI355X. If it does not, you will see `hipErrorNoBinaryForGpu`.

## Choose which GPUs a process sees

```
HIP_VISIBLE_DEVICES=0,1 python3 train.py

# the same inside a container
docker run --rm --device=/dev/kfd --device=/dev/dri --group-add video \
  -e HIP_VISIBLE_DEVICES=0,1 rocm/pytorch:latest \
  python3 -c "import torch; print(torch.cuda.device_count())"
```

The container command should print `2`. Visible GPUs are renumbered from 0 inside the process. HIP also honors `CUDA_VISIBLE_DEVICES`, and `ROCR_VISIBLE_DEVICES` filters at a lower level; set only one of them to avoid confusing combinations. To run two jobs side by side, give each its own set, such as GPUs 0 to 3 and 4 to 7.

## Serve models with vLLM in a ROCm container

AMD publishes ROCm-enabled vLLM images. Start one with the same device flags, publish the port on the loopback interface only, and keep model weights on your data drive:

```
docker run -d --name vllm \
  --device=/dev/kfd --device=/dev/dri --group-add video \
  --ipc=host --security-opt seccomp=unconfined \
  -p 127.0.0.1:8000:8000 \
  -v /data/hf:/data/hf -e HF_HOME=/data/hf \
  rocm/vllm:latest \
  vllm serve Qwen/Qwen2.5-72B-Instruct --tensor-parallel-size 8 \
    --host 0.0.0.0 --port 8000
```

Follow startup with `docker logs -f vllm`, then run `curl http://127.0.0.1:8000/v1/models`: you should see the model listed. Inside the container vLLM listens on all interfaces, while the `127.0.0.1` mapping keeps it private on the host. A 72B model needs roughly 145 GB for its weights in BF16, so it fits on one MI355X; spreading it over eight GPUs leaves far more memory for the KV cache. Options, gated models and API testing are covered in [serving an LLM with vLLM](https://overgpu.com/docs/pytorch-vllm), and exposing the endpoint safely in the [firewall guide](https://overgpu.com/docs/networking-firewall).

## Multi-GPU communication with RCCL

RCCL is ROCm's collective communication library and implements the NCCL API. PyTorch on ROCm uses it when you select the `nccl` backend, so DDP and FSDP scripts launched with `torchrun --standalone --nproc_per_node=8` run unchanged, and RCCL reads the usual NCCL variables, such as `NCCL_DEBUG=INFO`. To measure GPU-to-GPU bandwidth, build AMD's rccl-tests, which mirror NVIDIA's nccl-tests:

```
sudo apt install -y git build-essential
git clone https://github.com/ROCm/rccl-tests.git
cd rccl-tests
make
./build/all_reduce_perf -b 8 -e 8G -f 2 -g 8
```

The `#wrong` columns should show 0 on every line. Reading the bandwidth columns is explained in [NCCL tests](https://overgpu.com/docs/multi-gpu-nccl). For multi-node jobs across several servers, ask for [private networking](https://overgpu.com/products/private-networking) between them in the notes for our engineers when you [deploy](https://overgpu.com/deploy), or open a ticket from your dashboard if your servers are already delivered.

## Troubleshooting

- **hipErrorNoBinaryForGpu.** The build contains no code for `gfx950`. Use a ROCm 7.x-based image or a wheel that supports the MI355X.
- **Permission denied on /dev/kfd, or no GPUs found.** Add the user to the render and video groups and log in again. In Docker, pass both `--device` flags.
- **Fewer than eight GPUs listed.** Run `sudo dmesg | grep -i amdgpu` and send the output, with `amd-smi list`, in a [support ticket](https://overgpu.com/help/getting-support) from your dashboard.

## Related pages

- [Rent an 8-GPU AMD MI355X server for $19,900 per month](https://overgpu.com/gpus/amd-mi355x): A bare-metal 8x AMD Instinct MI355X server (288 GB HBM3e per GPU) for $19,900/month, about $3.41 per GPU-hour. ROCm ready, paid in crypto.
- [AMD MI355X for AI: specs, ROCm and the workloads it suits](https://overgpu.com/blog/amd-mi355x-for-ai): 288 GB of HBM3e at 8 TB/s, FP8 and MXFP4, the real state of ROCm, and the workloads that benefit most. Read before you buy.
- [MI355X vs B200: memory, software and cost compared](https://overgpu.com/compare/mi355x-vs-b200): 288 GB vs 180 GB of HBM, ROCm vs CUDA, 4-bit formats and monthly cost per GPU. See which 8-GPU server fits your models and budget.
- [NCCL tests on a multi-GPU server: all_reduce_perf step by step](https://overgpu.com/docs/multi-gpu-nccl): Check NVLink topology, build nccl-tests, read busbw from all_reduce_perf and launch torchrun jobs across 8 GPUs.