AMD MI355X for AI: specs, ROCm and the workloads it suits

The case for the AMD MI355X for AI workloads fits in one line: 288 GB of HBM3e per GPU at 8 TB/s, the same memory capacity as NVIDIA's B300. In our catalog, an 8× MI355X server costs about half as much per month as an 8× B300 server with the same 2,304 GB of GPU memory. The real questions are about software and fit: what runs on ROCm today, what needs porting, and whether your workload is the kind that turns memory and bandwidth into results. This guide answers both, with numbers.

AMD MI355X for AI: the specs that matter

The MI355X is AMD's CDNA 4 data-center GPU. AMD lists the following per GPU:

• Architecture AMD CDNA 4
• Memory 288 GB HBM3e per GPU, 2,304 GB in an 8-GPU server
• Memory bandwidth 8 TB/s per GPU
• FP8 5.0 PFLOPS dense, 10.1 PFLOPS with sparsity
• BF16 2.5 PFLOPS dense, 5.0 PFLOPS with sparsity
• MXFP4 10.1 PFLOPS
• Low-precision formats FP8, MXFP6 and MXFP4
• GPU-to-GPU links Infinity Fabric, 7 links of 153.6 GB/s each
• Module and power OAM form factor, 1,400 W, liquid-cooled

Two of these lines matter more than the rest for most AI work. Capacity decides how many GPUs a model needs: 288 GB holds a 70B model in BF16 with room to spare, which on 80 GB GPUs takes at least two cards and tensor parallelism. Bandwidth decides how fast the GPU can stream weights while generating tokens, which is usually the bottleneck in inference. The 1,400 W, liquid-cooled OAM module (the Open Compute Project's accelerator form factor) is the provider's concern rather than yours, but it explains why these servers need liquid-cooled racks.

In an 8-GPU server, the seven Infinity Fabric links connect each GPU directly to each of the other seven: a full mesh, with no switch in between.

How the MI355X compares with B200, B300 and H200

Vendor-published figures per GPU, plus our monthly price for the 8-GPU server of each:

Three honest readings of this table. On memory and bandwidth, the MI355X matches the B300 at about half the monthly price in our catalog, and that is the core of its case. On peak compute, the FP8 figures are close and the B300 leads at 4-bit; in practice, software and kernel quality decide real throughput, so do not choose on PFLOPS alone. On GPU-to-GPU bandwidth and software maturity, NVIDIA's HGX servers lead: NVLink 5 gives each GPU 1.8 TB/s through a switched fabric, and CUDA has the broadest library ecosystem. For head-to-head detail, see our MI355X vs B200 comparison and the MI355X vs H200 comparison.

FP8, MXFP6 and MXFP4 on CDNA 4

Alongside FP8, CDNA 4 supports two block-scaled formats from the Open Compute Project's Microscaling (MX) specification:

• FP8 stores one byte per value. It is the usual next step down from BF16 for inference and halves weight memory.
• MXFP6 stores 6-bit values in blocks of 32 that share one power-of-two scale: about 6.25 bits per value, scale included.
• MXFP4 does the same with 4-bit values: about 4.25 bits per value.

The memory effect is easy to work out. A 405B-parameter dense model needs about 810 GB in BF16: four MI355X GPUs with headroom. In FP8 it needs about 405 GB: two GPUs. In MXFP4 it needs about 215 GB: a single MI355X, with roughly 70 GB left for the KV cache and runtime.

Two caveats. First, 4-bit weights change model quality, so validate on your own evaluation set. Second, the format must match the hardware: a checkpoint quantized to NVIDIA's NVFP4 (16-value blocks with FP8 scales) is not an MXFP4 checkpoint, so plan to quantize for the MI355X, for example with AMD's Quark toolkit. Our explainer on FP8, FP4 and BF16 number formats covers the details.

The ROCm software stack in practice

ROCm is AMD's open software stack: kernel driver, the HIP programming model and compilers, math libraries, and RCCL for collective communication between GPUs. For most teams the useful question is not "is ROCm mature?" but "do my framework and serving engine run on it, and do the features I use work?". The main pieces do run.

In practice, porting effort depends on how close you stay to the mainstream. A stack built on PyTorch plus vLLM or SGLang, serving a standard model architecture, mostly moves by switching containers and pinning versions. A stack with hand-written CUDA kernels needs engineering time: HIP is deliberately close to CUDA and HIPIFY handles much of the translation mechanically, but performance-critical kernels usually need tuning for the new hardware. For first steps on our AMD servers, see our ROCm setup guide.

Workloads where the MI355X pays off

Memory-bound inference of very large models

During decode, every step streams the active weights from memory to generate one token per sequence, so bandwidth sets the pace and capacity sets how many GPUs a model needs. At 8 TB/s and 288 GB per GPU, the MI355X runs large models on fewer GPUs, with less traffic between them. A 70B model in BF16 (about 140 GB) fits on one MI355X with well over 100 GB left for the KV cache, so one server can run eight independent replicas without any tensor parallelism.

Long contexts

The KV cache grows linearly with context length and with the number of concurrent sequences. For a 70B-class model with grouped-query attention (80 layers, 8 KV heads of 128 dimensions), the BF16 cache takes about 0.33 MB per token, so a single 128K-token sequence needs about 43 GB. Next to the weights, that does not fit on an 80 GB GPU at all; on a 288 GB GPU with FP8 weights, there is room for several such sequences.

Large mixture-of-experts models

MoE models activate only a fraction of their parameters per token, but every expert must sit in memory. DeepSeek-V3 and DeepSeek-R1 have 671B parameters in total: about 671 GB in FP8, or about 1.3 TB in BF16. Both fit within the 2,304 GB of a single 8× MI355X server, with room left for the KV cache. Our guide to DeepSeek R1 hardware requirements has the full math.

When an NVIDIA server is the better choice

The MI355X is not the right answer for everyone. Choose an NVIDIA server when:

• Your pipeline depends on CUDA-only components, such as TensorRT-LLM, custom CUDA kernels or closed-source CUDA libraries, and you cannot spare the time to port them.
• Your workload leans hard on GPU-to-GPU bandwidth, such as heavy tensor parallelism in training. HGX B200 and B300 servers give each GPU 1.8 TB/s of NVLink 5 bandwidth.
• You need an NVLink domain larger than one server, which rack-scale systems such as the GB300 NVL72 provide.
• Your models are small. A 7B to 14B model does not need 288 GB per GPU; a single RTX PRO 6000 at $1,390 per month or an L40S at $890 serves it.
• You rely on NVFP4 checkpoints or other NVIDIA-specific quantized formats.

How to evaluate the MI355X: run your own workload for a month

Published comparisons rarely match your model, prompt mix and latency target. The reliable test is your own workload on the hardware, for long enough to tune it, and a month gives you time to set up, tune and measure under real load. A practical plan:

1. Baseline first. Record quality on your evaluation set and throughput at your latency target on the hardware you use today, with the same model, precision and engine settings.
2. Bring up the stack. Start from a ROCm image, the matching PyTorch build and the ROCm container of vLLM or SGLang. Pin every version.
3. Check correctness. Compare outputs on your evaluation set before you measure speed.
4. Measure what you sell. Time to first token, inter-token latency and throughput at your real concurrency and context lengths. Compare tensor parallelism across all eight GPUs with several smaller replicas.
5. Test collectives if you train or shard models: run an all-reduce benchmark from rccl-tests across the eight GPUs.
6. Compute cost per unit of work. Divide $19,900 by the tokens or jobs per month you can serve at your target, and compare with your baseline.

# Check the GPUs, and that PyTorch is a ROCm build (torch.version.hip is set)
rocm-smi
python3 -c "import torch; print(torch.version.hip, torch.cuda.device_count())"

# Serve a model across all eight GPUs with the ROCm build of vLLM
vllm serve <model> --tensor-parallel-size 8

Our 8× MI355X server

We rent the MI355X as a bare-metal 8-GPU server: 8 × 288 GB of HBM3e (2,304 GB in total), two AMD EPYC 9575F CPUs with 128 cores in total, 3 TB of DDR5, 8 × 3.84 TB of NVMe and 8 × 400G RoCE networking. It costs $19,900 per month: about $3.41 per GPU-hour, or about $8.64 per GB of GPU memory per month, the lowest of any configuration in our catalog. It ships with ROCm images and root access over SSH, on the same terms as every server we rent: monthly billing, a one-month minimum, cancellation in any month with no fee, and delivery in under 10 minutes after payment. Stock is limited; the AMD MI355X server page shows current availability.

Frequently asked questions

Is the AMD MI355X good for AI?

Yes, especially for memory-bound inference of large models, long contexts and large mixture-of-experts models, thanks to 288 GB of HBM3e and 8 TB/s per GPU. The trade-off is software: your stack must run on ROCm.

Does PyTorch run on the MI355X?

Yes. PyTorch publishes ROCm builds, and code written against the torch.cuda API generally runs through HIP without changes. vLLM and SGLang also support ROCm.

Can I run CUDA code on AMD GPUs?

Not directly. CUDA kernels must be ported to HIP, which AMD's HIPIFY tools partly automate, and performance-critical kernels usually need retuning. NVIDIA-only software such as TensorRT-LLM does not run on ROCm.

How much memory does an 8× MI355X server have?

2,304 GB of HBM3e, with 288 GB on each of the eight GPUs. That is enough for DeepSeek-V3-class models in FP8 or BF16 on a single server.

How much does it cost to rent an MI355X server?

Our 8× MI355X server costs $19,900 per month, about $3.41 per GPU-hour, billed monthly with a one-month minimum. Stock is limited.