MI355X vs H200: double the memory or the proven stack?

The MI355X vs H200 decision is a trade between hardware and habit. AMD's Instinct MI355X has 288 GB of HBM3e per GPU, about twice the H200's 141 GB, with 8 TB/s of memory bandwidth against 4.8 TB/s and hardware support for FP6 and FP4. NVIDIA's H200 keeps the Hopper architecture of the H100 but moves to larger, faster HBM3e, and runs the CUDA stack most AI code is written for. We rent both, so this page sticks to what you can check before ordering: memory, formats, interconnect, price and porting effort.

MI355X vs H200 at a glance

On paper, the MI355X leads on every hardware line. AMD lists 5.0 PFLOPS of dense FP8 for the MI355X, about 2.5 times the 1,979 TFLOPS NVIDIA lists for the H200. It also has about twice the memory and about 1.7 times the bandwidth. The H200's strengths are elsewhere: CUDA compatibility, years of production use of Hopper across the industry, and a single-GPU plan.

Price favors the MI355X as well. Our 8× MI355X server is $19,900 a month and our HGX H200 server is $23,900: $4,000 less for twice the memory. If your software runs on ROCm, the MI355X is the stronger server. If it does not, its hardware lead does you no good.

Memory: 288 GB vs 141 GB per GPU

Capacity decides how many GPUs a model needs and how much room is left for KV cache. Weights take roughly 2 bytes per parameter in BF16, 1 in FP8 and 0.5 in FP4; plan 10–30% more for KV cache, activations and overhead. The table counts weights only.

The last two lines show the clearest split. DeepSeek-V3, a 671B mixture-of-experts model, was released with FP8 weights. Both 8-GPU servers hold them, but the MI355X server leaves about 1.6 TB for KV cache against about 450 GB on the HGX H200. In BF16, only the MI355X server holds the model at all, with roughly 960 GB to spare. Our DeepSeek hardware guide works through this case in detail.

Long contexts show the same gap on one GPU. A 70B-class model with grouped-query attention (80 layers and 8 KV heads, as in Llama 3 70B) needs about 0.16 MB of KV cache per token in FP8, so one 128k-token context takes about 21 GB. With FP8 weights loaded, an H200 has roughly 70 GB left: about three full-length contexts at once. An MI355X has roughly 218 GB left: about ten, before overhead.

Compute and precision formats

AMD lists the MI355X at 2.5 PFLOPS of dense BF16 and 5.0 PFLOPS of dense FP8. NVIDIA lists the H200 at 989 and 1,979 TFLOPS for the same formats, the same figures as the H100, since both use the same Hopper compute. Roughly 2.5 times the peak math is a generational gap, not a tuning difference. Peak figures are still not delivered throughput: your kernels and libraries decide how much of it you see.

The wider gap is at low precision. Hopper's lowest-precision floating-point tensor format is FP8. AMD's CDNA 4 adds MXFP6 and MXFP4, the microscaling formats standardized through the Open Compute Project, and AMD lists 10.1 PFLOPS for MXFP4. You can still serve 4-bit weights on an H200 with weight-only quantization such as AWQ or GPTQ, but the matrix math runs at higher precision: memory use drops, compute does not rise. On the MI355X, 4-bit and 6-bit models can run their math natively where your framework supports it. Our overview of the AMD MI355X for AI goes further.

Memory bandwidth and interconnect

Each generated token requires reading the active weights and the KV cache from memory, so bandwidth often sets the pace of interactive serving at small batch sizes. AMD lists 8 TB/s for the MI355X and NVIDIA lists 4.8 TB/s for the H200: on a model that fits both, the MI355X streams that data at about 1.7 times the rate.

Inside the server, the HGX H200 connects every GPU through NVLink switches with 900 GB/s of NVLink 4 bandwidth each, so any pair communicates at full NVLink speed. The MI355X uses a direct Infinity Fabric mesh: AMD lists seven links of 153.6 GB/s per GPU, one to each peer, about 1.08 TB/s in total. All-reduce and other collectives spread across every link; a transfer between two specific GPUs uses the single link between them. For training with heavy tensor parallelism, profile both. For inference, the MI355X's capacity often means splitting a model across fewer GPUs, which cuts communication at the source.

Between servers, both have eight 400G network ports: ConnectX-7 on the HGX H200 and RoCE on the MI355X. Private networking between your servers is available on request.

Monthly cost: $19,900 vs $23,900

Our 8× MI355X server works out to about $3.41 per GPU-hour and our HGX H200 to about $4.09. Per gigabyte of GPU memory, that is about $8.64 a month against $21.19: roughly 2.5 times as much memory per dollar on the MI355X.

The H200 wins on granularity. A single H200 costs $3,090 a month and holds a 70B model in FP8 with room for KV cache. If one GPU covers your workload, there is no MI355X plan to compare it with, because we rent the MI355X only as a full 8-GPU server. Two or three single H200s also stay under $10,000 a month, though as separate virtual machines they suit independent jobs rather than one model split across GPUs.

MI355X stock is limited, but the server is still delivered in under 10 minutes after payment. Once you have an active server, you can ask by ticket for pricing on 6- to 12-month terms or several servers. Model your own totals in the GPU cost calculator.

Software: the Hopper stack vs ROCm

The H200 runs the same software as the H100: same CUDA builds, same kernels, same container images. If your training or serving stack works on Hopper today, it works on an H200 with more memory and no porting. When a deadline is close, that predictability has real value.

The MI355X runs ROCm. PyTorch and vLLM support it, and our ROCm images give you a ready base. The friction sits in custom CUDA kernels, which need porting to HIP, and in NVIDIA-only tools such as TensorRT-LLM, which have no AMD version. Audit your dependencies, then run your real workload for a month: the minimum term is one month and you can cancel any month with no fee. Our ROCm documentation covers the setup.

How to choose

• One GPU, 70B in FP8 or smaller: a single H200 at $3,090 a month.
• Eight GPUs, ROCm-compatible stack: the MI355X server at $19,900, for more memory, more bandwidth and native FP4.
• Eight GPUs, CUDA-only dependencies: the HGX H200 at $23,900, or a Blackwell server if you also need FP4 (see H200 vs B200).
• Unsure whether your stack ports: test on the MI355X for one month before committing to a longer term.