A100 vs H100: what changed, and why we rent the H100

The A100 vs H100 comparison comes up because both GPUs have 80 GB of memory, which makes them look interchangeable on paper. They are not. To be clear from the start: OverGPU does not rent the A100. We offer the H100, as a single GPU or in an 8-GPU HGX server, along with newer Hopper and Blackwell GPUs. This page explains what changed between Ampere and Hopper, so you can judge whether an A100 you already own, or can get elsewhere, is still the right tool for your work.

A100 vs H100: the short answer

The H100 keeps the same 80 GB and improves almost everything else: FP8 support, 64% more memory bandwidth (3.35 TB/s against about 2.0 TB/s), about 3.2 times the dense BF16 throughput NVIDIA lists (989 against 312 TFLOPS) and NVLink 4 at 900 GB/s per GPU against 600 GB/s. The A100 still makes sense where you already have it and your jobs fit its limits. For a new rental in 2026, the H100 is the GPU we offer and the one we recommend.

FP8: the biggest practical difference

The A100 (Ampere) has no FP8 support. Its fastest floating-point format for training is BF16 or FP16, at 312 TFLOPS dense per NVIDIA. The H100 (Hopper) introduced FP8 tensor cores and the Transformer Engine, with 1,979 TFLOPS of dense FP8 listed by NVIDIA.

FP8 helps twice. It halves weight memory compared with BF16, so a 70B model drops from about 140 GB to about 70 GB and fits on one 80 GB H100, tightly. And the math itself runs in FP8 on the tensor cores. On an A100, the same model needs INT8 or 4-bit weight-only quantization, or two GPUs. FP8 checkpoints are now common for open models, and Hopper runs them natively. Our FP8 and FP4 explainer covers the formats and their accuracy trade-offs.

Same capacity, different options. Approximate weights for common model sizes:

Memory bandwidth and inference

Both cards hold 80 GB, but the H100 uses HBM3 at 3.35 TB/s while the A100 80 GB uses HBM2e at about 2.0 TB/s. When a model generates text, each new token requires reading the weights and the KV cache from memory, so at small and medium batch sizes decoding follows bandwidth more than compute. With 64% more bandwidth, the H100 reads the same model in less time at every step, which means shorter waits per token and more users per GPU.

That gives a simple way to judge value if you can rent A100 time elsewhere. On bandwidth-bound inference, an A100 is only the cheaper option if its hourly cost is below roughly 60% of an H100's (2.0 ÷ 3.35). On compute-bound BF16 training, the bar drops to roughly a third (312 ÷ 989), and lower again if your H100 job can use FP8. Real jobs rarely scale exactly with vendor ratios, so treat these thresholds as rough guides and measure your own workload.

Hopper also added features that kernel authors rely on: the Tensor Memory Accelerator for asynchronous data movement, thread block clusters and DPX instructions for dynamic-programming algorithms such as sequence alignment. Attention kernels tuned for Hopper, such as FlashAttention-3, use them and do not run on the A100.

NVLink and multi-GPU scaling

In 8-GPU HGX systems, the A100 uses third-generation NVLink at 600 GB/s per GPU and the H100 fourth-generation NVLink at 900 GB/s. Tensor-parallel inference and training with sharded optimizer states exchange data between GPUs constantly, so 50% more link bandwidth leaves more of each step for computation. Our 8-GPU H100 servers pair eight H100s with two Xeon 8480+ CPUs (112 cores), 2 TB of DDR5, 8× 3.84 TB NVMe and 8× 400G ConnectX-7 networking, for $16,900 a month.

On one 8-GPU server, memory sets the ceiling for full fine-tuning. With Adam at roughly 16–20 bytes per parameter before activations, 640 GB of GPU memory holds a full fine-tune of about a 30B model, on A100s or H100s alike. The H100 server simply works through each step with more compute and faster links. For serving, the same 640 GB holds a 405B model in 8 bits, about 405 GB of weights: natively in FP8 on H100s, through INT8 on A100s.

Who might still use A100s

The A100 remains a capable GPU, and there are honest reasons to keep using one:

• You already own or contract them. Hardware you have paid for is cheap to keep running for jobs that fit.
• Reproducibility. Results validated on Ampere, for papers or regulated pipelines, may need identical hardware.
• Small, undemanding jobs. Classic computer vision, small language models or batch jobs that already finish in time do not need Hopper.
• Power limits. At 400 W against 700 W per GPU, the A100 fits facilities with tight power budgets, even though the H100 does about 1.8 times more dense BF16 work per watt on NVIDIA's figures.
• Short, occasional jobs. If you need a GPU for a few hours a month, renting by the hour elsewhere, A100 or not, can cost less than any monthly server; our monthly vs hourly comparison shows the break-even.

For a new monthly rental, these reasons rarely outweigh FP8 and the bandwidth gain. If 80 GB is not enough, our H100 vs H200 comparison shows what 141 GB adds; if your models are small, the L40S vs H100 comparison covers a lower-cost option at $890 a month.

H100 cost and moving from A100

A single H100 at OverGPU costs $2,190 a month, about $3.00 per GPU-hour over a 730-hour month, with 24 vCPUs, 225 GB of RAM, 2 TB of NVMe and 25 Gbps networking. The 8-GPU HGX H100 server is $16,900, about $2.89 per GPU-hour. Moving from A100 to H100 rarely needs code changes, since CUDA, PyTorch and the main inference engines support both, but a short checklist helps:

• Use a CUDA 12 build of your framework. The H100 is compute capability 9.0, against 8.0 for the A100.
• Rebuild custom CUDA extensions for compute capability 9.0 if they were compiled only for Ampere.
• Try FP8 weights and an FP8 KV cache in your inference engine, and compare outputs on your evaluation set before switching.
• Re-tune batch sizes: with more bandwidth and compute, the best setting usually changes.

Our PyTorch and vLLM guide covers setup on our images, which come with NVIDIA drivers, CUDA, Docker and the NVIDIA Container Toolkit preinstalled.