L40S vs H100: which GPU fits your workload and budget?

The L40S vs H100 comparison is about matching the GPU to the job. The L40S is an Ada Lovelace card with 48 GB of GDDR6 at 864 GB/s on PCIe Gen4, designed for both AI and graphics workloads. The H100 SXM is a Hopper compute accelerator with 80 GB of HBM3 at 3.35 TB/s and NVLink. At OverGPU the L40S costs $890 a month and the H100 $2,190, so the right pick can save you $1,300 a month per GPU, and the wrong one can leave you with a card that does not suit the job.

L40S vs H100: the short answer

Pick the L40S for inference on models that fit in 48 GB, for image and video generation, and for rendering and transcoding. Pick the H100 for training, full fine-tuning, models above 48 GB and LLM serving where bandwidth sets the pace. The H100 has no RT cores and no hardware video encoders, so for rendering and video work the L40S is the better fit despite its lower price.

Memory: what fits on each GPU

Weights take roughly 2 bytes per parameter in BF16, 1 in FP8 and 0.5 in 4-bit formats, and both GPUs support FP8 in hardware. Free memory below is before KV cache and runtime overhead, so keep 10–30% of it in reserve.

Free memory also sets concurrency. Take a 14B model with grouped-query attention in a common layout (48 layers, 8 KV heads of dimension 128): its KV cache needs about 0.2 MB per token in BF16. After runtime overhead, roughly 14 GB remain for the cache on the L40S, about 70,000 tokens or eight conversations of 8K tokens at once. The H100 keeps roughly 44 GB, about 220,000 tokens or some 27 such conversations.

Image and video models are usually smaller than LLMs, but high resolutions and long clips need memory for activations. Image models such as SDXL and Flux-class models fit in 48 GB with their text encoders, while the largest video models can need more. If you need more than 48 GB without moving to Hopper, the RTX PRO 6000 with 96 GB of GDDR7 at 1.6 TB/s costs $1,390 a month; see RTX PRO 6000 vs H100. The VRAM calculator runs these numbers for your model.

Bandwidth and compute: why the H100 costs more

The H100 has about 3.9 times the L40S's memory bandwidth (3.35 TB/s against 864 GB/s) and, per NVIDIA, about 2.7 times its dense FP8 and BF16 throughput (1,979 against 733 TFLOPS in FP8, 989 against 362 TFLOPS in BF16). For LLM serving the bandwidth gap matters most, because generating each token reads the weights and KV cache from memory. A 14B model in BF16 runs on both GPUs, but on the H100 each decoding step reads memory at close to four times the rate, which leaves room for many more concurrent users per GPU.

Diffusion models for images and video are more compute-bound, so there the relevant gap is closer to the compute ratio than to the bandwidth ratio. Per $1,000 of monthly spend, the two GPUs compare like this:

• Memory About 54 GB on the L40S against 37 GB on the H100.
• Bandwidth About 0.97 TB/s against 1.53 TB/s.
• Dense FP8, vendor-listed About 824 against 904 TFLOPS.

Of the two, the L40S buys more memory per dollar, the H100 more bandwidth per dollar, and compute per dollar is close.

Media, rendering and video: where the L40S wins

GPU renderers that use NVIDIA's ray-tracing hardware need RT cores, and video pipelines that encode on the GPU need NVENC. The L40S has both, and its Ada-generation encoders handle AV1 as well as H.264 and HEVC; the H100 has neither RT cores nor NVENC, because it is built as a compute accelerator. For a render node, a transcoding service or a generative video pipeline that ends in encoded output, the L40S is the right choice, and two L40S servers cost $1,780 a month, less than one H100. See rendering and VFX and generative media for suggested configurations. For heavier scenes, the 96 GB RTX PRO 6000 is compared in RTX PRO 6000 vs L40S.

Training, fine-tuning and multi-GPU

For training, the H100 is in a different class. HBM3 keeps its tensor cores fed, 80 GB holds larger models and batches, and in an 8-GPU server NVLink 4 links the GPUs at 900 GB/s each. The L40S communicates over PCIe Gen4 with no NVLink, and we offer it as a single-GPU server. It handles LoRA or QLoRA on 7B to 14B models and small experiments well. QLoRA on a 70B model, about 35 GB of 4-bit weights, can squeeze into 48 GB with small batches, but an H100 runs it with far more headroom. Full fine-tuning with Adam needs roughly 16–20 bytes per parameter before activations, which caps a 48 GB card at about 2–3B parameters, against about 4–5B on one H100 and far more on an 8× H100 server at $16,900 a month.

Cost per month and how to decide

One L40S costs $890 a month, about $1.22 per GPU-hour over 730 hours, with 16 vCPUs, 128 GB of RAM and 1 TB of NVMe. One H100 costs $2,190, about $3.00 per GPU-hour, with 24 vCPUs, 225 GB of RAM and 2 TB of NVMe.

A worked example: you serve a 14B model in BF16 to steady traffic. Two L40S servers ($1,780) give you two independent replicas and redundancy if one needs maintenance. One H100 ($2,190) gives you a single replica with more bandwidth than both L40S cards combined (3.35 TB/s against 1.73 TB/s). If redundancy matters most, take the pair; if per-replica capacity and response time matter most, take the H100. For serving setups, see dedicated inference servers.

1. Rendering, transcoding or video output: L40S.
2. Model plus KV cache under about 40 GB: L40S, unless you need high concurrency on one replica.
3. Between 40 and 70 GB, or bandwidth-bound serving: H100.
4. Training, full fine-tuning or multi-GPU models: H100, often as an 8-GPU server.

If you are unsure, start small. Terms are monthly and you can cancel any month without a fee, so you can run real traffic on an L40S and move to an H100 if the numbers call for it.