H100 SXM
An H100 SXM runs 34 of the 34 open models tracked here at Q4_K_M with 8k context, computed on 16 October 2026. The largest is Mistral Small 4 119B (MoE), needing 72.5 GB and generating at most 622 tokens per second.
- Memory
- 80 GB VRAM
- Bandwidth
- 3,350 GB/s
- Board power
- 700 W
- Released
- 2022
- Availability
- Rented, not bought
Open the H100 SXM in the LLM checker
What an H100 SXM runs, model by model
H100 SXM 80 GB addresses 80 GB at 3,350 GB/s. Every model below is at Q4_K_M with 8k of context, the setting most people actually use. Once the 5% safety margin comes off, the card has 76 GB to spend: memory decides what loads, bandwidth how fast it answers. Both are estimates from stated formulas, and the VRAM calculator shows the working.
On this card a model fits at or under 76 GB, and above 68 GB it is tight: it loads, but there is little left for the conversation, and a desktop on the same 80 GB can push it over. Tokens per second are single-stream estimates from memory bandwidth; ≤ marks a ceiling real runtimes fall well below.
Computed 16 October 2026The model to actually run on it
Llama 3.3 70B is the best use of this card: 45.8 GB of the 80 GB available, an estimated 57 tokens per second, comfortable for chat, and 100,352 tokens of context still available.
How this is chosen, since no benchmark is quoted: models are compared by size, a mixture-of-experts model counting at the geometric mean of its total and active parameters (a rule of thumb, not a measurement). The pick is the biggest class that fits with memory to spare and answers at 25 tokens per second or better, and within that class the most recent general-purpose release; coding and reasoning specialists are listed above but not recommended by default.
Mistral Small 4 119B (MoE) is the largest model the card will hold, at 72.5 GB, with room for 131,072 tokens of context even at 91% of memory, because its attention design caches so little per token. It is a mixture-of-experts model reading only 6.5B parameters per token, so it is the faster of the two at an estimated 622 tokens per second; what it gives up is the depth of a dense model that reads all of its weights for every token.
If quality matters more than parameter count, Muse Glimmer 30B fits at Q8_0 in about 33.5 GB. Q8 is near-lossless; a smaller model at Q8 often beats a larger one squeezed into Q3.
How much context actually fits
Model size is the question people ask; context is the one that bites. The KV cache grows linearly with the conversation, so a model that loads comfortably can still run out of memory halfway through a long document. On 80 GB, the context you get is whatever the weights leave behind, which is why two cards that run the same model can feel completely different in use. These are the largest contexts this card holds, at Q4_K_M weights.
| Model | Max context, FP16 KV | With Q8 KV cache |
|---|---|---|
| Llama 3.3 70B | 100,352 tokens | 131,072 tokens |
| Qwen3 32B | 40,960 tokens · model limit | 40,960 tokens · model limit |
| Gemma 4 31B | 131,072 tokens | 131,072 tokens |
| Muse Glimmer 30B | 131,072 tokens | 131,072 tokens |
Quantising the KV cache to Q8 roughly doubles what you can hold, at a quality cost most people never notice: Llama 3.3 70B on this card goes from 100,352 to 131,072 tokens. Capped at 128k tokens, or at the model's own native window where that is smaller, marked "model limit": free memory beyond that point buys nothing. Rows that hold far more context than their size suggests are hybrid, sliding-window or latent-attention models, which cache only a fraction of what a standard transformer does; each model page shows the working.
Where this card stops
Nothing in this list is out of reach. Every model tracked here fits at Q4_K_M with 8k of context, so the ceiling you will meet is context length and batch size, not parameter count.
Buy or rent
Rent, and only when serving many users at once or running something genuinely time-critical. For one person, an A100 or a 48 GB card delivers the same experience for much less.
There is no purchase price to reason about, only an hourly one, and it differs by provider and by week. Put the figure you are actually looking at, with your electricity rate and your real monthly hours, into the build vs rent calculator, which already knows this card draws 700 W under load. It returns the month owning becomes cheaper, or tells you it never does. Hours per month decides it far more often than the price does.
Rented, an H100 SXM costs from $2.69 an hour (RunPod Community, third-party host), or $3.49 from a provider that runs the machine (Jarvislabs), on demand, before tax, read 19 October 2026. Every provider's price is in the cloud GPU price guide.
The honest take
The fastest memory here by a wide margin, and almost always more than one person needs.
At 3,350 GB/s the H100 SXM generates text faster than anything else on this page, roughly three and a third times a 4090 on the same model. That speed is designed to be amortised across many simultaneous users, which is why it is the backbone of inference providers and rarely the right answer for a single conversation. For one person chatting, most of what you rent goes unused.
What it is good at
- 3,350 GB/s of HBM3: even a 70B model answers faster than you can read.
- FP8 support, which newer runtimes use to cut memory and increase throughput.
- Where batching matters, nothing else on this list is close.
What it is not
- 700 W and an SXM baseboard: it does not go in a PCIe slot at all.
- The most expensive class to rent, for speed a single user cannot consume.
- 80 GB is less memory than an RTX PRO 6000, despite costing far more.
The thing people get wrong: The SXM version is not a card you can install. It mounts on an OEM baseboard, so "buying an H100" means buying a server. The PCIe variant exists and is meaningfully slower.
Compared with the alternatives
- A100: 80 GB · 2,039 GB/s · best fit Llama 3.3 70B at ~35 tok/s. The old datacenter workhorse, still fast where it counts, and cheap to rent.
- RTX PRO 6000 Blackwell: 96 GB · 1,792 GB/s · best fit Llama 3.3 70B at ~31 tok/s. Ninety-six gigabytes and 5090-class bandwidth on one card.
- L40S: 48 GB · 864 GB/s · best fit Muse Glimmer 30B at ~35 tok/s. The 48 GB card you meet as a rental line item, not as a purchase.
All 39 cards are compared side by side on the GPU index, and the same numbers from the model's point of view are on each model fit table. To check one specific pairing rather than read a table, use the can-I-run-it checker.
Questions
- Can an H100 SXM run a 70B model?
- Yes. Llama 3.3 70B at Q4_K_M with 8k of context needs about 45.8 GB, inside the 80 GB this card can address, and it generates an estimated 57 tokens per second, which is comfortable for chat.
- What is the best model to run on an H100 SXM?
- Llama 3.3 70B. At Q4_K_M and 8k context it needs about 45.8 GB of the 80 GB available and generates an estimated 57 tokens per second, which is comfortable for chat. If quality matters more than size, Muse Glimmer 30B fits at Q8 in about 33.5 GB.
- How many tokens per second does an H100 SXM generate?
- It depends entirely on the size of the model, because generating a token means reading every active weight out of memory. At 3,350 GB/s and 70% efficiency, this card produces an estimated 505 tokens per second on an 8B model at Q4, and about 622 on the largest model it holds, Mistral Small 4 119B (MoE). Reading speed is roughly 5 to 8 words per second, so anything above 25 feels immediate. Treat 505 as a ceiling on 3,350 GB/s rather than a measurement; the higher a figure, the further real runtimes fall below it.
- Is 80 GB enough for running LLMs locally?
- It runs 34 of the 34 models tracked here at Q4_K_M with 8k of context, up to 119B parameters. The honest test is not the model list but the context: Llama 3.3 70B on this card holds about 100,352 tokens before memory runs out.
- H100 SXM or A100 for local models?
- Both address about the same memory, so they run the same models. On speed, this card is faster: 3,350 GB/s against 2,039 GB/s, and bandwidth is what sets chat speed.
- Can I buy an H100 for a workstation?
- Not the SXM version, which is what most H100 references mean. It mounts on an OEM baseboard rather than a PCIe slot, so buying one means buying a server built around it, at 700 W per card. A PCIe variant exists and is meaningfully slower.
- Is an H100 overkill for one person?
- Almost always. Its 3,350 GB/s is designed to be shared across many simultaneous requests, and a single conversation cannot consume that. For one person chatting with a large model, an A100 or a 48 GB card delivers a very similar experience for a fraction of the rental cost.
- Memory: weights (parameters × bytes per parameter) + KV cache (2 × layers × KV heads × head dim × context × bytes for a standard transformer; sliding-window, hybrid and latent-attention models counted as they actually cache) + 0.5 GB runtime + 4% of weights. Full derivation in the VRAM calculator.
- Speed: 70% × bandwidth ÷ active weight bytes, single stream, no batching, prompt processing excluded. A ceiling, not a measurement: the 32 of 34 figures marked ≤ are where real runtimes fall furthest below the number, because so few weights are read per token. See the tokens-per-second estimator.
- Card specification (manufacturer figures): 80 GB, 3,350 GB/s, 700 W; see NVIDIA's own page for this card.
- Hourly prices: each provider's own pricing page or price API, read 19 October 2026, as listed in the cloud GPU price guide.
- Model architecture values come from each model's published config; the self-hosted LLM guide explains what each one changes.