Which GPU do you need to run LLMs locally?
Memory decides what runs: at Q4_K_M with 8k context, a 24 GB card runs 28 of the 34 open models tracked here and a 96 GB card all 34, computed on 16 October 2026. Bandwidth decides how fast. Below, the fastest card at each memory size, then all 39 compared.
- Cards
- 39
- Models
- 34
- Setting
- Q4_K_M, 8k context
- Computed
- 16 October 2026
The pick at each memory size
Cards with the same memory run exactly the same models: the fit depends on memory alone. So at each size the pick is the card that answers fastest, the one with the most memory bandwidth, among the cards you can put on a desk. Data-centre cards are left out: you rent them. The sizes are the ones the LLM checker and the model pages use.
No pick at 8 GB: no card here has that size. At 80 GB there are only the A100 and H100 SXM, data-centre cards you rent: their hourly prices. The RX 7900 XT (20 GB) sits between two sizes. Above 96 GB every machine here runs all 34 models, so more memory buys longer context and higher precision, not more models: those are the unified-memory machines and Macs below. No pick by price: shop prices move weekly, so this page carries none, and the build vs rent calculator takes yours.
Computed 16 October 2026The only two numbers that matter
Already know which card and which model? The can-I-run-it checker answers that pairing directly. Every card on this page is measured the same way: each of the 34 open models tracked on this site, at Q4_K_M with 8k of context, against the memory the card can actually address. Nothing here is benchmarked: memory comes from a stated formula, speed from memory bandwidth, and both say so.
Memory decides what you can run. A model has to fit, in full, along with its KV cache. If it does not fit, the answer is not "slower", it is ten to fifty times slower, because the shortfall spills to system RAM. Fit is binary, and it is the first filter.
Bandwidth decides how fast it answers. Generating one token means reading every active weight out of memory, so the ceiling is bandwidth divided by weight size. This is why a 1008 GB/s card is only about 7% faster than a 936 GB/s one for chat, whatever the rest of the specification says, and why the 16 GB RTX 4060 Ti at 288 GB/s generates more slowly than the cheaper 12 GB RTX 3060.
Compute (cores, TFLOPs, the number in most reviews) barely affects chat speed. It matters for prompt processing, for image and video generation, and for serving many people at once. For one person typing questions at a model, it is close to irrelevant.
Every card, compared
| Card | Usable | Bandwidth | Runs | Largest model it holds |
|---|---|---|---|---|
| RTX 3060 | 12 GB | 360 GB/s | 11/34 | Qwen3 14B · 29 tok/s |
| Arc B580 | 12 GB | 456 GB/s | 11/34 | Qwen3 14B · 37 tok/s |
| RTX 4070 | 12 GB | 504 GB/s | 11/34 | Qwen3 14B · 41 tok/s |
| RTX 5070 | 12 GB | 672 GB/s | 11/34 | Qwen3 14B · 55 tok/s |
| RTX 4060 Ti | 16 GB | 288 GB/s | 12/34 | gpt-oss 20B · 97 tok/s |
| RX 9060 XT | 16 GB | 320 GB/s | 12/34 | gpt-oss 20B · 107 tok/s |
| RTX 5060 Ti | 16 GB | 448 GB/s | 12/34 | gpt-oss 20B · 150 tok/s |
| RX 9070 XT | 16 GB | 640 GB/s | 12/34 | gpt-oss 20B · 215 tok/s |
| RTX 4080 SUPER | 16 GB | 736 GB/s | 12/34 | gpt-oss 20B · 247 tok/s |
| RTX 5070 Ti | 16 GB | 896 GB/s | 12/34 | gpt-oss 20B · 300 tok/s |
| RTX 5080 | 16 GB | 960 GB/s | 12/34 | gpt-oss 20B · 322 tok/s |
| RX 7900 XT | 20 GB | 800 GB/s | 19/34 | Muse Glimmer 30B · 32 tok/s |
| RTX PRO 4000 Blackwell SFF | 24 GB | 432 GB/s | 28/34 | Qwen3.6 35B-A3B (MoE) · 174 tok/s |
| Arc Pro B60 | 24 GB | 456 GB/s | 28/34 | Qwen3.6 35B-A3B (MoE) · 183 tok/s |
| RTX PRO 4000 Blackwell | 24 GB | 672 GB/s | 28/34 | Qwen3.6 35B-A3B (MoE) · 270 tok/s |
| RTX 3090 | 24 GB | 936 GB/s | 28/34 | Qwen3.6 35B-A3B (MoE) · 377 tok/s |
| RX 7900 XTX | 24 GB | 960 GB/s | 28/34 | Qwen3.6 35B-A3B (MoE) · 386 tok/s |
| RTX 4090 | 24 GB | 1,008 GB/s | 28/34 | Qwen3.6 35B-A3B (MoE) · 406 tok/s |
| Arc Pro B70 | 32 GB | 608 GB/s | 28/34 | Qwen3.6 35B-A3B (MoE) · 245 tok/s |
| Radeon AI PRO R9700 | 32 GB | 640 GB/s | 28/34 | Qwen3.6 35B-A3B (MoE) · 257 tok/s |
| RTX PRO 4500 Blackwell | 32 GB | 896 GB/s | 28/34 | Qwen3.6 35B-A3B (MoE) · 360 tok/s |
| RTX 5090 | 32 GB | 1,792 GB/s | 28/34 | Qwen3.6 35B-A3B (MoE) · 721 tok/s |
| Mac mini M4 Pro | 48 GB of 64 | 273 GB/s | 28/34 | Qwen3.6 35B-A3B (MoE) · 110 tok/s |
| Mac mini M5 Pro | 48 GB of 64 | 307 GB/s | 28/34 | Qwen3.6 35B-A3B (MoE) · 124 tok/s |
| RTX A6000 | 48 GB | 768 GB/s | 28/34 | Qwen3.6 35B-A3B (MoE) · 309 tok/s |
| L40S | 48 GB | 864 GB/s | 28/34 | Qwen3.6 35B-A3B (MoE) · 348 tok/s |
| RTX 6000 Ada | 48 GB | 960 GB/s | 28/34 | Qwen3.6 35B-A3B (MoE) · 386 tok/s |
| RTX PRO 5000 Blackwell | 48 GB | 1,344 GB/s | 28/34 | Qwen3.6 35B-A3B (MoE) · 541 tok/s |
| A100 | 80 GB | 2,039 GB/s | 34/34 | Mistral Small 4 119B (MoE) · 379 tok/s |
| H100 SXM | 80 GB | 3,350 GB/s | 34/34 | Mistral Small 4 119B (MoE) · 622 tok/s |
| Ryzen AI Max+ 395 | 96 GB of 128 | 256 GB/s | 34/34 | Mistral Small 4 119B (MoE) · 48 tok/s |
| MacBook Pro M4 Max | 96 GB of 128 | 546 GB/s | 34/34 | Mistral Small 4 119B (MoE) · 101 tok/s |
| MacBook Pro M5 Max | 96 GB of 128 | 614 GB/s | 34/34 | Mistral Small 4 119B (MoE) · 114 tok/s |
| RTX PRO 6000 Blackwell | 96 GB | 1,792 GB/s | 34/34 | Mistral Small 4 119B (MoE) · 333 tok/s |
| RTX Spark | 102 GB of 128 | 300 GB/s | 34/34 | Mistral Small 4 119B (MoE) · 56 tok/s |
| DGX Spark | 126 GB of 128 | 273 GB/s | 34/34 | Mistral Small 4 119B (MoE) · 51 tok/s |
| Mac Studio M2 Ultra | 144 GB of 192 | 800 GB/s | 34/34 | Mistral Small 4 119B (MoE) · 149 tok/s |
| Mac Studio M3 Ultra | 384 GB of 512 | 819 GB/s | 34/34 | Mistral Small 4 119B (MoE) · 152 tok/s |
| Mac Studio M5 Ultra | 384 GB of 512 | 1,200 GB/s | 34/34 | Mistral Small 4 119B (MoE) · 223 tok/s |
"Usable" is the memory a runtime can address: full VRAM on a graphics card, and about 75% of system RAM on Apple silicon, which is the macOS default. "Runs" counts models that fit at Q4_K_M with 8k of context, at or under 95% of usable memory.
Computed 16 October 2026Consumer cards
What goes in a desktop you already own. Memory is the ceiling on what you can run; bandwidth is the ceiling on how fast it answers.
- RTX 3060 12 GB · 360 GB/s · 170 W · 2021 · Mostly second-hand
The cheapest card that still makes local models worth doing.
Runs 11 of 34 models · best fit: Gemma 4 12B at ~36 tok/s - Arc B580 12 GB · 456 GB/s · 190 W · 2024 · Sold new
Intel's 12 GB card: faster memory than an RTX 3060 12 GB, and a younger software stack.
Runs 11 of 34 models · best fit: Gemma 4 12B at ~46 tok/s - RTX 4070 12 GB · 504 GB/s · 200 W · 2023 · Sold new
Fast for its class, and capped by 12 GB.
Runs 11 of 34 models · best fit: Gemma 4 12B at ~51 tok/s - RTX 5070 12 GB · 672 GB/s · 250 W · 2025 · Sold new
Fast memory, and too little of it: 12 GB at 672 GB/s.
Runs 11 of 34 models · best fit: Gemma 4 12B at ~68 tok/s - RTX 4060 Ti 16 GB · 288 GB/s · 165 W · 2023 · Sold new
More memory than its neighbours, and less bandwidth than a card two years older.
Runs 12 of 34 models · best fit: Gemma 4 12B at ~29 tok/s - RX 9060 XT 16 GB · 320 GB/s · 160 W · 2025 · Sold new
AMD's 16 GB budget card: the memory is right, the bandwidth is low.
Runs 12 of 34 models · best fit: Gemma 4 12B at ~32 tok/s - RTX 5060 Ti 16 GB · 448 GB/s · 180 W · 2025 · Sold new
The newest 16 GB GeForce below the high end, and a clear step up from the 4060 Ti it replaces.
Runs 12 of 34 models · best fit: Gemma 4 12B at ~45 tok/s - RX 9070 XT 16 GB · 640 GB/s · 304 W · 2025 · Sold new
AMD's RDNA 4 gaming card: 16 GB at 640 GB/s, with ROCm on Linux and Windows, and no CUDA.
Runs 12 of 34 models · best fit: Gemma 4 12B at ~64 tok/s - RTX 4080 SUPER 16 GB · 736 GB/s · 320 W · 2024 · Mostly second-hand
A 16 GB gaming card that a used 3090 beats on both numbers that matter for language models.
Runs 12 of 34 models · best fit: Gemma 4 12B at ~74 tok/s - RTX 5070 Ti 16 GB · 896 GB/s · 300 W · 2025 · Sold new
The 5080's memory bus and 16 GB, read about 7% more slowly.
Runs 12 of 34 models · best fit: Gemma 4 12B at ~90 tok/s - RTX 5080 16 GB · 960 GB/s · 360 W · 2025 · Sold new
A 16 GB card with 3090-class bandwidth: quick on what fits, and it fits no more than a 5060 Ti 16 GB.
Runs 12 of 34 models · best fit: Gemma 4 12B at ~97 tok/s - RX 7900 XT 20 GB · 800 GB/s · 315 W · 2022 · Sold new
Twenty gigabytes at 800 GB/s: the XTX's smaller sibling, with more memory than any 16 GB card.
Runs 19 of 34 models · best fit: Mistral Small 3.1 24B at ~40 tok/s - RTX 3090 24 GB · 936 GB/s · 350 W · 2020 · Mostly second-hand
The value benchmark for local models, and it has been for years.
Runs 28 of 34 models · best fit: Muse Glimmer 30B at ~38 tok/s - RX 7900 XTX 24 GB · 960 GB/s · 355 W · 2022 · Sold new
AMD's 24 GB card: 3090-class memory and bandwidth, without CUDA.
Runs 28 of 34 models · best fit: Muse Glimmer 30B at ~39 tok/s - RTX 4090 24 GB · 1,008 GB/s · 450 W · 2022 · Mostly second-hand
The card most local-model advice is implicitly written for.
Runs 28 of 34 models · best fit: Muse Glimmer 30B at ~41 tok/s - RTX 5090 32 GB · 1,792 GB/s · 575 W · 2025 · Sold new
The first consumer card whose memory bandwidth changes what is comfortable.
Runs 28 of 34 models · best fit: Muse Glimmer 30B at ~73 tok/s
Workstation cards
Large memory in a machine that lives under a desk, at power and noise levels a room can tolerate. You pay for the address space, not the speed.
- RTX PRO 4000 Blackwell SFF 24 GB · 432 GB/s · 70 W · 2025 · Workstation card
Twenty-four gigabytes at 70 W in a small card, the kind built into compact AI boxes.
Runs 28 of 34 models · best fit: Gemma 4 12B at ~43 tok/s - Arc Pro B60 24 GB · 456 GB/s · 200 W · 2025 · Workstation card
Intel's 24 GB workstation card: the B580's bandwidth with twice the memory.
Runs 28 of 34 models · best fit: Gemma 4 12B at ~46 tok/s - RTX PRO 4000 Blackwell 24 GB · 672 GB/s · 145 W · 2025 · Workstation card
Twenty-four gigabytes on a single-slot, 145 W workstation card.
Runs 28 of 34 models · best fit: Muse Glimmer 30B at ~27 tok/s - Arc Pro B70 32 GB · 608 GB/s · 230 W · 2026 · Workstation card
Intel's 32 GB workstation card: RTX 5090 capacity at 608 GB/s, on Intel's software.
Runs 28 of 34 models · best fit: Muse Glimmer 30B at ~25 tok/s - Radeon AI PRO R9700 32 GB · 640 GB/s · 300 W · 2025 · Workstation card
AMD's 32 GB workstation card: RX 9070 XT bandwidth with twice the memory, at 300 W.
Runs 28 of 34 models · best fit: Muse Glimmer 30B at ~26 tok/s - RTX PRO 4500 Blackwell 32 GB · 896 GB/s · 200 W · 2025 · Workstation card
A 5090's 32 GB at half its bandwidth, in a two-slot, 200 W card.
Runs 28 of 34 models · best fit: Muse Glimmer 30B at ~36 tok/s - RTX A6000 48 GB · 768 GB/s · 300 W · 2020 · Workstation card
The 48 GB card you often meet on rental lists: Ampere, 768 GB/s, and slower at reading memory than a 3090.
Runs 28 of 34 models · best fit: Muse Glimmer 30B at ~31 tok/s - RTX 6000 Ada 48 GB · 960 GB/s · 300 W · 2022 · Workstation card
Forty-eight gigabytes in a normal computer, at 300 watts, without the noise.
Runs 28 of 34 models · best fit: Muse Glimmer 30B at ~39 tok/s - RTX PRO 5000 Blackwell 48 GB · 1,344 GB/s · 300 W · 2025 · Workstation card
Forty-eight gigabytes at 1,344 GB/s in a two-slot, 300 W card, with a 72 GB version at the same speed.
Runs 28 of 34 models · best fit: Muse Glimmer 30B at ~54 tok/s - RTX PRO 6000 Blackwell 96 GB · 1,792 GB/s · 600 W · 2025 · Workstation card
Ninety-six gigabytes and 5090-class bandwidth on one card.
Runs 34 of 34 models · best fit: Llama 3.3 70B at ~31 tok/s
Datacenter cards
You meet these as a line on a rental invoice. They are here because they set the reference for what renting buys you.
- L40S 48 GB · 864 GB/s · 350 W · 2023 · Rented, not bought
The 48 GB card you meet as a rental line item, not as a purchase.
Runs 28 of 34 models · best fit: Muse Glimmer 30B at ~35 tok/s - A100 80 GB · 2,039 GB/s · 2020 · Rented, not bought
The old datacenter workhorse, still fast where it counts, and cheap to rent.
Runs 34 of 34 models · best fit: Llama 3.3 70B at ~35 tok/s - H100 SXM 80 GB · 3,350 GB/s · 700 W · 2022 · Rented, not bought
The fastest memory here by a wide margin, and almost always more than one person needs.
Runs 34 of 34 models · best fit: Llama 3.3 70B at ~57 tok/s
Unified memory desktops
An NVIDIA or AMD chip whose GPU shares system memory, so a small box holds models a gaming card cannot. Bandwidth is the catch: they read memory at about a quarter of an RTX 4090's speed.
- Ryzen AI Max+ 395 96 of 128 GB usable · 256 GB/s · 2025 · Sold new
AMD's 128 GB laptop and mini-PC chip: up to 96 GB for the GPU on Windows, and an x86 route to large mixture-of-experts models.
Runs 34 of 34 models · best fit: Nemotron 3.5 Bolt 74B-A9B (MoE) at ~34 tok/s - RTX Spark 102 of 128 GB usable · 300 GB/s · 2026 · Sold new
NVIDIA's chip for Windows laptops and mini PCs: up to 128 GB shared with the GPU, read at 300 GB/s, with CUDA.
Runs 34 of 34 models · best fit: Nemotron 3.5 Bolt 74B-A9B (MoE) at ~40 tok/s - DGX Spark 126 of 128 GB usable · 273 GB/s · 140 W · 2025 · Sold new
NVIDIA's small Linux box with 128 GB the GPU can share: large models fit, but they are read at about a quarter of an RTX 4090's speed.
Runs 34 of 34 models · best fit: Nemotron 3.5 Bolt 74B-A9B (MoE) at ~37 tok/s
Apple unified memory
The GPU addresses most of system RAM, so these hold models no single graphics card can. Generation is respectable; prompt processing is not.
- Mac mini M4 Pro 48 of 64 GB usable · 273 GB/s · 2024 · Apple unified memory
A small, quiet Mac that gives the GPU more memory than any consumer graphics card, read at 273 GB/s.
Runs 28 of 34 models · best fit: Gemma 4 12B at ~27 tok/s - Mac mini M5 Pro 48 of 64 GB usable · 307 GB/s · 2026 · Apple unified memory
Apple's current Mac mini for local models: up to 64 GB at 307 GB/s.
Runs 28 of 34 models · best fit: Gemma 4 12B at ~31 tok/s - MacBook Pro M4 Max 96 of 128 GB usable · 546 GB/s · 2024 · Apple unified memory
A laptop that runs models a desktop GPU cannot hold.
Runs 34 of 34 models · best fit: Nemotron 3.5 Bolt 74B-A9B (MoE) at ~73 tok/s - MacBook Pro M5 Max 96 of 128 GB usable · 614 GB/s · 2026 · Apple unified memory
Apple's 128 GB laptop: large models fit and are read at 614 GB/s, if you buy the 40-core GPU.
Runs 34 of 34 models · best fit: Nemotron 3.5 Bolt 74B-A9B (MoE) at ~82 tok/s - Mac Studio M2 Ultra 144 of 192 GB usable · 800 GB/s · 2023 · Apple unified memory
The quiet way to hold a very large model, as long as you are patient with long prompts.
Runs 34 of 34 models · best fit: Muse Glimmer 30B at ~32 tok/s - Mac Studio M3 Ultra 384 of 512 GB usable · 819 GB/s · 2025 · Apple unified memory
2025's 512 GB Mac Studio: still the way to hold very large models, now mostly second-hand.
Runs 34 of 34 models · best fit: Muse Glimmer 30B at ~33 tok/s - Mac Studio M5 Ultra 384 of 512 GB usable · 1,200 GB/s · 2026 · Apple unified memory
The most memory you can put on a desk: up to 512 GB at 1.2 TB/s, without CUDA.
Runs 34 of 34 models · best fit: Muse Glimmer 30B at ~49 tok/s
How to choose, in one paragraph
Decide the largest model you actually want to run, then buy the memory that holds it, then take whatever bandwidth comes with that memory. Going the other way (picking a fast card and hoping the model fits) is how people end up with a card that runs everything quickly except the thing they bought it for. If you are unsure which model you want, start from the VRAM calculator: it tells you the smallest card each model fits on. If you know the model and want to know how it will feel, the tokens-per-second estimator answers that. If you already know both and the question is whether to buy at all, the build vs rent calculator takes your own hours and electricity price.
And the honest version: most people over-buy. Hours per month decides the economics, and a card that idles is a card that has not paid for itself. Renting the class of card that runs Mistral Small 4 119B (MoE) for a few hours a month costs less than owning almost anything on this page. The calculator tells you when buying wins.
- Card specifications are manufacturer figures: memory capacity, memory bandwidth and board power. Each card's page links the manufacturer's own.
- Memory estimates: weights + KV cache + runtime overhead, per the formula in the VRAM calculator.
- Speed estimates: 70% × bandwidth ÷ active weight bytes, single stream, prompt processing and batching excluded. These are ceilings worked out from memory bandwidth, not measurements, and we do not state an error margin we have not measured. The faster the figure, the further real runtimes fall below it: when few weights are read per token, as with small models and mixture-of-experts models, the costs the formula leaves out (kernel launches, expert routing, KV cache reads, sampling) take a growing share of each token.
- No prices appear on this page on purpose. They move weekly; the build vs rent calculator uses yours.
- Every card's fit with every model, at every quantisation, is also open data: CSV and JSON under CC BY 4.0, archived on Zenodo with a DOI.