RTX 4090
An RTX 4090 runs 28 of the 34 open models tracked here at Q4_K_M with 8k context, computed on 16 October 2026. The largest is Qwen3.6 35B-A3B (MoE), needing 22.4 GB and generating at most 406 tokens per second.
- Memory
- 24 GB VRAM
- Bandwidth
- 1,008 GB/s
- Board power
- 450 W
- Released
- 2022
- Availability
- Mostly second-hand
Open the RTX 4090 in the LLM checker
What an RTX 4090 runs, model by model
RTX 4090 (24 GB) addresses 24 GB at 1,008 GB/s. Every model below is at Q4_K_M with 8k of context, the setting most people actually use. Once the 5% safety margin comes off, the card has 22.8 GB to spend: memory decides what loads, bandwidth how fast it answers. Both are estimates from stated formulas, and the VRAM calculator shows the working.
On this card a model fits at or under 22.8 GB, and above 20.4 GB it is tight: it loads, but there is little left for the conversation, and a desktop on the same 24 GB can push it over. Tokens per second are single-stream estimates from memory bandwidth; ≤ marks a ceiling real runtimes fall well below.
Computed 16 October 2026The model to actually run on it
Muse Glimmer 30B is the best use of this card: 18.7 GB of the 24 GB available, an estimated 41 tokens per second, comfortable for chat, and 131,072 tokens of context still available.
How this is chosen, since no benchmark is quoted: models are compared by size, a mixture-of-experts model counting at the geometric mean of its total and active parameters (a rule of thumb, not a measurement). The pick is the biggest class that fits with memory to spare and answers at 25 tokens per second or better, and within that class the most recent general-purpose release; coding and reasoning specialists are listed above but not recommended by default.
Qwen3.6 35B-A3B (MoE) is the largest model the card will hold, at 22.4 GB, but it is the wrong daily driver: filling 93% of memory leaves room for only 25,600 tokens of conversation. More parameters are not worth a context window that runs out mid-document. It is a mixture-of-experts model reading only 3B parameters per token, so it is the faster of the two at an estimated 406 tokens per second; what it gives up is the depth of a dense model that reads all of its weights for every token.
If quality matters more than parameter count, Gemma 4 12B fits at Q8_0 in about 14.2 GB. Q8 is near-lossless; a smaller model at Q8 often beats a larger one squeezed into Q3.
How much context actually fits
Model size is the question people ask; context is the one that bites. The KV cache grows linearly with the conversation, so a model that loads comfortably can still run out of memory halfway through a long document. On 24 GB, the context you get is whatever the weights leave behind, which is why two cards that run the same model can feel completely different in use. These are the largest contexts this card holds, at Q4_K_M weights.
| Model | Max context, FP16 KV | With Q8 KV cache |
|---|---|---|
| Muse Glimmer 30B | 131,072 tokens | 131,072 tokens |
| Qwen3 32B | 9,216 tokens | 18,432 tokens |
| Gemma 4 31B | 30,720 tokens | 72,704 tokens |
| Granite 4.2 30B | 17,408 tokens | 34,816 tokens |
Quantising the KV cache to Q8 roughly doubles what you can hold, at a quality cost most people never notice: Qwen3 32B on this card goes from 9,216 to 18,432 tokens. Capped at 128k tokens, or at the model's own native window where that is smaller, marked "model limit": free memory beyond that point buys nothing. Rows that hold far more context than their size suggests are hybrid, sliding-window or latent-attention models, which cache only a fraction of what a standard transformer does; each model page shows the working.
Where this card stops
The first model out of reach is Llama 3.3 70B: about 45.8 GB at Q4_K_M and 8k context, against 24 GB of usable memory. Dropping to Q3_K_M would need about 37.7 GB, which still does not fit. Offloading the remainder to system RAM works and is ten to fifty times slower; it is a way to see a model run, not a way to use one.
Buy or rent
Worth it if you also generate images or video, or if prompt processing speed matters to you. If you only chat with text models, the 3090 does the same work for much less.
Second-hand pricing is the whole argument for this card, and it moves week to week, so printing a number here would be worse than printing none. Put the figure you are actually looking at, with your electricity rate and your real monthly hours, into the build vs rent calculator, which already knows this card draws 450 W under load. It returns the month owning becomes cheaper, or tells you it never does. Hours per month decides it far more often than the price does.
Rented, an RTX 4090 costs from $0.33 an hour (SaladCloud, home PCs), or $0.89 from a provider that runs the machine (RunPod), on demand, before tax, read 19 October 2026. Every provider's price is in the cloud GPU price guide.
The honest take
The card most local-model advice is implicitly written for.
The 4090 is the reference point: 24 GB, 1008 GB/s, and enough compute that nothing else in a desktop is the bottleneck. It runs 32B-class models at Q4 quickly and comfortably, handles image and video generation without complaint, and stops dead at 70B, which needs roughly twice its memory. Most "can I run this locally" questions are really asking whether it fits on a 4090.
What it is good at
- 1008 GB/s puts 32B models comfortably above reading speed.
- Compute headroom that a 3090 does not have, which shows up in image and video work and in prompt processing.
- The best-supported consumer card in every runtime, so problems have already been solved by someone else.
What it is not
- 24 GB is the same ceiling as the 3090 that costs much less. You are paying for speed, not capability.
- 450 W and a four-slot cooler on most models; plan the case and the power supply around it.
- No NVLink, so two cards cannot pool memory the way two 3090s can.
The thing people get wrong: A 4090 is about 7% faster than a 3090 at generating text, not the two-generation leap the rest of its specification suggests. Single-stream decoding reads weights out of memory, and 1008 GB/s against 936 GB/s is the whole story. The gap only opens up on prompt processing, image generation and batching.
Compared with the alternatives
- RTX 3090: 24 GB · 936 GB/s · best fit Muse Glimmer 30B at ~38 tok/s. The value benchmark for local models, and it has been for years.
- RTX 5090: 32 GB · 1,792 GB/s · best fit Muse Glimmer 30B at ~73 tok/s. The first consumer card whose memory bandwidth changes what is comfortable.
- RTX 6000 Ada: 48 GB · 960 GB/s · best fit Muse Glimmer 30B at ~39 tok/s. Forty-eight gigabytes in a normal computer, at 300 watts, without the noise.
All 39 cards are compared side by side on the GPU index, and the same numbers from the model's point of view are on each model fit table. To check one specific pairing rather than read a table, use the can-I-run-it checker.
Questions
- Can an RTX 4090 run a 70B model?
- Not at Q4_K_M in one card. Llama 3.3 70B needs about 45.8 GB at 8k context and this card can address 24 GB. Your options are a smaller model, a harsher quantisation with very little context, splitting across two cards, or renting a larger one for the hours you need it.
- What is the best model to run on an RTX 4090?
- Muse Glimmer 30B. At Q4_K_M and 8k context it needs about 18.7 GB of the 24 GB available and generates an estimated 41 tokens per second, which is comfortable for chat. If quality matters more than size, Gemma 4 12B fits at Q8 in about 14.2 GB.
- How many tokens per second does an RTX 4090 generate?
- It depends entirely on the size of the model, because generating a token means reading every active weight out of memory. At 1,008 GB/s and 70% efficiency, this card produces an estimated 152 tokens per second on an 8B model at Q4, and about 406 on the largest model it holds, Qwen3.6 35B-A3B (MoE). Reading speed is roughly 5 to 8 words per second, so anything above 25 feels immediate. Treat 152 as a ceiling on 1,008 GB/s rather than a measurement; the higher a figure, the further real runtimes fall below it.
- Is 24 GB enough for running LLMs locally?
- It runs 28 of the 34 models tracked here at Q4_K_M with 8k of context, up to 36B parameters. The honest test is not the model list but the context: Muse Glimmer 30B on this card holds about 131,072 tokens before memory runs out.
- RTX 4090 or RTX 3090 for local models?
- Both address about the same memory, so they run the same models. On speed, this card is faster: 1,008 GB/s against 936 GB/s, and bandwidth is what sets chat speed.
- RTX 4090 or RTX 3090 for running LLMs?
- For text generation they are close: both hold 24 GB and the 4090 reads memory about 7% faster, which is what sets chat speed. The 4090 pulls ahead on prompt processing, image and video generation, and anything compute-bound. If you only chat with text models, the 3090 does the same work for much less; if you also generate images, the 4090 earns the difference.
- Can two RTX 4090s run a 70B model?
- Two cards give 48 GB of combined memory, which holds a 70B model at Q4 when the runtime splits it across both. What the 4090 cannot do is pool that memory over NVLink, which was removed on this generation, so the cards talk over PCIe and the link between them becomes the constraint for training.
- Memory: weights (parameters × bytes per parameter) + KV cache (2 × layers × KV heads × head dim × context × bytes for a standard transformer; sliding-window, hybrid and latent-attention models counted as they actually cache) + 0.5 GB runtime + 4% of weights. Full derivation in the VRAM calculator.
- Speed: 70% × bandwidth ÷ active weight bytes, single stream, no batching, prompt processing excluded. A ceiling, not a measurement: the 13 of 28 figures marked ≤ are where real runtimes fall furthest below the number, because so few weights are read per token. See the tokens-per-second estimator.
- Card specification (manufacturer figures): 24 GB, 1,008 GB/s, 450 W; see NVIDIA's own page for this card. The bandwidth figure: NVIDIA's Ada architecture whitepaper.
- Hourly prices: each provider's own pricing page or price API, read 19 October 2026, as listed in the cloud GPU price guide. Power: the card's full board power at $0.20 a kWh, before the rest of the PC.
- Model architecture values come from each model's published config; the self-hosted LLM guide explains what each one changes.