How this is calculated
Generating a token reads every active weight once, so the ceiling is memory bandwidth divided by the bytes of active weights. We multiply by 0.7 because real runtimes lose 20–40% to kernel overhead, KV cache reads and sampling. Mixture-of-experts models only read their active experts, which is why they are fast for their size. Prompt processing is not included; it is compute-bound and much faster per token. Batching is not included; it multiplies throughput for a server, not for one conversation. If the model does not fit the card, the number is meaningless, and the tool says so.
Why bandwidth, not compute, sets chat speed
When a model answers you, it produces one token at a time, and for every token it has to read all of its active weights out of memory. A 32B model at 4-bit is about 19 GB of weights, so a card that moves 1,000 GB/s can, at the very best, read it about 50 times a second. That is the ceiling. Compute matters for reading your prompt, which happens in one pass and is fast, and for serving many people at once, which batches the reads.
This is why two cards with the same memory can feel very different, why a 5090 is nearly twice as fast as a 4090 for the same model, and why mixture-of-experts models feel quick: they only read the experts they use.
How to read the number
- Above 25 tokens per second feels immediate in chat. Reading speed is around 5 to 8 words per second.
- 12 to 25 is fine for most people and slightly slow for long code outputs.
- Under 5 is only worth it for batch jobs you can walk away from.
- If the model does not fit the card, ignore the speed: offloading to system memory is ten to fifty times slower.
Check the fit first with the VRAM calculator, or ask both questions at once with the can-I-run-it checker. For the reasoning behind quantisation choices, read the self-hosted LLM guide.