Llama 3.3 70B has 70.6B parameters, 80 layers and 8 KV heads of dimension 128. At the everyday setting, Q4_K_M and 8k context, it needs an estimated 45.8 GB, so the smallest card that fits is 80 GB.
Memory by quantisation and context
| Quant | 4,096 ctx | 8,192 ctx | 32,768 ctx |
|---|---|---|---|
| Q8_0 | 79.7 GB · fits 96 GB | 81 GB · fits 96 GB | 89.1 GB · fits 96 GB |
| Q5_K_M | 54 GB · fits 80 GB | 55.3 GB · fits 80 GB | 63.4 GB · fits 80 GB |
| Q4_K_M | 44.4 GB · fits 48 GB | 45.8 GB · fits 80 GB | 53.8 GB · fits 80 GB |
Weights at this quant: 40.9 GB at Q4. Every extra 1,000 tokens of context adds about 0.3277 GB of KV cache at FP16. Try other settings in the VRAM calculator.
Speed by GPU, at Q4 and 8k context
| GPU | Memory | Bandwidth | Fits | Est. tokens/s | Feels like |
|---|---|---|---|---|---|
| RTX 3060 12 GB | 12 GB | 360 GB/s | no | – | does not fit |
| RTX 4060 Ti 16 GB | 16 GB | 288 GB/s | no | – | does not fit |
| RTX 4070 12 GB | 12 GB | 504 GB/s | no | – | does not fit |
| RTX 3090 24 GB | 24 GB | 936 GB/s | no | – | does not fit |
| RTX 4090 24 GB | 24 GB | 1,008 GB/s | no | – | does not fit |
| RTX 5090 32 GB | 32 GB | 1,792 GB/s | no | – | does not fit |
| RTX 6000 Ada 48 GB | 48 GB | 960 GB/s | no | – | does not fit |
| L40S 48 GB | 48 GB | 864 GB/s | no | – | does not fit |
| RTX PRO 6000 Blackwell 96 GB | 96 GB | 1,792 GB/s | yes | 31 | comfortable for chat |
| A100 80 GB | 80 GB | 2,039 GB/s | yes | 35 | comfortable for chat |
| H100 SXM 80 GB | 80 GB | 3,352 GB/s | yes | 57 | comfortable for chat |
Single-stream decode ceiling from memory bandwidth at 70% efficiency. Prompt processing and batching not included. See the speed estimator for other quantisations and Apple hardware, or every card compared if you are choosing hardware rather than a model.
Notes
- Architecture values from meta-llama/Llama-3.3-70B-Instruct config.json. Verify against the model card before buying hardware for this model.
- Estimates assume a single conversation on a card that is otherwise free. A desktop on the same GPU takes 0.5 to 2 GB.
Questions
Can I run Llama 3.3 70B on a 24 GB card?
Not comfortably. At Q4_K_M and 8k context it needs about 45.8 GB. The smallest tier that fits is 80 GB.
How much VRAM does Llama 3.3 70B need at Q8?
About 81 GB at 8k context, or 89.1 GB at 32k. Q8 is near-lossless; use it when it fits.
How fast is Llama 3.3 70B on an RTX 4090?
It does not fit a 4090 at Q4 and 8k context, so speed would collapse to CPU offloading. Use a larger card or a smaller quantisation.
See how it compares in which models fit on 8 to 96 GB, or run it without buying the card: Nodegrove attaches a 24, 48 or 96 GB GPU to a workspace that stays saved.