Nemotron 3.5 Bolt 74B-A9B (MoE)
Nemotron 3.5 Bolt 74B-A9B (MoE) needs 45.9 GB of VRAM at Q4_K_M with 8k context, computed from its config.json as checked on 16 October 2026. The smallest card that holds it has 80 GB, and the fastest desktop card that runs it, the RTX PRO 6000 Blackwell, generates at most 240 tokens per second.
- Parameters
- 74.9B, 9B active
- Context
- 262,144 tokens
- Licence
- OpenMDW 1.1
- Released
- October 2026
- Config checked
- 16 October 2026
Open Nemotron 3.5 Bolt 74B-A9B (MoE) in the LLM checker
Which card size runs it
45.9 GB against each memory size. A model may use 95% of a card, and only what is left over, or missing, is coloured; tight means above 85%, with little room for a longer conversation.
Other quantisations, context lengths or a Q8 KV cache: the VRAM calculator.
Computed 16 October 2026Which GPU to pick
The smallest card here that loads it at Q4_K_M with 8k of context is the A100 (80 GB): 45.9 GB needed, an estimated 273 tokens per second. The same card still has room at 32,768 tokens of context (46.1 GB), so there is no reason to buy above it for this model.
At Q8_0, which is near-lossless, the card is the RTX PRO 6000 Blackwell (96 GB), needing 83.3 GB. On Apple silicon, a MacBook Pro M4 Max holds it with 32,768 tokens of context at an estimated 73 tokens per second; prompt processing is slower than on Nvidia, which long documents make obvious. Among unified-memory desktops, the Ryzen AI Max+ 395 holds it at 32,768 tokens in 46.1 GB, at an estimated 34 tokens per second.
Speed on each card
| GPU | Fits | Est. tokens/s | Feels like | Memory | Bandwidth |
|---|---|---|---|---|---|
| H100 SXM | Yes | ≤ 449 | faster than you can read | 80 GB | 3,350 GB/s |
| A100 | Yes | ≤ 273 | faster than you can read | 80 GB | 2,039 GB/s |
| RTX PRO 6000 Blackwell | Yes | ≤ 240 | faster than you can read | 96 GB | 1,792 GB/s |
| RTX Spark | Yes | 40 | comfortable for chat | 128 GB (102 for the GPU) | 300 GB/s |
| DGX Spark | Yes | 37 | comfortable for chat | 128 GB (126 for the GPU) | 273 GB/s |
| Ryzen AI Max+ 395 | Yes | 34 | comfortable for chat | 128 GB (96 for the GPU) | 256 GB/s |
| RTX 6000 Ada | No | - | does not fit | 48 GB | 960 GB/s |
| L40S | No | - | does not fit | 48 GB | 864 GB/s |
| RTX A6000 | No | - | does not fit | 48 GB | 768 GB/s |
| RTX PRO 5000 Blackwell | No | - | does not fit | 48 GB | 1,344 GB/s |
| RTX 5090 | No | - | does not fit | 32 GB | 1,792 GB/s |
| RTX PRO 4500 Blackwell | No | - | does not fit | 32 GB | 896 GB/s |
| Radeon AI PRO R9700 | No | - | does not fit | 32 GB | 640 GB/s |
| Arc Pro B70 | No | - | does not fit | 32 GB | 608 GB/s |
| RTX 3090 | No | - | does not fit | 24 GB | 936 GB/s |
| RTX 4090 | No | - | does not fit | 24 GB | 1,008 GB/s |
| RX 7900 XTX | No | - | does not fit | 24 GB | 960 GB/s |
| RTX PRO 4000 Blackwell | No | - | does not fit | 24 GB | 672 GB/s |
| RTX PRO 4000 Blackwell SFF | No | - | does not fit | 24 GB | 432 GB/s |
| Arc Pro B60 | No | - | does not fit | 24 GB | 456 GB/s |
| RX 7900 XT | No | - | does not fit | 20 GB | 800 GB/s |
| RTX 4060 Ti | No | - | does not fit | 16 GB | 288 GB/s |
| RTX 5060 Ti | No | - | does not fit | 16 GB | 448 GB/s |
| RTX 5080 | No | - | does not fit | 16 GB | 960 GB/s |
| RTX 5070 Ti | No | - | does not fit | 16 GB | 896 GB/s |
| RTX 4080 SUPER | No | - | does not fit | 16 GB | 736 GB/s |
| RX 9070 XT | No | - | does not fit | 16 GB | 640 GB/s |
| RX 9060 XT | No | - | does not fit | 16 GB | 320 GB/s |
| RTX 3060 | No | - | does not fit | 12 GB | 360 GB/s |
| RTX 4070 | No | - | does not fit | 12 GB | 504 GB/s |
| RTX 5070 | No | - | does not fit | 12 GB | 672 GB/s |
| Arc B580 | No | - | does not fit | 12 GB | 456 GB/s |
Single-stream decode, from memory bandwidth at 70% efficiency. Prompt processing and batching not included. These are ceilings, not measurements: Nemotron 3.5 Bolt 74B-A9B (MoE) reads only 5.2 GB of weights per token at Q4, so little that the per-token costs the formula leaves out decide much of the real speed. The 3 figures marked ≤ are where a real runtime falls furthest below the number shown. See the speed estimator for other quantisations and Apple hardware, or every card compared if you are choosing hardware rather than a model.
Computed 16 October 2026Memory by quantisation and context
| Quant | 4,096 ctx | 8,192 ctx | 32,768 ctx |
|---|---|---|---|
| Q8_0 | 83.3 GB · fits 96 GB | 83.3 GB · fits 96 GB | 83.5 GB · fits 96 GB |
| Q5_K_M | 56 GB · fits 80 GB | 56 GB · fits 80 GB | 56.2 GB · fits 80 GB |
| Q4_K_M | 45.9 GB · fits 80 GB | 45.9 GB · fits 80 GB | 46.1 GB · fits 80 GB |
Weights at this quant: 43.4 GB at Q4. Every extra 1,000 tokens of context adds about 0.008 GB of KV cache at FP16, on top of a fixed 0.17 GB of recurrent state that does not grow. Try other settings in the VRAM calculator.
About the model
Nemotron 3.5 Bolt 74B-A9B (MoE) has 74.9B parameters with 9B active per token (mixture of experts). It has 88 blocks, of which only 8 are attention layers (2 KV heads of dimension 128); 40 are Mamba-2 layers with a small fixed state and 40 are mixture-of-experts layers with no cache at all. NVIDIA's 74B-A9B vision-language model, October 2026, compressed from its Nemotron 3.5 Super VL 120B-A12B to run agent work on one machine. It is the same three-way hybrid as Lightning (Mamba-2 layers, mixture-of-experts layers and only eight attention layers), more than twice the size, and it reads images and video.
Text, images and video 262,144-token window OpenMDW 1.1 October 2026 9B of 74.9B active
Reasoning. Switchable per request in the chat template, with a medium-effort setting between off and full reasoning.
NVIDIA's own 24GB release is a native 2-bit build, below the lowest row computed here: its GGUF weights are 22.0 GB, and the vision encoder comes as a separate 1.7 GB file.
Native context window: 262,144 tokens. Licence: OpenMDW 1.1, a custom licence with its own conditions; read it before commercial use. Weights published in October 2026.
Good at
- A 24 GB card through NVIDIA's own 2-bit build, which NVIDIA sized for that memory instead of leaving the squeeze to community quants.
- Long agent sessions: eight attention layers with two KV heads each keep the cache small even near its 262,144-token window.
- An 80 GB card or a 128 GB machine at Q4, or a 48 GB card at Q3_K_M.
- Tool calling and MCP: NVIDIA trained it with reinforcement learning in MCP and software-engineering environments, the MCP one built on a simulated MCP server.
- Screenshots, documents and videos of up to two minutes alongside the conversation.
Watch out for
- The 24 GB in NVIDIA's name is its 2-bit build. At Q3_K_M, the lowest level this page computes, it needs a 48 GB card, and 2-bit weights give up some quality; NVIDIA publishes results for both versions, so compare them for your task.
- The BF16 checkpoint is meant for customisation. For running it, NVIDIA points to its 24GB builds: a GGUF for llama.cpp and a version for vLLM.
- NVIDIA's own instructions use vLLM with a reasoning-parser file from the repo; other runtimes need a build that knows the Nemotron hybrid layers.
- OpenMDW 1.1 is its own licence; read it rather than assuming Apache.
- The card names its languages twice and the lists differ: English with Spanish, French, German, Italian and Japanese in one place, and those plus Hindi, Korean, Brazilian Portuguese and Chinese in another. Test yours.
Similar models
The models that need about the same memory as Nemotron 3.5 Bolt 74B-A9B (MoE) at Q4_K_M and 8k, which makes them the real alternatives on whatever card you have. The context column is what each extra 1,000 tokens costs, and it is where models of the same size differ most.
| Model | Size | Needs | Per 1k context | Window | Released |
|---|---|---|---|---|---|
| Nemotron 3.5 Bolt 74B-A9B (MoE) | 74.9B · 9B active | 45.9 GB | 0.008 GB | 256k | 2026-10 |
| DeepSeek-R1 Distill Llama 70B | 70.6B | 45.8 GB | 0.328 GB | 128k | 2025-01 |
| Llama 3.3 70B | 70.6B | 45.8 GB | 0.328 GB | 128k | 2024-12 |
| Qwen3-Coder-Next 80B-A3B (MoE) | 79.7B · 3B active | 48.9 GB | 0.025 GB | 256k | 2026-01 |
| gpt-oss 120B | 117B · 5.1B active | 71.4 GB | 0.037 GB | 128k | 2025-08 |
See every model compared, which models fit on 8 to 96 GB, or the cheapest hour of every cloud card.
Questions
- Can I run Nemotron 3.5 Bolt 74B-A9B (MoE) on a 24 GB card?
- Not comfortably. At Q4_K_M and 8k context it needs about 45.9 GB. The smallest tier that fits is 80 GB.
- How much VRAM does Nemotron 3.5 Bolt 74B-A9B (MoE) need at Q8?
- About 83.3 GB at 8k context, or 83.5 GB at 32k. Q8 is near-lossless; use it when it fits.
- What is the context length of Nemotron 3.5 Bolt 74B-A9B (MoE)?
- 262,144 tokens natively. Holding all of it at Q4_K_M takes about 48 GB, of which 2.3 GB is context.
- Why does Nemotron 3.5 Bolt 74B-A9B (MoE) need so little memory for long context?
- Because most of its layers do not keep a cache that grows. It has 88 blocks, of which only 8 are attention layers (2 KV heads of dimension 128); 40 are Mamba-2 layers with a small fixed state and 40 are mixture-of-experts layers with no cache at all. At 32k tokens the context costs about 0.4 GB; if all 88 layers kept a conventional full-length cache, the same conversation would cost about 3 GB.
- Does Nemotron 3.5 Bolt run on a 24 GB card?
- With a 2-bit build, such as NVIDIA's own 24GB release, which NVIDIA made to fit 24 GB of memory. At Q3_K_M and above, the levels this page computes, it needs more memory than a 24 GB card has.
- How fast is Nemotron 3.5 Bolt 74B-A9B (MoE) on an RTX 4090?
- It does not fit a 4090 at Q4 and 8k context, so speed would collapse to CPU offloading. Use a larger card or a smaller quantisation.
- Architecture values from config.json in nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16 on Hugging Face, and its model card, checked 16 October 2026. Every figure on this page is computed from those values; check them before buying hardware for this model.
- Estimates assume a single conversation on a card that is otherwise free. A desktop on the same GPU takes 0.5 to 2 GB.