Why people self-host, and why it is not about price
People self-host language models for three reasons: their prompts and documents stay with them, they choose the model and can change it, and nobody upstream decides what they may ask. Price is rarely the reason. For ordinary chat, a per-token API is cheaper than any GPU you could run, and it is honest to say so. Self-hosting starts to pay when you generate a lot, when you run agents for hours, when you fine-tune, or when the data simply must not leave your hands.
How VRAM is actually used
Three things share the card's memory.
- The weights. Every parameter stored at some precision. At FP16 that is two bytes per parameter, so an 8B model is 16 GB before anything else. Quantisation shrinks this: at Q4_K_M it is about 0.58 bytes per parameter, so the same 8B model is under 5 GB.
- The KV cache. For every token in the context, the model keeps a key and a value for every layer and every KV head. This grows linearly with context length. For Llama 3.3 70B at FP16 KV, that is about 0.33 MB per token, so a 32k context costs roughly 10 GB on its own. Models with grouped-query attention keep this manageable; older architectures did not. Models from 2026 go further: Qwen3.8 27B keeps a growing cache on only 16 of its 64 layers and costs about 0.07 MB per token, a fifth of that. Why newer models pay less for context has the detail.
- Overhead. The CUDA context, activations and runtime buffers. Budget around half a gigabyte plus a few percent of the weights.
This is why "does it fit?" depends on context length as much as model size. A model that fits at 4k context can fail at 32k.
Calculator
Pick a model, a quantisation and a context length. The tool shows weights, KV cache and overhead separately, and which memory tier it fits with headroom. The formula is written out under the tool.
Qwen3.8 27B at Q4_K_M, 8,192 context, on RTX 4090
- Needs
- 18 GB
- Card has
- 24 GB
- Spare
- 4.8 GB
~44 tokens/s · comfortable for chat · room for 81,920 tokens of context at these settings
You also have room for
- You have room for Q5_K_M: 21.7 GB. Higher precision is free quality when the memory is there.
- You could run a larger model: Gemma 4 31B at 20.9 GB, ~39 tokens/s.
- You can push the context to 81,920 tokens before it stops fitting.
Or rent a bigger card
27.8B parameters · 1,008 GB/s · The parameter count includes the vision encoder; a text-only GGUF is slightly smaller.
Fits a 24 GB card (RTX 3090, RTX 4090) with headroom.
- Weights16.1 GB
- KV cache0.7 GB
- Overhead1.1 GB
By card size
- 8 GB No
- 12 GB No
- 16 GB No
- 24 GB Yes
- 32 GB Yes
- 48 GB Yes
- 80 GB Yes
- 96 GB Yes
The parameter count includes the vision encoder; a text-only GGUF is slightly smaller.
How this is calculated
Weights = parameters × bytes per parameter for the chosen quantisation (FP16 2.00, Q8 1.06, Q6_K 0.82, Q5_K_M 0.71, Q4_K_M 0.58, Q3_K_M 0.47; effective averages that include scales and the unquantised embedding and output layers). KV cache = 2 × layers × KV heads × head dim × context × bytes per value for a standard transformer. Three newer designs are counted the way they actually cache: sliding-window layers (Gemma, gpt-oss) stop at their window, hybrid models (Qwen3.5 and later, Nemotron) only grow a cache on their few full-attention layers and carry a small fixed state for the rest, and latent-attention models (Mistral Small 4, GLM Flash) cache one compressed vector per layer. Each model page says which applies. Overhead = 0.5 GB runtime context plus 4% of weights. "Fits" means at or under 95% of the card, leaving room for the desktop and the runtime. Mixture-of-experts models load every expert, so total parameters are used. Architecture numbers come from each model's public config; the list says which.
Comfortable for chat. 18 GB of 24 GB used.
- Active weights read per token
- 16.1 GB
- Memory needed at this setting
- 18 GB
- Card memory
- 24 GB
- Bandwidth
- 1,008 GB/s
The parameter count includes the vision encoder; a text-only GGUF is slightly smaller.
How this is calculated
Generating a token reads every active weight once, so the ceiling is memory bandwidth divided by the bytes of active weights. We multiply by 0.7 as a stated allowance for kernel overhead, KV cache reads and sampling; it is an assumption, not a measured figure. Mixture-of-experts models only read their active experts, which is why they are fast for their size. These are ceilings worked out from memory bandwidth, not measurements, and we do not state an error margin we have not measured. The faster the figure, the further real runtimes fall below it: when few weights are read per token, as with small models and mixture-of-experts models, the costs the formula leaves out (kernel launches, expert routing, KV cache reads, sampling) take a growing share of each token. Prompt processing is not included; it is compute-bound and much faster per token. Batching is not included; it multiplies throughput for a server, not for one conversation. If the model does not fit the card, the number is meaningless, and the tool says so.
Computed 16 October 2026Weights + KV cache + runtime overhead · an estimate from a stated formula
The runtimes that matter
| Runtime | Best for | Notes |
|---|---|---|
| Ollama | Getting a model running in one command; pairing with Open WebUI | Built on llama.cpp. Pulls GGUF models from its library or Hugging Face. The default for personal use. |
| llama.cpp | CPU, Apple Silicon, and squeezing a model onto a small card | The engine under Ollama and LM Studio. Most quantisation formats start here. |
| vLLM | Serving many requests at once; an OpenAI-compatible endpoint for agents | Continuous batching and paged attention. The right choice when throughput matters more than one-command simplicity. |
| SGLang | Structured generation and high-throughput serving | Comparable to vLLM; strong on constrained decoding. |
| Open WebUI | A chat interface with documents, users and history | Sits in front of Ollama or any OpenAI-compatible server. Keeps its history in its own database on your disk. |
Quantisation, plainly
Quantisation stores weights with fewer bits. Q8 is close to lossless. Q6_K and Q5_K_M are hard to distinguish from Q8 in ordinary use. Q4_K_M is the common default and costs a small, measurable amount of quality, most visible in long reasoning and code. Q3 and below are a last resort. A good rule: run the largest model that fits at Q4 or better, rather than a smaller model at higher precision. A 32B model at Q4 beats a 14B model at Q8 on most tasks.
The KV cache can be quantised too. Q8 KV halves cache memory at a small cost and is worth turning on for long context.
Hardware, owned or rented
| Memory | Cards | Comfortable with, at Q4 |
|---|---|---|
| 16 GB | RTX 4060 Ti 16 GB, 5060 Ti 16 GB | 8B to 14B; gpt-oss 20B |
| 24 GB | RTX 3090, 4090 | 27B to 31B: Qwen3.8 27B with long context, Gemma 4 31B to about 30k; older 32B at modest context |
| 32 GB | RTX 5090 | any 27B to 36B model with long context; 24B at Q8 |
| 48 GB | RTX 6000 Ada, L40S, A6000 | 70B at Q4 with modest context |
| 80 to 96 GB | A100/H100 80 GB, RTX PRO 6000 96 GB | 70B with long context; gpt-oss 120B, Mistral Small 4, Qwen3-Coder-Next |
Owning a 24 GB card is the classic starting point and is often the right call. Above 24 GB the price curve bends sharply, which is where renting starts to make sense: a 48 or 96 GB card for the evenings you need it, on a workspace that stays saved in between. The build vs rent calculator puts your own hours and electricity price against a monthly plan and finds the break-even month.
Mistakes that cost people weeks
- Buying for the model, not the context. A 24 GB card runs a 32B model at 4k context and chokes at 32k. Decide your context first.
- Running FP16 out of habit. Q5 or Q6 gives you a bigger model for the same memory with no visible loss.
- Ignoring idle power. A workstation left on draws 40 to 80 W all day. Over a year that is real money; the calculator includes it.
- Letting a rented machine's meter run. If you rent by the hour, automate the shutdown or choose a service that sleeps by itself.
- Trusting "private" without reading the policy. Ask any hosted service who can see what, layer by layer. If they cannot say, they log.
Questions
What is the best self-hosted LLM?
There is no single best. For general chat on a 24 GB card, the 27B to 31B class (Qwen3.8 27B, Gemma 4 31B at 4-bit) is the sweet spot in late 2026. On that card Qwen3.8 27B has room for roughly 80,000 tokens of context and Gemma 4 31B for about 30,000, where the 2025 Qwen3 32B managed about 9,000. For coding, Devstral Small 2 24B on a 24 GB card, or Qwen3-Coder-Next if you have 80 GB to put it in. For speed on modest hardware, mixture-of-experts models like Qwen3.6 35B-A3B, Gemma 4 26B-A4B or gpt-oss 20B, which load everything but only compute a few billion parameters per token.
Can I self-host an LLM without a GPU?
Yes, for small models. llama.cpp and Ollama run on CPUs and Apple Silicon. A 7B to 8B model at 4-bit on a modern laptop is usable; 30B and above is slow without a GPU, and image or video generation is impractical.
How much VRAM do I need for a 70B model?
About 40 GB for the weights at Q4_K_M, plus KV cache and overhead, so a 48 GB card at modest context, or 80 GB for long context. Use the calculator on this page with your context length; it shows the KV cache separately.
Is self-hosting cheaper than an API?
For plain chat, almost never: per-token APIs are extremely cheap at low volume. Self-hosting wins on privacy, on control over the model, and on workloads APIs charge dearly for, such as image and video generation, long-running agents, and fine-tuning.
- Bytes-per-parameter figures are effective averages for GGUF quantisation types, including scales and the unquantised embedding and output layers.
- KV cache formula: 2 × layers × KV heads × head dimension × context × bytes per value. Architecture values from each model's public config.json, listed inside the calculator.
- Card memory sizes are manufacturer specifications.