Why people self-host, and why it is not about price
People self-host language models for three reasons: their prompts and documents stay with them, they choose the model and can change it, and nobody upstream decides what they may ask. Price is rarely the reason. For ordinary chat, a per-token API is cheaper than any GPU you could run, and it is honest to say so. Self-hosting starts to pay when you generate a lot, when you run agents for hours, when you fine-tune, or when the data simply must not leave your hands.
How VRAM is actually used
Three things share the card's memory.
- The weights. Every parameter stored at some precision. At FP16 that is two bytes per parameter, so an 8B model is 16 GB before anything else. Quantisation shrinks this: at Q4_K_M it is about 0.58 bytes per parameter, so the same 8B model is under 5 GB.
- The KV cache. For every token in the context, the model keeps a key and a value for every layer and every KV head. This grows linearly with context length. For Llama 3.3 70B at FP16 KV, that is about 0.33 MB per token, so a 32k context costs roughly 10 GB on its own. Models with grouped-query attention keep this manageable; older architectures did not.
- Overhead. The CUDA context, activations and runtime buffers. Budget around half a gigabyte plus a few percent of the weights.
This is why "does it fit?" depends on context length as much as model size. A model that fits at 4k context can fail at 32k.
Calculator
Pick a model, a quantisation and a context length. The tool shows weights, KV cache and overhead separately, and which memory tier it fits with headroom. The formula is written out under the tool.
How this is calculated
Weights = parameters × bytes per parameter for the chosen quantisation (FP16 2.00, Q8 1.06, Q6_K 0.82, Q5_K_M 0.71, Q4_K_M 0.58, Q3_K_M 0.47; effective averages that include scales and the unquantised embedding and output layers). KV cache = 2 × layers × KV heads × head dim × context × bytes per value. Overhead = 0.5 GB runtime context plus 4% of weights. "Fits" means at or under 95% of the card, leaving room for the desktop and the runtime. Mixture-of-experts models load every expert, so total parameters are used. Architecture numbers come from each model's public config; the list says which.
The runtimes that matter
| Runtime | Best for | Notes |
|---|---|---|
| Ollama | Getting a model running in one command; pairing with Open WebUI | Built on llama.cpp. Pulls GGUF models from its library or Hugging Face. The default for personal use. |
| llama.cpp | CPU, Apple Silicon, and squeezing a model onto a small card | The engine under Ollama and LM Studio. Most quantisation formats start here. |
| vLLM | Serving many requests at once; an OpenAI-compatible endpoint for agents | Continuous batching and paged attention. The right choice when throughput matters more than one-command simplicity. |
| SGLang | Structured generation and high-throughput serving | Comparable to vLLM; strong on constrained decoding. |
| Open WebUI | A chat interface with documents, users and history | Sits in front of Ollama or any OpenAI-compatible server. Keeps its history in its own database on your disk. |
Quantisation, plainly
Quantisation stores weights with fewer bits. Q8 is close to lossless. Q6_K and Q5_K_M are hard to distinguish from Q8 in ordinary use. Q4_K_M is the common default and costs a small, measurable amount of quality, most visible in long reasoning and code. Q3 and below are a last resort. A good rule: run the largest model that fits at Q4 or better, rather than a smaller model at higher precision. A 32B model at Q4 beats a 14B model at Q8 on most tasks.
The KV cache can be quantised too. Q8 KV halves cache memory at a small cost and is worth turning on for long context.
Hardware, owned or rented
| Memory | Cards | Comfortable with, at Q4 |
|---|---|---|
| 16 GB | RTX 4060 Ti 16 GB, 5060 Ti 16 GB | 8B to 14B; gpt-oss 20B |
| 24 GB | RTX 3090, 4090 | up to 32B at modest context; Gemma 3 27B |
| 32 GB | RTX 5090 | 32B with long context; Qwen3 30B-A3B with room |
| 48 GB | RTX 6000 Ada, L40S, A6000 | 70B at Q4 with modest context |
| 80 to 96 GB | A100/H100 80 GB, RTX PRO 6000 96 GB | 70B with long context; gpt-oss 120B |
Owning a 24 GB card is the classic starting point and is often the right call. Above 24 GB the price curve bends sharply, which is where renting starts to make sense: a 48 or 96 GB card for the evenings you need it, on a workspace that stays saved in between. The build vs rent calculator puts your own hours and electricity price against a monthly plan and finds the break-even month.
Mistakes that cost people weeks
- Buying for the model, not the context. A 24 GB card runs a 32B model at 4k context and chokes at 32k. Decide your context first.
- Running FP16 out of habit. Q5 or Q6 gives you a bigger model for the same memory with no visible loss.
- Ignoring idle power. A workstation left on draws 40 to 80 W all day. Over a year that is real money; the calculator includes it.
- Letting a rented machine's meter run. If you rent by the hour, automate the shutdown or choose a service that sleeps by itself.
- Trusting "private" without reading the policy. Ask any hosted service who can see what, layer by layer. If they cannot say, they log.
Questions
What is the best self-hosted LLM?
There is no single best. For general chat on a 24 GB card, the 27B to 32B class (Gemma 3 27B, Qwen3 32B at 4-bit) is the sweet spot in 2026. For coding, Qwen2.5 Coder 32B or the Qwen3 Coder models. For speed on modest hardware, mixture-of-experts models like Qwen3 30B-A3B or gpt-oss 20B, which load everything but only compute a few billion parameters per token.
Can I self-host an LLM without a GPU?
Yes, for small models. llama.cpp and Ollama run on CPUs and Apple Silicon. A 7B to 8B model at 4-bit on a modern laptop is usable; 30B and above is slow without a GPU, and image or video generation is impractical.
How much VRAM do I need for a 70B model?
About 40 GB for the weights at Q4_K_M, plus KV cache and overhead, so a 48 GB card at modest context, or 80 GB for long context. Use the calculator on this page with your context length; it shows the KV cache separately.
Is self-hosting cheaper than an API?
For plain chat, almost never: per-token APIs are extremely cheap at low volume. Self-hosting wins on privacy, on control over the model, and on workloads APIs charge dearly for, such as image and video generation, long-running agents, and fine-tuning.
- Bytes-per-parameter figures are effective averages for GGUF quantisation types, including scales and the unquantised embedding and output layers.
- KV cache formula: 2 × layers × KV heads × head dimension × context × bytes per value. Architecture values from each model's public config.json, listed inside the calculator.
- Card memory sizes are manufacturer specifications.