What a private AI server is
A private AI server is a machine that runs AI models where only you decide what goes in and what comes out. Open-weight models like Llama, Qwen, Gemma, Mistral and the gpt-oss family run on it. Your prompts, your documents and the outputs stay on that machine. No provider reads them to improve a product, and no provider decides what you are allowed to ask.
That is the whole idea. Everything else is a question of where the machine is, who operates it, and how much memory it has.
What "private" means, layer by layer
"Private" gets used loosely. It helps to ask the question at each layer of the stack, because the answer differs at each one.
| Layer | The question | Home build | GPU VPS | Persistent cloud grove |
|---|---|---|---|---|
| Prompts and files | Who can read them? | only you | you; provider could, in principle | you; operator commits not to log |
| Environment | Whose processes run beside yours? | nobody's | nobody's (VM) | nobody's (isolated) |
| GPU | Is the card shared? | no | usually dedicated | by plan, stated on the card |
| Physical machine | Who operates the hardware? | you | the provider | a named provider |
| Network | Does data leave a network you control? | never | yes, to the provider | yes, to the provider |
Only a home build is private at every layer. A rented option can be private at the layers most people care about, prompts, files and environment, provided the operator states its logging policy in writing and tells you who runs the hardware. If a vendor cannot answer the table above for its own product, that is the answer.
The three ways to get one
1. Build one at home
A PC with a graphics card that has enough VRAM. A 24 GB card, the used RTX 3090 or an RTX 4090, runs most models up to about 30B parameters at 4-bit quantisation. The RTX 5090 gives you 32 GB. Above that you are into workstation cards, 48 GB and 96 GB, and the price rises steeply.
Good when: you run models for hours every day, you need it to work offline, you already own a suitable card, or you enjoy the build. Costs you should count: the card, the rest of the machine, electricity at load and at idle, the hours you spend maintaining drivers and runtimes, and the fact that a card you buy today is fixed at that memory size.
2. Rent a GPU VPS
A virtual machine with a GPU attached, billed by the hour or the month, from a cloud or a hosting company. You install everything yourself: drivers, a model runtime such as Ollama or vLLM, a web interface, storage. You get root and total freedom.
Good when: you are comfortable administering Linux, you want a specific configuration, or you need it briefly. Watch for: hourly meters that keep running when you forget, storage that is wiped when the instance ends, and the time it takes to rebuild a machine from scratch every time.
3. A persistent cloud grove
This is the model Nodegrove is built on. Your workspace, meaning files, models, app settings and history, persists on its own disk. A GPU attaches when you wake the workspace and detaches when you are done. The apps arrive installed. Hours count only while the GPU is attached and the app is ready.
Good when: you use a GPU a few hours a week rather than all day, you want 48 or 96 GB without buying it, or you want the setup to persist without maintaining a box. Trade-offs: it is not offline, a provider runs the hardware, and the operator's privacy policy is a promise you have to be able to read and verify.
What runs on how much memory
Memory decides what you can run. A model's weights need to fit in VRAM alongside the KV cache that grows with context length. Use the calculator below with the models you care about. It uses each model's public architecture numbers and a stated formula; it is an estimate, not a benchmark.
Rule of thumb at 4-bit: a model needs roughly 0.6 GB of VRAM per billion parameters for weights, plus the KV cache, plus about a gigabyte of overhead. 8B fits 8 GB comfortably, 32B wants 24 GB, 70B wants 48 GB, and the 120B mixture-of-experts models want 80 GB or more.
Which one is right for you
- You run models most of the day, or need it offline: build. The hardware pays for itself, and nothing beats a box in your own room for privacy.
- You are a Linux administrator who wants total control for a short project: a GPU VPS.
- You use a GPU a few evenings a week and want your setup to be there when you come back: a persistent grove. Use the build vs rent calculator with your own numbers to see the break-even month.
Questions
Is a private AI server the same as running AI locally?
Local means the model runs on hardware in your home or office. A private AI server is broader: any machine only you use, at home or rented, where your prompts and files are not logged by the operator. Every local setup is private; not every private setup is local.
Do I need a GPU for a private AI server?
For anything above a few billion parameters at a usable speed, yes. Small models run on CPUs and Apple Silicon, but image and video generation and 30B-plus language models want a GPU with enough VRAM to hold the model.
Can a rented server really be private?
It can be private in the ways that matter to most people: an isolated environment, an operator that does not log prompts, and data you can export and delete. It cannot be air-gapped, and the provider operates the hardware. Ask any vendor to state, layer by layer, who can see what.
- Architecture numbers in the calculator come from each model's public config.json or model card, listed per model inside the tool.
- VRAM figures for consumer and workstation cards are manufacturer specifications.