What is a private AI server, and which of the three kinds should you get?

What a private AI server is

A private AI server is a machine that runs AI models where only you decide what goes in and what comes out. Open-weight models like Llama, Qwen, Gemma, Mistral and the gpt-oss family run on it. Your prompts, your documents and the outputs stay on that machine. No provider reads them to improve a product, and no provider decides what you are allowed to ask.

That is the whole idea. Everything else is a question of where the machine is, who operates it, and how much memory it has.

What "private" means, layer by layer

"Private" gets used loosely. It helps to ask the question at each layer of the stack, because the answer differs at each one.

Only a home build is private at every layer. A rented option can be private at the layers most people care about, prompts, files and environment, provided the operator states its logging policy in writing and tells you who runs the hardware. If a vendor cannot answer the table above for its own product, that is the answer.

The three ways to get one

1. Build one at home

A PC with a graphics card that has enough VRAM. A 24 GB card, the used RTX 3090 or an RTX 4090, runs most models up to about 30B parameters at 4-bit quantisation. The RTX 5090 gives you 32 GB. Above that you are into workstation cards, 48 GB and 96 GB, and the price rises steeply.

Good when: you run models for hours every day, you need it to work offline, you already own a suitable card, or you enjoy the build. Costs you should count: the card, the rest of the machine, electricity at load and at idle, the hours you spend maintaining drivers and runtimes, and the fact that a card you buy today is fixed at that memory size.

2. Rent a GPU VPS

A virtual machine with a GPU attached, billed by the hour or the month, from a cloud or a hosting company. You install everything yourself: drivers, a model runtime such as Ollama or vLLM, a web interface, storage. You get root and total freedom.

Good when: you are comfortable administering Linux, you want a specific configuration, or you need it briefly. Watch for: hourly meters that keep running when you forget, storage that is wiped when the instance ends, and the time it takes to rebuild a machine from scratch every time.

3. Pay per run: serverless and managed GPUs

You send work, not a machine: a ComfyUI workflow, a batch or a request to a model, and pay for the seconds it runs. Comfy Cloud does this for ComfyUI; Modal and RunPod Serverless do it for a container you build. Nothing is left running between jobs.

Good when: you use a GPU a few hours a week rather than all day, you want 48 or 96 GB without buying it, or the work comes in bursts. Trade-offs: it is not offline, a provider runs the hardware, the first job after a pause can wait for a cold start, and the operator's privacy policy is a promise you have to be able to read. The ComfyUI cloud guide and the ComfyUI API guide compare what each service states about training on your content.

What runs on how much memory

Memory decides what you can run. A model's weights need to fit in VRAM alongside the KV cache that grows with context length. Use the calculator below with the models you care about. It uses each model's public architecture numbers and a stated formula; it is an estimate, not a benchmark.

Rule of thumb at 4-bit: a model needs roughly 0.6 GB of VRAM per billion parameters for weights, plus the KV cache, plus about a gigabyte of overhead. 8B fits 8 GB comfortably, 32B wants 24 GB, 70B wants 48 GB, and the 120B mixture-of-experts models want 80 GB or more.

Which one is right for you

• You run models most of the day, or need it offline: build. The hardware pays for itself, and nothing beats a box in your own room for privacy.
• You are a Linux administrator who wants total control for a short project: a GPU VPS.
• You use a GPU a few evenings a week and want your setup to be there when you come back: a service that bills per run. Use the build vs rent calculator with your own numbers to see the break-even month.

Questions

Is a private AI server the same as running AI locally?

Local means the model runs on hardware in your home or office. A private AI server is broader: any machine only you use, at home or rented, where your prompts and files are not logged by the operator. Every local setup is private; not every private setup is local.

Do I need a GPU for a private AI server?

For anything above a few billion parameters at a usable speed, yes. Small models run on CPUs and Apple Silicon, but image and video generation and 30B-plus language models want a GPU with enough VRAM to hold the model.

Can a rented server really be private?

It can be private in the ways that matter to most people: an isolated environment, an operator that does not log prompts, and data you can export and delete. It cannot be air-gapped, and the provider operates the hardware. Ask any vendor to state, layer by layer, who can see what.

• Architecture numbers in the calculator come from each model's public config.json or model card, listed per model inside the tool.
• VRAM figures for consumer and workstation cards are manufacturer specifications.