Do You Need a GPU to Self-Host an AI Agent? Almost Certainly Not
The answer for most people reading this is no, and getting it wrong is expensive in one direction only.
If your agent calls Claude, GPT, Gemini or any hosted model API, there is no GPU work happening on your server. Your machine assembles a prompt, makes an HTTPS request, and waits. The GPU that does the thinking belongs to the model provider and is priced into the tokens you’re already paying for.
A $6 CPU VPS is the right purchase. A GPU instance would idle at hundreds of dollars a month doing nothing.
What your agent server actually does
Worth being concrete, because “AI agent” sounds like it should need AI hardware.
An agent orchestrates. It holds messaging connections, reads and writes files, runs tools, keeps conversation state, and calls a model. Every one of those is ordinary server work. The only genuinely GPU-shaped task in the stack — running the model — is happening in someone else’s datacentre.
That’s why the sizing guidance for agents is about RAM, not compute:
| What you’re running | Server |
|---|---|
| Agent, text-only, hosted model API | 4GB |
| Agent with headless browser automation | 8GB |
A single headless Chromium instance wants 1–2GB to itself, which is why browser automation doubles the requirement. Neither number involves a GPU.
When you genuinely do need one
Five cases. Each is real, and each is narrower than it first appears.
1. Running the model locally
The main one. If you’re self-hosting a model with Ollama for privacy, offline operation, or to escape per-token pricing, then the model is on your hardware and VRAM becomes the constraint:
| Model | VRAM | Typical card |
|---|---|---|
| 7–8B quantised | 8–12GB | RTX 3060 12GB, RTX 4090 |
| 13B quantised | 16–20GB | RTX 4090 (24GB) |
| 70B quantised | ~42GB | A100/H100 80GB |
That 42GB figure for a 70B model is from Ollama’s own documentation, and it’s the number that catches people — a 70B model does not fit on a 24GB card.
The important qualifier: you only need the GPU if someone is waiting. CPU inference works; it’s just slow, because generation speed is bounded by memory bandwidth rather than cores. For overnight summarisation, queue enrichment or scheduled analysis, a CPU box at reading-speed output is genuinely fine and dramatically cheaper.
2. Real-time voice
Speech-to-text and text-to-speech in a live conversation have a latency budget measured in hundreds of milliseconds. CPU inference blows through it. This is one of the clearest GPU cases, because the failure isn’t slowness — it’s the interaction not working at all.
3. Image generation
Diffusion models are GPU work by construction. If your agent generates images locally rather than calling an API, you need a card. If it calls an image API, you don’t.
4. Embedding a large document corpus
Nuanced, and often misjudged. Embedding models are small and run acceptably on CPU. The question is volume: embedding a few hundred documents on CPU is an afternoon you can ignore. Embedding a few million is a job that finishes in a week rather than an hour.
And it’s usually a one-off. Renting a GPU for two days to build the index, then serving queries from a CPU box, is the correct shape — not buying a GPU server for a workload that runs once.
5. Fine-tuning
Genuinely needs a GPU, and genuinely needs a big one. It’s also intermittent by nature, which points at the same conclusion as embedding: rent for the run, not for the month.
The mistake that costs the most
Notice the pattern in three of those five cases. The work is bursty, and the cheap answer is not to own the GPU.
An RTX 4090 at RunPod’s Secure Cloud rate is $0.74/hour. Left running for a month that’s roughly $540. Used two hours a day, it’s about $45. Same card, same work, an order of magnitude apart — the difference is entirely whether you destroy the pod when you’re done.
Serverless goes further: it bills only while a request is processing, so an idle endpoint costs nothing. The trade is a cold start while the model loads into VRAM, which is real for a large model — but for an agent that answers a handful of requests an hour, it’s the correct purchase by a wide margin.
We worked the crossovers properly in RunPod vs DigitalOcean GPU Droplets: serverless stays cheaper until you’re running the GPU roughly 60% of the month, which is far later than most people assume. Above that, a dedicated monthly server beats hourly rental outright — RunPod vs Hetzner compares the same 96GB card on both.
What to buy
If you’re running an agent against a hosted model — the common case:
Hostinger is the easiest starting point, with 8GB tiers priced near what some managed platforms charge for a fraction of that and a one-click agent template. Hetzner is the cheapest per gigabyte when its CX tier is in stock — worth checking before you commit, because that tier was showing as unavailable when we last looked, and the CPX line you’d fall back to is several times dearer. That assumes you’re comfortable administering a plain box. Cloudzy bills monthly with no term, which suits a machine you’re still experimenting with.
DigitalOcean is also a common choice here, though its tracked link isn’t working on our end so we’ve left it unlinked.
Size on RAM: 4GB text-only, 8GB with browser automation. Our best VPS for OpenClaw guide covers the brackets, and best VPS for AI agents compares providers.
If you genuinely need a GPU:
RunPod — per-second billing, serverless that costs nothing while idle, and cards from an RTX 4090 through H100s. For the bursty cases above, serverless is the correct product rather than the cheap one.
Before you rent anything, our Ollama install guide covers measuring your own CPU throughput first — it takes two minutes and frequently ends the discussion.
How we checked this
The VRAM figures other than the 70B anchor are conventional sizing guidance. The 42GB figure for a quantised 70B model is from Ollama’s own documented ollama ps output, and the agent RAM floors are the published guidance we use across our AI hosting coverage. RunPod’s $0.74/hour Secure Cloud rate for an RTX 4090 is from its published pricing, and the monthly figures are arithmetic on it at 730 hours and at two hours a day.
What we did not do: we did not benchmark CPU inference speed, so “slow, but fine for batch work” is a qualitative claim rather than a measured one — deliberately, because throughput moves more with quantisation and context length than with the hardware, and a number from our machine wouldn’t transfer to yours. Measure your own with ollama run <model> --verbose, which reports timings per response.
We also haven’t measured serverless cold-start times, which is the variable that decides whether the cheapest option is usable for interactive work. Loading a large model into VRAM is not instant.
The hosting links above are affiliate links, and the honest note is that the recommendation for most readers — a $6 CPU server rather than a GPU instance — is the cheaper one.
FAQ
Does an AI agent need a GPU?
Not if it calls a hosted model API like Claude, GPT or Gemini. The provider runs the GPU and it’s priced into your tokens; your server just orchestrates. Buy 4GB of RAM, or 8GB if you’re doing browser automation.
When do I actually need a GPU?
Running a model locally for interactive use, real-time voice, local image generation, embedding a very large corpus, or fine-tuning. Three of those five are intermittent, which usually means renting rather than owning.
Can I run a local LLM without a GPU?
Yes. CPU inference works for quantised 7–8B models with about 8GB of RAM — it’s just slow, because speed is bounded by memory bandwidth rather than cores. Fine for batch and background work, frustrating for anything a person waits on.
How much VRAM do I need?
8–12GB for a quantised 7–8B model, 16–20GB for a 13B, and about 42GB for a quantised 70B — which means an 80GB card, not a 24GB one.
Is it cheaper to rent a GPU by the hour?
Usually, and by a lot. An RTX 4090 at $0.74/hour is roughly $540 a month always-on against about $45 at two hours a day. Serverless goes further by billing only while processing, at the cost of a cold start.
What about embedding a large document set?
Usually a one-off job. Rent a GPU for the days it takes to build the index, then serve queries from a CPU server — buying a GPU box for a workload that runs once is the expensive version of the right idea.
What’s the cheapest correct setup for a normal agent?
A CPU VPS with 4GB of RAM against a hosted model API, moving to 8GB when you turn on browser automation. That’s a single-figure monthly bill, and it’s what most self-hosted agents should be running on.
