Running Ollama on a GPU Server: Why It's Still Using the CPU
You rented a GPU server, installed Ollama, and it works. Responses come back. Nothing errors.
It’s running on the CPU.
This is the most common failure in GPU-hosted Ollama, and it’s insidious precisely because nothing breaks — you just quietly pay for a card you aren’t using while getting the throughput of a machine you could have rented for a tenth of the price.
First, find out
Ollama answers this directly:
ollama ps
NAME ID SIZE PROCESSOR UNTIL
llama3:70b bcfb190ca3a7 42 GB 100% GPU 4 minutes from now
The PROCESSOR column is the whole answer. Per Ollama’s documentation:
-
100% GPU— loaded entirely into VRAM. This is what you want. -
100% CPU— loaded entirely into system memory. You’re paying for nothing. -
48%/52% CPU/GPU— split across both, which is slow in a way that feels like the GPU isn’t helping, because mostly it isn’t.
Run it while a model is loaded — ollama ps reports what’s currently in memory, so query the model first.
A partial split usually means the model doesn’t fit in VRAM. 100% CPU usually means Ollama can’t see the card at all, which is the rest of this article.
Check the card is supported
Before debugging anything, confirm the hardware and driver clear Ollama’s floor. From its documentation:
“Ollama supports Nvidia GPUs with compute capability 5.0+ and driver version 550 and newer. Nvidia GPUs with compute capability 5.0 through 6.2 require driver version 570 or newer.”
Common rental cards and their compute capability:
| Compute capability | Cards |
|---|---|
| 12.0 | RTX 5060 → 5090, RTX PRO Blackwell (4000/4500/5000/6000) |
| 9.0 | H100, H200 |
| 8.9 | RTX 4090, 4080, 4070 series, L4, L40, RTX 6000 |
| 8.6 | RTX 3090 Ti, RTX 3090, 3080, 3070, 3060 |
Then check the driver:
nvidia-smi
If that command isn’t found or errors, the driver isn’t installed and nothing else matters. If it prints a table, note the driver version in the header — below 550 and Ollama won’t use the card, regardless of how new the GPU is. On rented instances an old driver is common enough to check first.
If you’re running Ollama in Docker
This is where most people lose the GPU, and the reason is simple: installing the NVIDIA driver on the host does not give containers access to it. Docker needs the NVIDIA Container Toolkit as well, and without it the container sees no device and Ollama falls back to CPU without complaint.
Ollama’s documented setup, on Debian or Ubuntu:
# 1. Add the toolkit repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
| sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
| sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
# 2. Install it
sudo apt-get install -y nvidia-container-toolkit
# 3. Point Docker at the NVIDIA runtime
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
On RHEL-family systems the equivalent repo file goes to /etc/yum.repos.d/ and you install with yum or dnf.
Then start the container with the GPU flag:
docker run -d --gpus=all \
-v ollama:/root/.ollama \
-p 127.0.0.1:11434:11434 \
--name ollama ollama/ollama
Two things about that command.
--gpus=all is not optional. Omit it and everything above is still correct and the container still won’t see the card.
And note the port binding. Ollama’s documented example publishes -p 11434:11434, which binds all interfaces — on a public GPU server that’s an unauthenticated inference endpoint facing the internet. Bind it to 127.0.0.1 as above. This matters more on a GPU box than anywhere else, because the hardware is expensive and 31% of scanned Ollama servers answered a stranger’s prompt.
Verify the container can see the card:
docker exec -it ollama nvidia-smi
If that fails inside the container while working on the host, the toolkit step didn’t take.
Size the model to the VRAM
The model has to fit. If it doesn’t, Ollama splits it and you get that 48%/52% line — technically working, practically pointless.
| Model | VRAM needed | Cards that fit it |
|---|---|---|
| 7–8B quantised | ~5–8GB | RTX 3060 12GB and up |
| 13B quantised | ~10–14GB | RTX 3090/4090 (24GB) |
| 30–34B quantised | ~20–24GB | RTX 4090, L40S (48GB) |
| 70B quantised | ~42GB | A100/H100 80GB, or multi-GPU |
That 70B figure is from Ollama’s own ollama ps example, which shows llama3:70b occupying 42 GB at 100% GPU — a useful anchor, because 70B is the size people most often assume will fit on a 24GB card and it does not.
Leave headroom above the model size for context. A long context window costs additional VRAM beyond the weights, and it’s the usual reason a model that loaded fine yesterday spills to CPU today.
When it worked, then stopped
A specific and maddening failure with a documented fix. From Ollama’s troubleshooting:
“If Ollama initially works on the GPU in a docker container, but then switches to running on CPU after some period of time with errors in the server log reporting GPU discovery failures, this can be resolved by disabling systemd cgroup management in Docker.”
Edit /etc/docker/daemon.json on the host:
{
"exec-opts": ["native.cgroupdriver=cgroupfs"]
}
Then restart Docker. If your GPU box has been silently degrading to CPU after hours of uptime, this is very likely why — and it’s almost impossible to guess at.
Reading the server logs, GPU initialisation failures surface as numeric error codes. Ollama documents several: 3 (not initialized), 46 (device unavailable), 100 (no device), 999 (unknown). A 100 points at discovery — the container or driver isn’t presenting a device. Ollama’s general advice is that running the latest driver yields the best results.
Leaving it on is the expensive mistake
The other way to waste a GPU server is to rent it by the hour and never turn it off.
Take an RTX 4090 at RunPod’s Secure Cloud rate of $0.74/hour. Left running for a month that’s roughly $540. Used two hours a day, it’s about $45. Same card, same work, an order of magnitude apart — and the difference is entirely whether you destroy the pod when you’re done.
Three patterns, in rough order of how well they suit a personal setup:
- Serverless — bills only while processing a request, so idle costs nothing. The trade is a cold start while the model loads into VRAM, which is real for a 42GB model.
- Per-session pods — spin up, work, destroy. Cheapest for interactive sessions if you’re disciplined about the destroying.
- Always-on — only rational above roughly 60% duty cycle, and at that point a dedicated monthly server usually beats hourly rental outright.
We worked those crossovers properly in RunPod vs DigitalOcean GPU Droplets and RunPod vs Hetzner for Ollama — the short version is that the crossover arrives later than people assume, so per-hour usually wins.
Where to run it
RunPod is the sensible default for Ollama on a GPU: per-second billing, a serverless tier that costs nothing between requests, and cards from an RTX 4090 at $0.74/hour on Secure Cloud through H100s. Its Community Cloud tier is cheaper again on third-party hardware, which is fine for experimentation and unsuitable for anything with a compliance story.
If the GPU will genuinely run around the clock, a dedicated monthly server is cheaper than any hourly marketplace — RunPod vs Hetzner compares the same 96GB card on both.
And if you haven’t decided whether you need a GPU at all, our Ollama install guide covers the CPU track honestly — for batch and background work, a CPU server at reading-speed output is often enough.
How we checked this
The ollama ps PROCESSOR values, the compute-capability and driver-version requirements (5.0+ and driver 550+, with 570+ for 5.0–6.2), the NVIDIA Container Toolkit sequence, the --gpus=all flag, the cgroup-driver fix for CPU fallback, and the GPU initialisation error codes are all from Ollama’s own documentation, read in August 2026. The 42GB figure for a 70B model is from Ollama’s own ollama ps example output. RunPod’s rates are its published Secure Cloud pricing.
What we did not do: we did not benchmark throughput on any of these cards, and the VRAM figures other than the 70B anchor are conventional sizing guidance rather than our measurements. Quantisation level and context length move them meaningfully — a longer context window is the usual reason a model that fitted yesterday spills to CPU today. Check your own with ollama ps rather than trusting any table, including this one.
We also haven’t measured serverless cold-start times, which is the variable that decides whether the cheapest billing model is usable for interactive work. Loading 42GB into VRAM is not instant, and only your model and provider can tell you how long it takes.
The RunPod link above is an affiliate link. The most valuable thing on this page — ollama ps, to find out you’ve been paying for a GPU you weren’t using — costs nothing.
FAQ
How do I know if Ollama is using my GPU?
Run ollama ps while a model is loaded and read the PROCESSOR column: 100% GPU means it’s fully in VRAM, 100% CPU means the GPU isn’t being used at all, and a split like 48%/52% CPU/GPU means the model didn’t fit.
Why is Ollama using the CPU when I have a GPU?
Usually one of three things: the NVIDIA driver is older than version 550, the model is too large for available VRAM, or — in Docker — the NVIDIA Container Toolkit was never installed, so the container can’t see the device. Installing the driver on the host is not sufficient for containers.
What driver version does Ollama need?
550 or newer. Cards with compute capability 5.0 through 6.2 need 570 or newer. Check with nvidia-smi; a driver below that floor means the card won’t be used no matter how capable it is.
Do I need --gpus=all with Docker?
Yes. Without it the container has no GPU access even with the toolkit installed and configured, and Ollama falls back to CPU silently rather than erroring.
How much VRAM do I need for a 70B model?
About 42GB quantised, per Ollama’s own example output — so an 80GB A100 or H100, or multiple GPUs. It does not fit on a 24GB card, which is the most common mistaken assumption in this area.
My GPU worked and then Ollama switched to CPU. Why?
A documented Docker issue: add "exec-opts": ["native.cgroupdriver=cgroupfs"] to /etc/docker/daemon.json and restart Docker. The symptom is GPU discovery failures in the server log after a period of correct operation.
Is it cheaper to leave the GPU server running?
Rarely. An RTX 4090 at $0.74/hour is about $540 a month always-on against roughly $45 at two hours a day. Use serverless or destroy the pod between sessions unless the card is busy most of the time.
