← Benchmarks with RTX 3080

vLLM first, and why it did not stay

Part 3 of 9 · Apr 19, 2026 · 5 min read

In short vLLM claimed 8.5 of 10 GiB for a 7B model and sharing failed twice, so Ollama became the single GPU tenant.Full verdict ↓

The goal from post 1 was one card carrying many jobs. The first engine I reached for was the one built for serious inference, and it taught me the most important rule about this card quickly.

8.5 GiBclaimed by vLLM at gpu_memory_utilization=0.90
1.5 GiBleft for everything else
4weeks that GPU time-slicing lasted
0requests a resident chat model received in 72 hours
VRAM budget over time GiB of the 10 GiB card. From my GPU budget notes: Aug and Sep are planned allocations, Oct is measured and derived parts.
Jun · vLLM
vLLM 8.51.5 free
Aug · Ollama + STT
OCR 4.0KV 1.7WhisperLive 3.0
Sep · Ollama
OCR 4.0KV 1.73.9 free
Oct · Qwen3 8B
weights 4.7KV 3.01.4 free
vLLM (all in)model weightsKV cacheoverheadembedderreal-time STT
  1. Ray Serve + vLLM. An inference image built in CI, served through KubeRay, with OpenTelemetry and monitoring from day one.
  2. Tuned and pinned. Serving Qwen2.5-7B-Instruct (AWQ 4-bit): memory from 0.85 to 0.90, prefix caching on, context capped at 4,096, concurrency limit 8 to 128. vLLM pinned back from 0.8.5 to 0.7.3.
  3. Ollama arrives. Added next to vLLM as a multi-model server, first for the embedder. The notes already say one 7B model saturates the card.
  4. Chat out, OCR in. A general chat model on Ollama had seen zero requests in 72 hours, so its slot became an OCR model. By then vLLM is disabled.
  5. A second GPU tenant. Real-time speech to text added alongside Ollama using GPU time-slicing.
  6. Removed again. Ollama alone did not need time-slicing, so both went.

The physics: where the memory goes

A model on a GPU needs three kinds of memory:

  • Weights: parameters × bytes per parameter. A 7B model is 14 GB in FP16 and roughly a third of that at 4 bits.
  • KV cache: the attention keys and values for every token in the context, which grows with context length and with the number of requests in flight.
  • Overhead: the CUDA context, framework buffers and activations, commonly several hundred megabytes up to a gigabyte.

Only the first is fixed. The KV cache is a budget you choose, and that is where engines differ.

Round 1: vLLM on its own

vLLM is built for throughput: continuous batching, a paged KV cache, many concurrent requests. On paper it is the serious option. On this card it had one visible problem.

The top row of the chart above is the whole story of round 1. gpu_memory_utilization=0.90 tells vLLM to claim 90% of the card up front and carve its own pool for the KV cache. That is good engineering for a dedicated server and the wrong shape for a shared card: the weights of a 7B 4-bit model are a fraction of the 8.5 GiB it took, and the rest was reserved for concurrency I was not using.

Round 2: sharing the card

Ollama loads and unloads models on demand, so several can coexist. I had put an embedder on it earlier, then an OCR model, then added real-time speech to text through GPU time-slicing, which makes the cluster report two GPUs so two pods can each request one.

Gotcha What happened What I do now
Time-slicing does not isolate memory Both pods became schedulable, but they still shared one 10 GiB pool. The 3 GiB for speech to text was never verified under load Treat the budget as a sum I must check with nvidia-smi running both
Config outside the chart The device-plugin patch lived in kube-system, not in Helm Document it, or avoid it
A resident model nobody called The chat model had zero requests in 72 hours Measure traffic before assigning VRAM. keepAlive went from 24 h to 10 min

The experiment lasted about four weeks and was then removed. It was a useful failure: it showed that "schedulable" and "fits" are different questions.

What I produced

  • A GPU budget document stating every allocation and how to verify it (ollama ps, nvidia-smi).
  • A rule: nothing gets VRAM without measured traffic.
  • The decision to make Ollama the single GPU tenant.

Where it stands, and my verdict

Against the starting position from post 1:

Criterion Status
Fits, resident, shareable vLLM took 8.5 of 10 GiB. Time-slicing shared the schedule, not the memory sharing failed twice
Fast enough Not measured yet. vLLM was never benchmarked head to head untested
Decisions I would trust Not measured yet untested
Easy to run vLLM needed Ray, KubeRay, a custom image and a version pin. Ollama is one deployment Ollama simpler
Need vLLM + Ray Ollama
Share the GPU with other models wants the whole card loads on demand
Several models resident no up to two
Many concurrent requests its strength (not benchmarked here) one parallel request per model

Verdict: Ollama, for operational reasons. I never benchmarked vLLM, so I will not claim it is slower. What I can say is that it wants the whole card, my workload is one user plus agents, and everything else on this GPU got simpler once it left. vLLM would win on a dedicated card with many concurrent users.

Next: with one engine chosen, which model fits.