← Benchmarks with RTX 3080

Engines head to head

Part 6 of 9 · Nov 11, 2026 · 6 min read

In short Stay on Ollama: it costs 2 to 8% decode speed. In my run Strata did not reuse a prompt, which made a warm start 42x slower.Full verdict ↓

Post 3 chose Ollama for operations and left a doubt: does it cost speed compared with the llama.cpp server it runs on? I put the same model, prompts and client in front of both, then later added a third engine.

2–8%decode speed Ollama gives up to llama.cpp
< 6%first-token time Ollama gives up
2.5×aggregate throughput from a 4-slot llama.cpp server
42×slower warm first token on Strata, which did not reuse a prompt in my run
Engine Decode at 1k Warm first token, 14k 4 requests at once
Ollama, 8B 101 tok/s 0.20 s 77 tok/s aggregate keep
llama.cpp server, 8B 109 tok/s 0.16 s 226 tok/s with 4 slots if agents run in parallel
Strata, 35B 54 tok/s 8.27 s 47 tok/s, flat no prompt reuse in this run
  1. Ollama against llama.cpp on Qwen3 8B, then llama.cpp with 4 parallel slots.
  2. A third engine, Strata, running the 35B mixture-of-experts model, through the same harness.

Ollama against llama.cpp

Same model, same prompts, same client:

Context Prompt tokens Decode: Ollama Decode: llama.cpp Cold first token: Ollama Cold first token: llama.cpp
1k about 800 101.3 tok/s 109.2 tok/s +7.8% 0.356 s 0.350 s
8k 6,138 89.0 tok/s 93.0 tok/s +4.5% 1.879 s 1.854 s
14k about 10,460 80.2 tok/s 82.0 tok/s +2.1% 3.430 s 3.253 s

A warm first token (the same prompt again, which is what an agent resending its system prompt sees) is 0.16 to 0.20 s on both. Against the 151 tok/s ceiling from post 5:

Decode speed against the bandwidth ceiling Qwen3 8B Q4_K_M; the line is 151 tok/s
Ceiling151
llama-bench, empty119.3 · 79%
llama-server, 1k109.2 · 72%
Ollama, 1k101.3 · 67%
llama-server, 14k82.0 · 54%
Ollama, 14k80.2 · 53%

Speed falls as context grows because every generated token must also read the whole KV cache, so the bytes per token rise even though the weights do not. The gap between engines shrinks for the same reason: at long context the KV reads dominate and the engine matters less.

Concurrency is where the gap opens

Aggregate tokens per second with N requests in flight 1k-token prompts
Ollama, 1 / 2 / 483 · 75 · 77
llama-server, default79 · 82 · 82
llama-server, 4 slots89 · 158 · 226
Bars show the 4-in-flight case. Axis 0 to 230 tok/s

With one slot per model, extra requests queue and aggregate speed stays flat near 80 tok/s. With four slots, 4 requests in flight deliver 226 tok/s, 2.5 times a single stream, while each request slows from 89 to about 57 tok/s. One pass over the weights can serve several streams, so batching buys throughput at the cost of per-request speed. The 4-slot run covered only the 1k context point.

A third engine: Strata

In a later run I ran Strata, a CUDA inference engine, on the 35B model through the same harness. The model differs from the other runs, so compare engine behaviour and not absolute speed.

Ollama, 8B llama.cpp, 35B best split Strata, 35B
Decode at 1k 101.3 tok/s 67.5 tok/s 54.1 tok/s
Decode at 14k 80.2 tok/s not measured 44.2 tok/s
Cold first token, 8k 1.88 s not measured 4.58 s
Warm first token, 14k 0.20 s not measured 8.27 s
Aggregate, 4 in flight 77 tok/s not measured 47 tok/s, flat
Seconds to the first token on a prompt it has seen before Warm start: what an agent loop sees when it resends the same system prompt
8B on Ollama, 1k0.16
35B on Strata, 1k0.75
8B on Ollama, 14k0.20
35B on Strata, 14k8.27
Axis 0 to 8.3 s

On Strata a warm first token costs the same as a cold one at every length (4.58 s against 4.58 s at 8k). In this run it did not reuse a cached prompt, while Ollama does. I ran only the 35B pack, so I cannot say whether that is the engine, the pack or a setting, and I have not checked. For an agent that resends a 10k-token catalog every turn, that is a 42× difference at 14k. Strata does process long prompts about 1.5 times faster than llama.cpp's best split (about 1,300 against 840 tok/s) and generates about 20% slower, with a different quantization, so treat that as rough.

Gotchas

Gotcha Effect
The GPU is shared Run ollama ps first and make sure it is empty, and delete the benchmark pod afterwards
Strata plans from the whole node's RAM The pod has no memory limit, a real risk on a shared node
First start downloads the model A 40 GiB volume and a long first boot
Model differs between engines Strata ran the 35B model, the others the 8B

Where it stands, and my verdict

Question Answer
Fits 8B at 40k, 8.3 GB (post 4) held
Generation 79% of the ceiling empty, 53 to 72% in real use held
Prompt speed Over 4,000 tok/s, ceiling assumption was wrong (post 5) corrected
What does Ollama cost? 2 to 8% decode, under 6% first token small
Can an engine reuse a prompt? Ollama and llama.cpp yes. Strata, with the 35B pack in this run, no decides it

Verdict: stay on Ollama. It costs 2 to 8% against the engine underneath, and gives me model management for free. If agents ever run in parallel, a llama-server with 4 slots gives 2.5× the aggregate. Strata, as I ran it, is out for agent loops until a prompt can be reused, because that single behaviour turns a 0.2 s step into 8 s. On the models I have run, speed is settled. Whether a bigger model is worth its cost is a quality question, and that is next.