← Benchmarks with RTX 3080

The 35B with parallel slots and image input

Part 8 of 9 · Nov 16, 2026 · 5 min read

In short Parallel slots raise the 35B's throughput from 47 to 71 tok/s at four requests, and image input costs nothing in text speed. It still cannot reuse a prompt and still needs the rest of the stack switched off — but on the corrected journey suite it passes 72 of 81, against the 8B's 32.Full verdict ↓

Post 7 ended with the 35B model out. Two things were still open: whether more parallel slots rescue its throughput, and whether image input, which the 8B on Ollama does not have, changes the picture. I ran both on Strata, the same engine as post 6.

71 tok/saggregate with 4 slots and 4 requests, up from 47 with one
18 tok/swhat each of those 4 requests gets
54 tok/ssingle-stream decode, unchanged by slots or image input
~11 GBof resident services that must be paused to run it
Measured, 35B on Strata 1 slot (post 6) 2 slots, image input 4 slots, image input
Decode at 1k 54.1 tok/s 54.4 tok/s 54.4 tok/s unchanged
Cold first token at 1k 0.754 s 0.754 s 0.754 s unchanged
Warm first token at 1k 0.753 s 0.753 s 0.753 s still no reuse
Aggregate, 1 request 46.9 tok/s 46.9 tok/s 46.5 tok/s
Aggregate, 2 requests 47.2 tok/s 56.6 tok/s 57.0 tok/s +20%
Aggregate, 4 requests 47.1 tok/s 57.0 tok/s 70.8 tok/s +50%

Result 1: parallel slots

Aggregate tokens per second with 4 requests in flight 1k-token prompts. The last bar is the 8B model from [post 6](/blog/benchmarks-rtx-3080/engines-head-to-head), for scale.
35B, 1 slot47.1
35B, 2 slots57.0
35B, 4 slots70.8
8B on llama.cpp, 4 slots226.4
Axis 0 to 230 tok/s. The two models are not comparable in quality, only in how throughput scales.

Slots help, but the gain is modest. Four slots take aggregate throughput to 1.5 times a single stream, against 2.5 times for the 8B in post 6, and each request slows from 46 to about 18 tok/s. A likely reason, if part of the model lives on the CPU as in post 5, is that CPU-side work is limited by memory traffic, and more streams do not make system memory faster. I have not isolated it.

Result 2: image input costs nothing in text speed

The runs are labelled as vision runs, which I take to mean image input was enabled. Text speed with it on is identical to the run without: 54.4 against 54.1 tok/s decode, and the same 0.754 s cold first token. So enabling image input appears free when no image is sent. The runs sent text prompts; what an image itself costs — image encode plus prompt read — is not measured here.

Prompt Cold first token Prompt speed
811 tokens 0.754 s about 1,080 tok/s
3,100 tokens 2.444 s about 1,270 tok/s
6,150 tokens 4.581 s about 1,340 tok/s

Result 3: the price of testing it

Running a model this size means making room for it. bench-mode.sh scales seven deployments to zero and restores them afterwards.

Paused while benchmarking Why
The LLM server (GPU) The 35B needs the card
Embeddings and both rerankers RAM
Speech to text and text to speech RAM
The task classifier RAM

About 11 GB of resident services go down. Until bench-mode.sh off, the local LLM, embeddings, reranking, speech and the shadow-mode classifier are all unavailable. This is post 1's criterion in practice: a model that cannot coexist with the rest of the stack is not a candidate for an always-on agent, however fast it is.

Gotchas

Gotcha Effect
Benchmark mode takes the stack down Every local service stops until it is switched off. Do not run it while anything depends on them
The warm and cold first tokens are identical The engine does not reuse a prompt, with or without slots
Four slots on a CPU-bound model Throughput rises 50%, but per-request speed falls by more than half
Image latency not measured The speed runs use text prompts; what an image itself costs is a separate experiment

Where it stands, and my verdict

How the 35B has fared against the criteria from post 1:

Criterion The 35B so far
Fits, resident, shareable Needs about 11 GB of other services paused no
Fast enough 54 tok/s single, 71 aggregate at 4 slots. Cannot reuse a prompt partly
Decisions I would trust 72/81 on the corrected journey suite, above the 8B's 32 (evals). Not re-run with slots better
Easy to run A separate engine and a mode that stops the stack no

Verdict: slots and vision do not rescue the 35B's fit. They improve throughput by half and add image input at no text cost. It still cannot reuse a prompt and still needs the stack switched off to run. On decisions, the corrected journey suite reverses the earlier result: the 35B passes 72 of 81 and the 8B 32, so the blocker is fit, not quality. The question for the next post is different: is there a model that is fast enough, small enough, and good enough to sit beside everything else?