The 35B with parallel slots and image input
In short Parallel slots raise the 35B's throughput from 47 to 71 tok/s at four requests, and image input costs nothing in text speed. It still cannot reuse a prompt and still needs the rest of the stack switched off — but on the corrected journey suite it passes 72 of 81, against the 8B's 32.Full verdict ↓
Post 7 ended with the 35B model out. Two things were still open: whether more parallel slots rescue its throughput, and whether image input, which the 8B on Ollama does not have, changes the picture. I ran both on Strata, the same engine as post 6.
| Measured, 35B on Strata | 1 slot (post 6) | 2 slots, image input | 4 slots, image input | |
|---|---|---|---|---|
| Decode at 1k | 54.1 tok/s | 54.4 tok/s | 54.4 tok/s | unchanged |
| Cold first token at 1k | 0.754 s | 0.754 s | 0.754 s | unchanged |
| Warm first token at 1k | 0.753 s | 0.753 s | 0.753 s | still no reuse |
| Aggregate, 1 request | 46.9 tok/s | 46.9 tok/s | 46.5 tok/s | |
| Aggregate, 2 requests | 47.2 tok/s | 56.6 tok/s | 57.0 tok/s | +20% |
| Aggregate, 4 requests | 47.1 tok/s | 57.0 tok/s | 70.8 tok/s | +50% |
Result 1: parallel slots
Slots help, but the gain is modest. Four slots take aggregate throughput to 1.5 times a single stream, against 2.5 times for the 8B in post 6, and each request slows from 46 to about 18 tok/s. A likely reason, if part of the model lives on the CPU as in post 5, is that CPU-side work is limited by memory traffic, and more streams do not make system memory faster. I have not isolated it.
Result 2: image input costs nothing in text speed
The runs are labelled as vision runs, which I take to mean image input was enabled. Text speed with it on is identical to the run without: 54.4 against 54.1 tok/s decode, and the same 0.754 s cold first token. So enabling image input appears free when no image is sent. The runs sent text prompts; what an image itself costs — image encode plus prompt read — is not measured here.
| Prompt | Cold first token | Prompt speed |
|---|---|---|
| 811 tokens | 0.754 s | about 1,080 tok/s |
| 3,100 tokens | 2.444 s | about 1,270 tok/s |
| 6,150 tokens | 4.581 s | about 1,340 tok/s |
Result 3: the price of testing it
Running a model this size means making room for it. bench-mode.sh scales seven deployments to zero and restores them afterwards.
| Paused while benchmarking | Why |
|---|---|
| The LLM server (GPU) | The 35B needs the card |
| Embeddings and both rerankers | RAM |
| Speech to text and text to speech | RAM |
| The task classifier | RAM |
About 11 GB of resident services go down. Until bench-mode.sh off, the local LLM, embeddings, reranking, speech and the shadow-mode classifier are all unavailable. This is post 1's criterion in practice: a model that cannot coexist with the rest of the stack is not a candidate for an always-on agent, however fast it is.
Gotchas
| Gotcha | Effect |
|---|---|
| Benchmark mode takes the stack down | Every local service stops until it is switched off. Do not run it while anything depends on them |
| The warm and cold first tokens are identical | The engine does not reuse a prompt, with or without slots |
| Four slots on a CPU-bound model | Throughput rises 50%, but per-request speed falls by more than half |
| Image latency not measured | The speed runs use text prompts; what an image itself costs is a separate experiment |
Where it stands, and my verdict
How the 35B has fared against the criteria from post 1:
| Criterion | The 35B so far | |
|---|---|---|
| Fits, resident, shareable | Needs about 11 GB of other services paused | no |
| Fast enough | 54 tok/s single, 71 aggregate at 4 slots. Cannot reuse a prompt | partly |
| Decisions I would trust | 72/81 on the corrected journey suite, above the 8B's 32 (evals). Not re-run with slots | better |
| Easy to run | A separate engine and a mode that stops the stack | no |
Verdict: slots and vision do not rescue the 35B's fit. They improve throughput by half and add image input at no text cost. It still cannot reuse a prompt and still needs the stack switched off to run. On decisions, the corrected journey suite reverses the earlier result: the 35B passes 72 of 81 and the 8B 32, so the blocker is fit, not quality. The question for the next post is different: is there a model that is fast enough, small enough, and good enough to sit beside everything else?