Raw speed against the ceilings
In short The 8B generates at 79% of the bandwidth ceiling and reads prompts faster than my ceiling allowed, so the ceiling was wrong. The 35B runs but is 1.8x slower to generate.Full verdict ↓
With the model chosen, I wanted to know how close the card gets to what the datasheet allows. I built the sweep from post 2 and ran it with the GPU free.
| Measured on the 8B and the 35B | Result | Ceiling or comparison | |
|---|---|---|---|
| 8B generation | 119.3 tok/s | 151 tok/s ceiling | 79% |
| 8B prompt processing | 4,580 tok/s | about 3,660 tok/s ceiling | ceiling was wrong |
| 35B generation, best split | 67.5 tok/s | 8B does 119.3 | 1.8× slower |
| 35B prompt processing | 852 tok/s | 8B does 4,580 | 5.4× slower |
| CPU threads on the 35B | 6 to 8 is best | 16 threads costs 26% | more is worse |
The physics: two ceilings
Generating a token needs every weight read from memory once, with almost no arithmetic per byte. So the speed limit is bandwidth over weight bytes:
generation tok/s ≤ 760 GB/s ÷ bytes of weights
For Qwen3 8B Q4_K_M, 4.68 GiB or 5.02 GB, that is 151 tok/s. Reading a prompt is the opposite: all its tokens go through together, so each weight is reused many times and the limit is compute, about 2 operations per parameter per token. For this 8.19B model that is 16.4 GFLOP per token, and at the FP16 tensor rate I assumed in post 1 (about 60 TFLOPS), a ceiling near 3,660 tok/s.
What was actually run
sweep.shSo the findings cover one dense 8B model and one 35B mixture-of-experts model at several CPU/GPU splits. The other two model families are untested.
Result 1: the dense 8B model
| Qwen3 8B Q4_K_M, all layers on GPU | Speed | Ceiling | |
|---|---|---|---|
| Generation, 128 tokens | 119.27 ± 0.05 tok/s | 151 | 79% |
| Prompt processing, 512 tokens | 4,580 ± 10 tok/s | about 3,660 | 125%, ceiling wrong |
| Prompt processing, 4,096 tokens | 4,224 ± 2 tok/s | about 3,660 | 115% |
Generation behaves: 79% of the ceiling is a healthy result. The prompt ceiling does not. 4,580 tok/s × 16.4 GFLOP is about 75 TFLOPS of useful work, 25% above the figure I assumed. The likely explanation is that llama.cpp's 4-bit kernels use the tensor cores' integer path, which has a higher peak than FP16. I have not confirmed that with a profiler, but the lesson stands: a ceiling is only as good as the assumption under it.
Result 2: a 35B model on a 10 GB card
Qwen3.6 35B-A3B is a mixture-of-experts model: 34.7B parameters in a 20.6 GiB file, but only about 3B active per token. -ncmoe N keeps the experts of the first N layers on the CPU so the rest fits. The more experts on the CPU, the slower:
| Experts on CPU (layers) | Prompt 512 | Prompt 4096 | Generation | vs best |
|---|---|---|---|---|
| 24 | no result | no result | no result | did not fit |
| 28 | 852 tok/s | 840 tok/s | 67.5 tok/s | best |
| 32 | 784 tok/s | 766 tok/s | 62.6 tok/s | −7% |
| 40 | 670 tok/s | 654 tok/s | 54.0 tok/s | −20% |
Threads: more is not better
Using all 16 hardware threads on 8 physical cores cost a quarter of the generation speed. Generation that runs on the CPU is limited by memory traffic, and extra threads contend for it. I did not record system RAM bandwidth, so I cannot draw that ceiling yet.
Dense 8B against the 35B model
| Qwen3 8B, all GPU | 35B-A3B, best split | Ratio | |
|---|---|---|---|
| File size | 4.68 GiB | 20.60 GiB | 4.4× |
| Prompt speed | 4,580 tok/s | 852 tok/s | 5.4× slower |
| Generation | 119.3 tok/s | 67.5 tok/s | 1.8× slower |
Gotchas
| Gotcha | Effect |
|---|---|
| Only 2 repetitions per config | Dense results are tight (±0.05), but MoE prompt speed varied by up to ±28 tok/s. Treat small gaps there as noise |
| Thread count | Using every hardware thread was 26% slower than 6 to 8 |
| One out-of-memory per run | Configurations are separate runs so one failure does not hide the rest |
| Two model families skipped | Nothing was measured for Qwen3-30B-A3B or gpt-oss-20b |
Where it stands, and my verdict
The datasheet's predictions against the card so far:
| Question from post 1 | Paper said | Measured | |
|---|---|---|---|
| Fits | 7 to 8B at 4-bit with room | 8.3 GB at 40k (post 4) | held |
| Generation speed | at most 151 tok/s for this model | 119.3 tok/s, 79% | held |
| Prompt speed | about 3,660 tok/s | 4,580 tok/s | wrong, ceiling assumption |
| Sharing | Memory must sum under 10 GB | One LLM stays, the rest left the GPU (post 3) | held |
Verdict: the card is not the bottleneck for an 8B model. It generates at 79% of the physical limit and reads prompts at over 4,000 tok/s. A 35B model runs but at 1.8× slower generation and 5.4× slower prompts, so it has to buy something to be worth it. That is post 7. First, the question I put off in post 3: how much speed does the engine itself cost?