← Benchmarks with RTX 3080

Raw speed against the ceilings

Part 5 of 9 · Nov 11, 2026 · 6 min read

In short The 8B generates at 79% of the bandwidth ceiling and reads prompts faster than my ceiling allowed, so the ceiling was wrong. The 35B runs but is 1.8x slower to generate.Full verdict ↓

With the model chosen, I wanted to know how close the card gets to what the datasheet allows. I built the sweep from post 2 and ran it with the GPU free.

4,580 tok/sprompt processing, Qwen3 8B fully on the GPU
119 tok/sgeneration, same model
79%of the 151 tok/s bandwidth ceiling
67 tok/sa 35B mixture-of-experts model with experts partly on CPU
Measured on the 8B and the 35B Result Ceiling or comparison
8B generation 119.3 tok/s 151 tok/s ceiling 79%
8B prompt processing 4,580 tok/s about 3,660 tok/s ceiling ceiling was wrong
35B generation, best split 67.5 tok/s 8B does 119.3 1.8× slower
35B prompt processing 852 tok/s 8B does 4,580 5.4× slower
CPU threads on the 35B 6 to 8 is best 16 threads costs 26% more is worse

The physics: two ceilings

Generating a token needs every weight read from memory once, with almost no arithmetic per byte. So the speed limit is bandwidth over weight bytes:

generation tok/s  ≤  760 GB/s ÷ bytes of weights

For Qwen3 8B Q4_K_M, 4.68 GiB or 5.02 GB, that is 151 tok/s. Reading a prompt is the opposite: all its tokens go through together, so each weight is reused many times and the limit is compute, about 2 operations per parameter per token. For this 8.19B model that is 16.4 GFLOP per token, and at the FP16 tensor rate I assumed in post 1 (about 60 TFLOPS), a ceiling near 3,660 tok/s.

What was actually run

Planned configurations 15 planned in sweep.sh
Produced numbers6
Skipped: model file missing8
Ran, returned nothing1
Skipped: both Qwen3-30B-A3B and gpt-oss-20b. The empty one was the 35B model with only 24 layers of experts on CPU, which did not fit

So the findings cover one dense 8B model and one 35B mixture-of-experts model at several CPU/GPU splits. The other two model families are untested.

Result 1: the dense 8B model

Qwen3 8B Q4_K_M, all layers on GPU Speed Ceiling
Generation, 128 tokens 119.27 ± 0.05 tok/s 151 79%
Prompt processing, 512 tokens 4,580 ± 10 tok/s about 3,660 125%, ceiling wrong
Prompt processing, 4,096 tokens 4,224 ± 2 tok/s about 3,660 115%
Generation speed against the bandwidth ceiling Qwen3 8B Q4_K_M, tokens per second
Ceiling151
Measured, empty context119.3 · 79%

Generation behaves: 79% of the ceiling is a healthy result. The prompt ceiling does not. 4,580 tok/s × 16.4 GFLOP is about 75 TFLOPS of useful work, 25% above the figure I assumed. The likely explanation is that llama.cpp's 4-bit kernels use the tensor cores' integer path, which has a higher peak than FP16. I have not confirmed that with a profiler, but the lesson stands: a ceiling is only as good as the assumption under it.

Result 2: a 35B model on a 10 GB card

Qwen3.6 35B-A3B is a mixture-of-experts model: 34.7B parameters in a 20.6 GiB file, but only about 3B active per token. -ncmoe N keeps the experts of the first N layers on the CPU so the rest fits. The more experts on the CPU, the slower:

Experts on CPU (layers) Prompt 512 Prompt 4096 Generation vs best
24 no result no result no result did not fit
28 852 tok/s 840 tok/s 67.5 tok/s best
32 784 tok/s 766 tok/s 62.6 tok/s −7%
40 670 tok/s 654 tok/s 54.0 tok/s −20%

Threads: more is not better

Generation speed by thread count 35B-A3B, 28 layers of experts on CPU; the CPU has 8 physical cores
6 threads67.4
8 threads67.5
16 threads50.0 · −26%
Prompt processing did not care: 862, 852 and 863 tok/s. Axis 0 to 69 tok/s

Using all 16 hardware threads on 8 physical cores cost a quarter of the generation speed. Generation that runs on the CPU is limited by memory traffic, and extra threads contend for it. I did not record system RAM bandwidth, so I cannot draw that ceiling yet.

Dense 8B against the 35B model

Qwen3 8B, all GPU 35B-A3B, best split Ratio
File size 4.68 GiB 20.60 GiB 4.4×
Prompt speed 4,580 tok/s 852 tok/s 5.4× slower
Generation 119.3 tok/s 67.5 tok/s 1.8× slower

Gotchas

Gotcha Effect
Only 2 repetitions per config Dense results are tight (±0.05), but MoE prompt speed varied by up to ±28 tok/s. Treat small gaps there as noise
Thread count Using every hardware thread was 26% slower than 6 to 8
One out-of-memory per run Configurations are separate runs so one failure does not hide the rest
Two model families skipped Nothing was measured for Qwen3-30B-A3B or gpt-oss-20b

Where it stands, and my verdict

The datasheet's predictions against the card so far:

Question from post 1 Paper said Measured
Fits 7 to 8B at 4-bit with room 8.3 GB at 40k (post 4) held
Generation speed at most 151 tok/s for this model 119.3 tok/s, 79% held
Prompt speed about 3,660 tok/s 4,580 tok/s wrong, ceiling assumption
Sharing Memory must sum under 10 GB One LLM stays, the rest left the GPU (post 3) held

Verdict: the card is not the bottleneck for an 8B model. It generates at 79% of the physical limit and reads prompts at over 4,000 tok/s. A 35B model runs but at 1.8× slower generation and 5.4× slower prompts, so it has to buy something to be worth it. That is post 7. First, the question I put off in post 3: how much speed does the engine itself cost?