← Benchmarks with RTX 3080

Fitting a real model in 10 GB

Part 4 of 9 · Sep 26, 2026 · 4 min read

In short Qwen3 8B at 40k context with a q8_0 KV cache: the only candidate that is fully resident and best at choosing tools.Full verdict ↓

With Ollama as the only GPU tenant, one question remained: which model do I leave resident? The card gave very different answers for three candidates.

12.3 tok/sa 14B model with 21% pushed onto the CPU
~100 tok/sa 9B model, fully on the GPU
~90 tok/sQwen3 8B at 40k context, fully on the GPU
5.6 GiBwhat an fp16 KV cache at 40k would have cost

The three candidates

Generation speed by model tokens per second, one stream, measured on this card
qwen2.5:14b · 16k12.3
ornith-1.5:9b · 64k~100
qwen3:8b · 40k~90
axis 0 to 120 tok/s
Model Context VRAM On GPU Tool-selection suite
qwen2.5:14b 16k over budget 79%, 21% on CPU not scored
ornith-1.5:9b 64k 6.9 GB 100% 4 of 8 (vision, but weak at tools)
qwen3:8b 40k 8.3 GB 100% 6 of 8 with thinking on

The 14B fell off the cliff from post 1: roughly 9 GB of weights before any KV cache, so a fifth of it ran on the CPU and generation crawled. The 9B was faster and lighter but picked the wrong tool half the time. Qwen3 8B gave up some speed and all vision for the best tool accuracy, and an agent needs tool accuracy more than anything.

The physics: predicting fit

Memory is arithmetic. Weights are parameters × bytes each:

Precision Bytes per parameter 7B 13B
FP16 2 14 GB 26 GB
INT8 1 7 GB 13 GB
4-bit about 0.5 about 3.5 GB about 6.5 GB

The KV cache grows per token:

bytes per token = 2 × layers × kv_heads × head_dim × bytes_per_value
def fits(params_b, bits, layers, kv_heads, head_dim, ctx, vram_gb=10, overhead_gb=1.0, kv_bytes=2):
    weights = params_b * 1e9 * bits / 8
    kv = 2 * layers * kv_heads * head_dim * kv_bytes * ctx
    total = (weights + kv) / 2**30 + overhead_gb
    return round(total, 1), total <= vram_gb

Qwen3 8B has 36 layers, 8 key/value heads and a head dimension of 128 (from its published config). The table is where a setting decides whether it runs at all:

KV cache type Per token At 32k At 40k Weights + KV at 40k
fp16 147,456 B 4.50 GiB 5.62 GiB 10.3 GiB, does not fit
q8_0 78,336 B 2.39 GiB 2.99 GiB 7.7 GiB, fits

Weights are 4.68 GiB, so the q8_0 cache gives 7.07 GiB at 32k and 7.67 GiB at 40k. I measured 7.6 and 8.3, an implied overhead of 0.5 to 0.6 GiB. The q8_0 cache is what makes 40k context possible at all. The server settings that came out of it:

OLLAMA_CONTEXT_LENGTH=40960
OLLAMA_NUM_PARALLEL=1
OLLAMA_FLASH_ATTENTION=1
OLLAMA_KV_CACHE_TYPE=q8_0
OLLAMA_KEEP_ALIVE=10m

What I produced

  • A LiteLLM front door where local-llm and local-llm-think are the same Qwen3 8B weights, differing only in a per-request thinking flag, so switching never reloads the model.
  • Embeddings and both rerankers moved to the CPU, so they stop competing with the language model.
  • A calculator for fit, above, that I now run before downloading any model.

Where it stands, and my verdict

How the datasheet's predictions from post 1 held up so far:

Question from post 1 Paper said Measured
Does a 7 to 8B model fit? Yes, with room Qwen3 8B at 40k uses 8.3 GB, fully on GPU held
Is 10 GB a cliff? Yes The 14B offloaded 21% and ran at 12.3 tok/s held
Weights + KV + overhead 7.7 GiB + 0.5 to 1.0 8.3 GB at 40k held
Can several jobs share the card? Only if memory sums to under 10 GB Embedder and STT moved off the GPU (post 3) held

Verdict: Qwen3 8B at 40k context, q8_0 KV cache. It is the only candidate that is both fully resident and good at choosing tools. It leaves about 1.4 GiB free, which is thin, so the embedder had to leave the GPU. Next: how fast it really is against the ceilings the datasheet drew.