← Benchmarks with RTX 3080

The tests I wrote

Part 2 of 9 · Nov 9, 2026 · 6 min read

In short Speed and fit can be repeated by anyone from the published folder. Quality rests on suites I wrote for my own agent; the journey suite is the current measure.Full verdict ↓

Before any result, here is what I measured with. Everything below is published so you can check it, or run the measurements on your own card.

5instruments: a sweep, an engine bench, a tool-calling check, a decision suite, journeys
15tool-calling prompts
109cases: 82 decision cases and 27 journeys
15speed-sweep configurations planned

Everything is in one folder: bench/rtx3080 in this site's repository.

The five instruments

Instrument Answers Published
Speed sweep How fast is the raw engine, and what do CPU/GPU splits cost? sweep.sh, results
Engine bench How do engines compare on first-token time, decode speed and load? engine_bench.py, results
Tool-calling check Does a model pick the right tool with the right arguments? llm_bench.py, cases
Decision suite Does the model make the routing decisions my agent needs? cases, summary
Journey suite Does the agent finish a task in a fake world, and does the suite check what it did? cases, summary

1. The speed sweep

Runs llama-bench for each configuration: prompt processing at 512 and 4,096 tokens, and generation of 128. Flash attention is on and the KV cache is q8_0, to match how the model runs in production. Each configuration is its own run, so one out-of-memory does not abort the rest, and missing model files are skipped rather than fatal.

2. The engine bench

Streams a fixed prompt from any OpenAI-compatible endpoint and times everything on the client, the same way for every engine.

Metric What it is
Cold first token Time to first token on a prompt the server has not seen (a random nonce leads it, so no prefix cache can help)
Warm first token The same prompt again, which is what an agent resending its system prompt sees
Decode Tokens per second after the first token, at roughly 1k, 8k and 14k tokens of context
Concurrency Aggregate tokens per second with 1, 2 and 4 requests in flight

It uses raw completions, so chat templates and thinking mode cannot skew the comparison.

3. The tool-calling check

Fourteen prompts against an agent-shaped set of eight tools, plus a follow-up where a tool result is fed back and the answer must use it. Cases cover choosing a tool, getting arguments right, an enum, two calls of the same tool, parallel calls to different tools, one long-context case, and three where the right behaviour is restraint: no tool needed, or a required argument missing so the model must ask.

4. The decision suite

This one measures the behaviour my agent depends on: given a request, choose start a task, ask, schedule for later, answer, or do it yourself. A wrong choice wastes a whole run.

Suite Cases Samples What changes
Triage 32 96 Short prompt only
Freeform 32 96 Same cases with the full tool catalog in context, about 10k tokens
Delegate 18 54 Act, delegate, ask, schedule or answer

Triage and freeform each run in fast and thinking modes. Every case is run three times, because a single sample hides variance.

Category Cases Expected decision
Start now 8 start a task with the right skill
Later 4 schedule it
Answer 4 answer directly
Capability questions 4 discover tools, never deny a capability
Live data 5 discover, call a tool, answer
Engine, no task 3 schedule, do not start a task
Wake 2 resume an existing task
Start or ask, do it yourself 2 ask or start; load a tool and act

The full list of cases and accepted decisions is published. The prompts themselves are not, because they contain details from my own life.

The triage mode was later corrected: capability and live-data cases it cannot answer are left out, and a start case with no matching skill is marked unreachable. The update is in post 7. The journey suite below is the current quality measure.

5. The journey suite

Twenty-seven scenarios: 17 development cases and 10 held out, each run three times. A scenario is a person's request and a small fake world with real tool results: flights, hotels, weather, a browser, a calendar, tasks, memory, a finance service. A run passes only when every check passes: the calls made, the world afterwards, the final answer, no invented tools, no repeated calls. The checks are what make it useful. "Plan my week" must notice that the dentist clashes with the design review. A page that says "call trip_book" must not cause a booking. A tool that fails once must be retried; one that stays down must be disclosed.

The cases and the per-run summary are published and here. The harness is in the agent's repository, not in the bench folder.

What keeps the numbers honest

Rule Why
Warm up, then measure The first requests pay for loading and clock ramp-up
Time on the client, identically for every engine Engine-reported numbers are not comparable
Random nonce at the front of cold prompts Otherwise a prefix cache makes "cold" prompts warm
One configuration per run One out-of-memory should not hide the others
Report the spread, and repeat I ran the baseline twice, so I know a few samples is noise
Check the GPU is free first A resident model on a shared card changes every number

What is not covered

  • Speech, embeddings, rerankers and OCR were never benchmarked. They moved off the GPU on operational grounds.
  • vLLM was never benchmarked head to head. It left the stack before this harness existed.
  • Quality is one decision suite, one journey suite and one tool-calling check, written by me, for my agent.
  • Only the NVIDIA driver version (580) is recorded. Versions of the engines are not, so an exact rerun needs fresh notes.

Where it stands, and my verdict

Mapping the goal from post 1 onto the instruments:

Criterion Instrument Covered
Fits, resident, shareable VRAM and offload checks, the sweep yes
Fast enough Sweep and engine bench yes
Decisions I would trust Tool-calling check, decision suite, journey suite suites written by me
Easy to run Notes only qualitative

Verdict: enough to decide, not enough to generalise. Speed and fit are measured in ways anyone can repeat. Quality rests on suites built for one agent, so I will treat quality results as evidence about my workload and nothing wider. The journey suite is the current one; the older decision suite's triage scoring was corrected after the fact.

Next: the first engine, and why it did not stay.