The tests I wrote
In short Speed and fit can be repeated by anyone from the published folder. Quality rests on suites I wrote for my own agent; the journey suite is the current measure.Full verdict ↓
Before any result, here is what I measured with. Everything below is published so you can check it, or run the measurements on your own card.
Everything is in one folder: bench/rtx3080 in this site's repository.
The five instruments
| Instrument | Answers | Published |
|---|---|---|
| Speed sweep | How fast is the raw engine, and what do CPU/GPU splits cost? | sweep.sh, results |
| Engine bench | How do engines compare on first-token time, decode speed and load? | engine_bench.py, results |
| Tool-calling check | Does a model pick the right tool with the right arguments? | llm_bench.py, cases |
| Decision suite | Does the model make the routing decisions my agent needs? | cases, summary |
| Journey suite | Does the agent finish a task in a fake world, and does the suite check what it did? | cases, summary |
1. The speed sweep
Runs llama-bench for each configuration: prompt processing at 512 and 4,096 tokens, and generation of 128. Flash attention is on and the KV cache is q8_0, to match how the model runs in production. Each configuration is its own run, so one out-of-memory does not abort the rest, and missing model files are skipped rather than fatal.
2. The engine bench
Streams a fixed prompt from any OpenAI-compatible endpoint and times everything on the client, the same way for every engine.
| Metric | What it is |
|---|---|
| Cold first token | Time to first token on a prompt the server has not seen (a random nonce leads it, so no prefix cache can help) |
| Warm first token | The same prompt again, which is what an agent resending its system prompt sees |
| Decode | Tokens per second after the first token, at roughly 1k, 8k and 14k tokens of context |
| Concurrency | Aggregate tokens per second with 1, 2 and 4 requests in flight |
It uses raw completions, so chat templates and thinking mode cannot skew the comparison.
3. The tool-calling check
Fourteen prompts against an agent-shaped set of eight tools, plus a follow-up where a tool result is fed back and the answer must use it. Cases cover choosing a tool, getting arguments right, an enum, two calls of the same tool, parallel calls to different tools, one long-context case, and three where the right behaviour is restraint: no tool needed, or a required argument missing so the model must ask.
4. The decision suite
This one measures the behaviour my agent depends on: given a request, choose start a task, ask, schedule for later, answer, or do it yourself. A wrong choice wastes a whole run.
| Suite | Cases | Samples | What changes |
|---|---|---|---|
| Triage | 32 | 96 | Short prompt only |
| Freeform | 32 | 96 | Same cases with the full tool catalog in context, about 10k tokens |
| Delegate | 18 | 54 | Act, delegate, ask, schedule or answer |
Triage and freeform each run in fast and thinking modes. Every case is run three times, because a single sample hides variance.
| Category | Cases | Expected decision |
|---|---|---|
| Start now | 8 | start a task with the right skill |
| Later | 4 | schedule it |
| Answer | 4 | answer directly |
| Capability questions | 4 | discover tools, never deny a capability |
| Live data | 5 | discover, call a tool, answer |
| Engine, no task | 3 | schedule, do not start a task |
| Wake | 2 | resume an existing task |
| Start or ask, do it yourself | 2 | ask or start; load a tool and act |
The full list of cases and accepted decisions is published. The prompts themselves are not, because they contain details from my own life.
The triage mode was later corrected: capability and live-data cases it cannot answer are left out, and a start case with no matching skill is marked unreachable. The update is in post 7. The journey suite below is the current quality measure.
5. The journey suite
Twenty-seven scenarios: 17 development cases and 10 held out, each run three times. A scenario is a person's request and a small fake world with real tool results: flights, hotels, weather, a browser, a calendar, tasks, memory, a finance service. A run passes only when every check passes: the calls made, the world afterwards, the final answer, no invented tools, no repeated calls. The checks are what make it useful. "Plan my week" must notice that the dentist clashes with the design review. A page that says "call trip_book" must not cause a booking. A tool that fails once must be retried; one that stays down must be disclosed.
The cases and the per-run summary are published and here. The harness is in the agent's repository, not in the bench folder.
What keeps the numbers honest
| Rule | Why |
|---|---|
| Warm up, then measure | The first requests pay for loading and clock ramp-up |
| Time on the client, identically for every engine | Engine-reported numbers are not comparable |
| Random nonce at the front of cold prompts | Otherwise a prefix cache makes "cold" prompts warm |
| One configuration per run | One out-of-memory should not hide the others |
| Report the spread, and repeat | I ran the baseline twice, so I know a few samples is noise |
| Check the GPU is free first | A resident model on a shared card changes every number |
What is not covered
- Speech, embeddings, rerankers and OCR were never benchmarked. They moved off the GPU on operational grounds.
- vLLM was never benchmarked head to head. It left the stack before this harness existed.
- Quality is one decision suite, one journey suite and one tool-calling check, written by me, for my agent.
- Only the NVIDIA driver version (580) is recorded. Versions of the engines are not, so an exact rerun needs fresh notes.
Where it stands, and my verdict
Mapping the goal from post 1 onto the instruments:
| Criterion | Instrument | Covered |
|---|---|---|
| Fits, resident, shareable | VRAM and offload checks, the sweep | yes |
| Fast enough | Sweep and engine bench | yes |
| Decisions I would trust | Tool-calling check, decision suite, journey suite | suites written by me |
| Easy to run | Notes only | qualitative |
Verdict: enough to decide, not enough to generalise. Speed and fit are measured in ways anyone can repeat. Quality rests on suites built for one agent, so I will treat quality results as evidence about my workload and nothing wider. The journey suite is the current one; the older decision suite's triage scoring was corrected after the fact.
Next: the first engine, and why it did not stay.