← all posts

Five of six models were correct.
That told me nothing.

I gave six models two tasks whose answers I had already computed. Five returned perfect results. The number that decided my choice appears on no leaderboard.

Benchmarks answered the wrong question

My agent runs on a task board. It reads data, computes, writes a result file. Sometimes a task dies and restarts. I wanted to know which model would crash less and cost less.

BFCL measures a single tool call: did the model pick the right function and fill the arguments. My failures don't happen inside one call. They happen across thirty of them, where each step depends on the last. Different skill.

The write-ups were worse than useless. Two numbers I found fell apart on inspection: a widely quoted "97.2% tool-call accuracy" belongs to a different model than the one it was attached to, and a protocol I was told had real volume turned out to be four times smaller once test transactions came out.

Method — know the answer first

I took two tasks from my actual work and computed the correct answers myself, in separate code. Not a quality rubric. Exact numbers to compare against.

Light task

Pull the top 2,000 plugins from the WordPress API, filter to abandoned ones — untouched for over two years, 10k+ active installs — and report five figures.

abandoned:            139
total installs:   4770000
of those >=50k:        28
alive (daily dls):    139
tagged adopt-me:        0

Heavy task

Read 17 MB of local JSONL — 50,605 public procurement tenders — and apply a stated algorithm to one company. No network. Just reading, filtering, and holding state while the context window compacts underneath you.

wins:                  14
candidates:          1259
zero bidders:        1129
sum:            148183285

The output format was deliberately rigid: four lines, numbers only. Nothing to hide behind.

Results

* one rounding difference, explained below. Cost estimated on my real volume: roughly 1.3M input and 158k output tokens per working day.

Finding 1 — retries decided it, not answers

Look at the correctness column. Five of six models nailed both tasks. Judging by output files alone, the choice would have been a coin flip.

Everything useful sat in the runs column. GLM-5.2 burned seven runs where others used two. Its answers were flawless. The agent kept dying on a protocol error and restarting. On paper GLM costs half what Kimi does. With the restarts counted, it costs three times more and takes three times longer.

No leaderboard reports this, because it only shows up when a model runs inside an agent loop holding tools — not when it answers a question.

Finding 2 — the model that answered a different question

GPT-5.6 Luna is the cheapest model I tested, fifteen times cheaper than what I was running. It failed twice out of three attempts, and both failures are worth describing.

On the first run it skipped the five-line output entirely and produced "a shortlist of 9 relevant plugins across security, commerce and ecosystem categories." Useful work. Not the requested work.

The second failure is the one that matters:

wins: 14           ✓
candidates: 1259   ✓
zero bidders: 1129 ✓
sum: 32917788      ✗   (correct: 148183285)

Three of four numbers matched exactly. I traced the fourth. 32,917,788 is the sum of only the tenders with zero bidders — arguably the more interesting figure for the underlying business question. It wasn't the one I asked for.

Without a precomputed answer I would have shipped that number.

Finding 3 — the ambiguity was mine

DeepSeek returned 148,183,119 against my 148,183,285. A gap of 166 out of 148 million, one ten-thousandth of a percent. I went looking for the cause:

exact float sum:      148,183,285.07
rounded at the end:   148,183,285      ← my reference
int() per record:     148,183,119      ← DeepSeek

It truncated the kopecks on each record before summing. 343 of the 1,259 tenders carried a fraction. My spec said "report the sum as an integer" and never said when to round.

That is not a model error. It is an ambiguous instruction, and the model picked a defensible reading. The distinction from Luna is the whole point: Luna computed a different quantity; DeepSeek computed the same one with different rounding.

What I run now

DeepSeek-V4-Pro as the main model: correct on both tasks, one run each, second-fastest on the heavy one, eight times cheaper than what it replaced. A small model for subagents, the most reliable expensive one as a fallback when the primary rate-limits.

Where this test is thin

Two tasks per model is a small sample. I trust the retry count more than the correctness column, because the spread there is measured in multiples rather than decimals.

Both tasks are deterministic computations with one right answer. That says nothing about open-ended research, where no reference exists. Worth noting: the crashes I had seen the day before, on exactly that kind of open task, never reproduced here.

The habit I'm keeping: compute the answer before you ask for it. Not a rubric, not a grading scale — a number, derived separately. Otherwise you are measuring your own willingness to believe.