Fleet bench — solve rate and dollars per task (kvxah)

cyberine eval bench scores every available worker, and the judged winner, on a versioned suite of coding tasks with programmatic pass/fail. Router eval above compares routing decisions against recorded verdicts; this measures whether a worker actually solves the task and what that cost. It is the quality-regression signal a fleet, model-pin or judge change otherwise ships without.

No LLM anywhere in the grading loop. Each task grades by one of two oracles:

A worker that edited the target file in its own cwd instead of printing it is picked up by a workspace fallback, so an implementer-shaped worker is not scored zero for a formatting habit.

An unavailable worker is skipped with a recorded reason, never scored zero — a zero for an absent worker corrupts every comparison it appears in. Likewise a worker that ran but returned a provider error (402, rate limit, open circuit) is counted as errored and excluded from its own solve-rate denominator: an outage is not a wrong answer. Both numbers stay visible.

Run recipe

# List the suite (ids, gates, titles) — no run at all
npm run bench -- --list

# DRY RUN (the default): exercises the whole path — task loading, prompt and Task shaping,
# the real ConsensusJudge, grading, report writing — against the suite's own reference
# candidates. ZERO provider calls, no credentials needed, safe in CI. Takes ~15s.
npm run bench

# A real sweep needs TWO flags. --real on its own prints the cost estimate and STOPS:
npm run bench -- --real
#   ESTIMATED COST: $0.3200  (16 tasks x 2 metered workers x $0.010/task)
#   Re-run with --real --confirm-spend to actually dispatch. Nothing was spent.

# Only this second form spends money:
npm run bench -- --real --confirm-spend

# Narrow a real run to one task while iterating (this is how the plumbing was verified live)
npm run bench -- --real --confirm-spend --task rle --concurrency 1

# Other flags
npm run bench -- --limit 4              # first n tasks only
npm run bench -- --keep-workspaces      # keep the throwaway grading dirs for inspection
npm run bench -- --json                 # report as JSON instead of the table

# Equivalent global CLI form
cyberine eval bench [--real --confirm-spend] [--task <id>] [--limit <n>]

Reports land in .local/bench/ (gitignored): a timestamped bench-<iso>-<mode>.json plus bench-latest.json. The suite itself is tracked — it is product knowledge, not run history.

Comparing two runs

Every report stamps suiteVersion, suiteHash, the repo HEAD and whether the tree was dirty. Two runs are comparable only when suiteVersion:suiteHash matches exactly, and the compare path refuses rather than printing a delta across two different task sets:

npm run bench -- --compare .local/bench/bench-2026-09-03T05-41-33-580Z-real.json
# second path defaults to .local/bench/bench-latest.json

suiteHash is a sha256 over everything that can move a score (id, mode, target, export, seed files, spec text, case table, assertion source, typecheck flag) — so an edited task without a version bump still changes the hash and cannot masquerade as the same suite. Rewording a task title deliberately does not move it.

Extend recipe — adding a task

Tasks live in src/eval/benchSuite.ts. Add an entry to BENCH_TASKS with:

• id (never reuse one), title, mode, targetPath, exportName;
• spec — the task text. It must pin every behaviour the assertions test, tie-breaks and edge cases included; an assertion testing something the prompt never stated measures prompt archaeology, not skill;
• cases — a table of { args, expect } compared by structural deep-equality (key order does not matter). Minimum three rows; the smoke fails a table shorter than that, because two rows is a coin flip a hardcoded return value can win. Use assertSource instead for stateful tasks a call table cannot express (see lru-cache);
• seedFiles — required for a diff task (the patch needs something to patch);
• typecheck: true to add a real tsc gate (costs about a second per candidate; the behavioural assertions are the stronger signal, so this is opt-in);
• referencePass / referenceFail — a known-good and a plausible-but-wrong candidate, written as a worker would reply (fenced block included).

Then bump BENCH_SUITE_VERSION (MINOR for an added task) and run npm run selftest:fleet-bench. The smoke grades both references for every task and fails unless the good one passes and the bad one fails at a real gate — so a task with no natural wrong answer, or an assertion table that accepts anything, is rejected at authoring time rather than quietly diluting the score later.

A task is eligible only if the answer is a pure function of its inputs: no clock, no network, no filesystem, no randomness, no concurrency. Anything else disagrees with itself on rerun.

No new environment variables. The bench reads no config of its own; the fleet it scores is whatever buildDefaultAdapters() builds from the credentials already documented above, and a real run is bounded by the same daily spend cap as any other dispatch.

Known Limitations and Deferred Work

• It measures self-contained function-writing, not agentic work. Every task is one small module or one small patch with the whole spec in the prompt. It says nothing about multi-file refactors, long tool-use loops, repo navigation, or recovering from a failed build — the things the implementer fleet actually does. A worker can top this board and still be poor at a real ticket.
• The cost estimate is a planning number, not a measurement. --real prices a flat $0.01 per task per metered worker; the real accrued cost always comes from each Result's own usage and is what the report records. Measured on this machine, one task across two loop workers actually cost $0.0060, so the estimate runs conservative by roughly 3x. Subscription-billed CLI workers spend quota rather than dollars and are listed separately instead of being priced at all.
• No statistical treatment. One sample per (worker, task), no repeats, no confidence interval. A one-task swing between two runs of the same fleet is noise, not a regression; the suite is sized to catch a step change, not a two-point drift.
• The judged score is only as broad as the judges that ran. orchestrate() builds the default judge set from config, so a machine with no LLM-judge credential is scoring ConsensusJudge alone, which on crisp-answer tasks with mutually-distinct outputs has no majority to find. Read judged next to fleet, never on its own.
• Prompt-format compliance is part of the score, deliberately. A worker whose reply carries no extractable artifact fails at the extract stage. The workspace fallback covers the common implementer shape, but a worker that neither prints nor writes the module is scored as not having solved the task.
• Grading runs the candidate's code in a plain subprocess under tsx, bounded only by a 60s timeout. It is not sandboxed. That is acceptable because the candidates come from this fleet on this machine, but do not point the bench at untrusted output.
• No public leaderboard is committed. Reports stay in gitignored .local/bench/.