History replay bench (k0t77)

cyberine eval replay discovers past repository commits that added a scripts/*smoke*.ts grader and changed src/, starting on 2026-08-01. Each task starts from a git archive export of the parent commit, without .git or the grader. The grader is copied from the source commit only after the candidate diff has been applied. Dependencies come from a symlink to the current repository's node_modules; install them before grading.

Run recipe

npm run bench:replay
cyberine eval replay --limit 12
cyberine eval replay --controls --limit 12
cyberine eval replay --controls --limit 12 --timeout-ms 60000 --json
cyberine eval replay --ref HEAD --limit 12 --json
cyberine eval replay --real --worker WORKER_ID --budget-usd 1 --limit 1

The default only lists candidate task ids, source commits, grader paths and brief sources. --limit selects candidates before controls, newest first. --ref pins discovery to a commit; save its resolved head and suiteHash from JSON to reproduce the selection. Dry discovery does not claim that candidates are eligible.

--controls runs two fresh exports per candidate: the positive control applies the entire original commit diff except its grader; the negative control applies an empty diff. The grader runs with npx --no-install tsx scripts/<grader>.ts. Exit 0 is PASS, exit 78 is SKIP, every other exit is FAIL. Only positive PASS plus negative FAIL from an executed, non-timeout smoke admits a task. Excluded tasks retain both results and their reasons. An empty eligible suite exits nonzero. Controls reports include discoveryHash for the candidates and suiteHash for the eligible tasks, each hashing the ordered (commit, smoke path) list.

Briefs use only task-file title, do, Intent and Pointers fields from .local/tasks/tasks/archive/<id>.md or .local/tasks/tasks/<id>.md; in a worktree discovery also checks the main checkout's task files. Completion sections and evidence are omitted, commit hashes are stripped, and smoke filenames become hidden-check. Missing task files fall back to the commit subject and body. Discovery never builds a brief from the diff. Since suite version 1.1.0 every brief ends with an ## Interface the hidden check uses section: each src/ module the grader imports (static, type-only, namespace and dynamic await import(...) forms), the names it imports from it, and (new file) when the parent commit lacks that path. Without it a candidate must guess the module path and export names the grader loads, and a correct change in another file fails with ERR_MODULE_NOT_FOUND; the kp1mv pilot measured 0 of 8 baseline solves on version 1.0.0 for that reason. Since suite version 1.2.0 the section also carries a ### Declarations the hidden check relies on block: for each imported name that the reference commit exports and the parent does not (or, for functions, classes and consts, exports with a different signature), the declaration extracted with the TypeScript compiler API, meaning parameter lists, return types, class public members, and the interface or type shapes those signatures reference in the same module (two levels deep, comments stripped). A function with no return annotation lists its returned expressions as a // returns: line. Known limitations: the block shows declarations, not behaviour, so a grader that checks a computed value still needs the Intent text; shapes imported from other modules are not expanded; an inferred return type is shown by its return expressions, not resolved by the type checker; the dynamic-import destructure no longer swallows an enclosing const { (fixed in the same change). Guard: npm run selftest:replay-interface.

Paired runs

--real alone prints the maximum attempt count and allocated worker budget, then stops. Only --real --confirm-spend --worker WORKER_ID --budget-usd N dispatches through orchestrate(). --harness-a PATH and --harness-b PATH load prompt preamble files. Each eligible task runs with the same pinned worker in three independent arms: A, B and plain-retry. Plain-retry uses A's preamble and fresh attempts without grader feedback; every arm has the same USD allocation and attempt cap (--attempts, default 2). An arm stops early on PASS. The per-attempt allocation is the arm budget divided by its attempt cap. Real reports are JSON with task/worker pairs, harness hashes, attempt outcomes, actual reported costs and configured budgets. Unknown costs remain null.

Known Limitations and Deferred Work

• The suite requires this repository's git history. Historical dependency changes are not installed: the shared current node_modules can make an otherwise valid control fail, and that task is excluded. Full discovery can be larger than a selected control run; excluded tasks never enter its score denominator.
• Controls use a fresh HOME and Cyberine config directory, an allowlisted environment, offline npm resolution, and a Node preload that rejects external sockets, DNS lookups and common real agent/network executables. Loopback fixture servers are allowed. A denied dependency excludes the task as SKIP. This is a guard for trusted historical smokes, not an operating system sandbox; arbitrary native subprocesses or hostile candidate code require separate isolation. Shared dependencies must be treated as read-only.
• Task files are local and may be absent or changed. The required suite hash pins commit/grader identities; it does not pin local brief text, dependency versions or machine capabilities. Preserve the JSON briefs and environment information when comparing reports. A commit-message fallback may describe the completed implementation more directly than an archived intent.
• The interface section names modules and imported names only, never signatures, argument shapes or behaviour, so a candidate can still fail on a contract the grader checks; a report from suite version 1.0.0 is not comparable with one from 1.1.0. Discovery also reads every grader once more, which added about 17% to a full discovery (95 s to 111 s for 302 candidates on this machine).
• Current harness support compares prompt preambles in the same running Cyberine build. Comparing two runtime implementations, automated report comparison, confidence intervals and a published leaderboard are deferred. Real dispatch was not exercised as part of this feature's offline verification. The plain-retry arm is an independent baseline with A's configuration, not an additional harness.
• Worker budget allocations follow the existing orchestrator's budget contract; CLI quota and additional judge charges are not a guaranteed USD cap. Historical smokes that need network, unavailable tools, host sandbox permissions or repository metadata are excluded when controls do not hold. --timeout-ms (default 30000) applies to each smoke run; --dispatch-timeout-ms (default 1200000, 20 minutes) applies to each real worker attempt. All temporary exports are removed after use; redirect JSON output to retain a report.