Router eval — compare routing strategies on the corpus (T66)
Replays .local/runs/ offline and compares three routing strategies against each step's
recorded judge verdict:
| Strategy | Description |
|---|---|
| historical | Workers that were actually dispatched (ground truth) |
| deterministic | routeTask() on the step prompt — no LLM, no cost |
| smart | routeTaskSmart() — offline: same as deterministic for high-confidence tasks; live: LLM call for low-confidence ones (--live flag) |
Scoring uses recorded verdicts — no live provider calls. Cost and wall-clock for the
deterministic/smart picks are estimated from the historical worker runs (marked *).
# Offline replay — default (no live calls, free)
cyberine eval router
# Specify a runs directory (e.g. from a worktree or alternate project)
cyberine eval router --dir /path/to/project
# With CYBERINE_RUNS_DIR env override
CYBERINE_RUNS_DIR=/path/to/.local/runs cyberine eval router
# Via npm script (runs from repo root, uses .local/runs in cwd)
npm run eval:router
# Verbose: print progress per run to stderr
cyberine eval router --verbose
# Live mode: calls the LLM router for low-confidence tasks (requires CYBERINE_SMART_ROUTER providers)
cyberine eval router --liveThe report is saved to .local/eval/router-eval-latest.json and printed as a compact table
with an overall summary and a per-task-kind breakdown (qa / code-small / code-large).
Smart (LLM-as-router) for low-confidence tasks (T61)
When the deterministic router is unsure about a task (confidence < 0.7 — the "default fallback" bucket), you can opt in to a secondary LLM call that refines the worker ordering and per-worker...
Model strength table and fleet.json override (kyhae)
Cyberine ships a sourced answer to "which model is strong and which is weak": every model id the fleet can run has a coarse rank from 1 to 5 (5 = strongest) and a cost class from 1 to 3, keyed by...