cyberine
UsageRun recipes

Router eval — compare routing strategies on the corpus (T66)

Replays .local/runs/ offline and compares three routing strategies against each step's recorded judge verdict:

StrategyDescription
historicalWorkers that were actually dispatched (ground truth)
deterministicrouteTask() on the step prompt — no LLM, no cost
smartrouteTaskSmart() — offline: same as deterministic for high-confidence tasks; live: LLM call for low-confidence ones (--live flag)

Scoring uses recorded verdicts — no live provider calls. Cost and wall-clock for the deterministic/smart picks are estimated from the historical worker runs (marked *).

# Offline replay — default (no live calls, free)
cyberine eval router

# Specify a runs directory (e.g. from a worktree or alternate project)
cyberine eval router --dir /path/to/project

# With CYBERINE_RUNS_DIR env override
CYBERINE_RUNS_DIR=/path/to/.local/runs cyberine eval router

# Via npm script (runs from repo root, uses .local/runs in cwd)
npm run eval:router

# Verbose: print progress per run to stderr
cyberine eval router --verbose

# Live mode: calls the LLM router for low-confidence tasks (requires CYBERINE_SMART_ROUTER providers)
cyberine eval router --live

The report is saved to .local/eval/router-eval-latest.json and printed as a compact table with an overall summary and a per-task-kind breakdown (qa / code-small / code-large).