Worker-profile eval coverage (T678)

Generated by npx tsx scripts/t678-profile-coverage.ts --write. Re-run it after any fleet dispatch to refresh; do not hand-edit.

8 direct · 2 provider-path · 1 failed-only · 4 retired-unreachable · 26 uncovered out of 41 declared profiles, from 278 run(s) across 3 runs directories and 999 recorded worker run(s).

Of the 26 uncovered, 0 belong to a worker this machine can currently dispatch — those are the only actionable coverage gaps. The rest need a credential, a funded account, or a host this machine is not.

Evidence	Meaning
direct	A recorded workerRuns[].agentId IS this profile id — the fleet actually spawned it.
provider-path	The same provider:model ran under a different agentId (a demo, a dogfood workflow, or a lane-scoped c-NN-label@adapter worker). Transport, auth and model are proven; the profile's own registration is not.
failed-only	A run DID try this profile and every recorded attempt ERRORED. Louder than uncovered: the worker is broken or its account is unusable, and the reason is in the last column.
retired	The worker is in RETIRED_WORKERS — it cannot be dispatched at all, so it is NOT a coverage gap. Reason in the last column. Reported separately because "uncovered" invites someone to go cover it and "failed-only" invites someone to go fix it; neither applies to a worker the operator switched off.
uncovered	No scanned run touched this provider+model, and the worker is not retired. Its tier/cost/speed/rank are hand-written estimates only.

Profile	Kind	Tier	Rank	Evidence	OK runs	Failed runs	Seen as / why
claude	cli-agent	strong	1	direct	57	2	claude:claude-sonnet-4-6, claude:claude-sonnet-5
codex	cli-agent	strong	2	direct	24	9	codex:default, codex:deepseek-v4-flash
opencode-deepseek	cli-agent	standard	3	direct	159	5	opencode:deepseek/deepseek-v4-pro, opencode:moonshotai/kimi-k2.6
opencode-kimi	cli-agent	standard	4	direct	4	26	opencode:moonshotai/kimi-k2.6, opencode:moonshotai/kimi-k2.7-code · RETIRED 2026-08-27
loop-deepseek-build	key-loop	standard	7	direct	238	9	deepseek:deepseek-v4-pro, openrouter:deepseek/deepseek-v4-pro +1
loop-mimo-build	key-loop	standard	7	direct	51	2	xiaomi:mimo-v2.5-pro
loop-ollama-cloud-build	key-loop	standard	7	direct	14	4	ollamaCloud:gpt-oss:120b-cloud
loop-kimi-build	key-loop	standard	8	direct	1	2	moonshot:kimi-k2.6 · RETIRED 2026-08-27
loop-claude-build	key-loop	strong	9	provider-path	7	5	anthropic/claude
loop-gpt-build	key-loop	strong	9	provider-path	7	5	openai/gpt
loop-openrouter-build	key-loop	standard	7	failed-only	—	1	402 This request requires more credits, or fewer max_tokens. You requested up to 45000 tokens, but can only afford 1292. To increase, visit https://openrouter.ai/workspaces/default
gemini	cli-agent	standard	6	retired	—	1	RETIRED 2026-08-27 — Google revoked the free-tier client (IneligibleTierError) and no paid tier is funded (kp7wq)
loop-minimax-build	key-loop	standard	7	retired	—	3	RETIRED 2026-08-27 — MiniMax account out of balance (402 insufficient balance 1008) -- operator retired it (kf2mx)
mmx-text-build	cli-agent	standard	7	retired	—	—	RETIRED 2026-08-27 — MiniMax mmx CLI -- same unfunded MiniMax account (kf2mx)
loop-gemini-build	key-loop	strong	8	retired	—	—	RETIRED 2026-08-27 — Gemini API key path -- same unfunded Google account as the gemini CLI worker (kp7wq)
cursor-agent	cli-agent	strong	2	uncovered	—	—	not dispatchable here (no credential / not installed)
loop-ollama-build	key-loop	flash	5	uncovered	—	—	not dispatchable here (no credential / not installed)
opencode2-deepseek	cli-agent	standard	5	uncovered	—	—	not dispatchable here (no credential / not installed)
amp	cli-agent	strong	5	uncovered	—	—	not dispatchable here (no credential / not installed)
loop-copilot-build	key-loop	standard	6	uncovered	—	—	not dispatchable here (no credential / not installed)
aider	cli-agent	standard	6	uncovered	—	—	not dispatchable here (no credential / not installed)
qwen-code	cli-agent	standard	6	uncovered	—	—	not dispatchable here (no credential / not installed)
crush	cli-agent	standard	6	uncovered	—	—	not dispatchable here (no credential / not installed)
goose	cli-agent	standard	6	uncovered	—	—	not dispatchable here (no credential / not installed)
openhands	cli-agent	standard	6	uncovered	—	—	not dispatchable here (no credential / not installed)
loop-groq-build	key-loop	standard	7	uncovered	—	—	not dispatchable here (no credential / not installed)
loop-together-build	key-loop	standard	7	uncovered	—	—	not dispatchable here (no credential / not installed)
loop-fireworks-build	key-loop	standard	7	uncovered	—	—	not dispatchable here (no credential / not installed)
loop-qwen-build	key-loop	standard	7	uncovered	—	—	not dispatchable here (no credential / not installed)
loop-zhipu-build	key-loop	standard	7	uncovered	—	—	not dispatchable here (no credential / not installed)
loop-cerebras-build	key-loop	standard	7	uncovered	—	—	not dispatchable here (no credential / not installed)
loop-mistral-build	key-loop	standard	8	uncovered	—	—	not dispatchable here (no credential / not installed)
loop-perplexity-build	key-loop	standard	8	uncovered	—	—	not dispatchable here (no credential / not installed)
loop-nvidia-build	key-loop	standard	8	uncovered	—	—	not dispatchable here (no credential / not installed)
loop-cohere-build	key-loop	standard	8	uncovered	—	—	not dispatchable here (no credential / not installed)
loop-openai-build	key-loop	strong	9	uncovered	—	—	not dispatchable here (no credential / not installed)
loop-xai-build	key-loop	strong	9	uncovered	—	—	not dispatchable here (no credential / not installed)
loop-azure-build	key-loop	strong	9	uncovered	—	—	not dispatchable here (no credential / not installed)
loop-claude-native-build	key-loop	strong	9	uncovered	—	—	not dispatchable here (no credential / not installed)
loop-bedrock-build	key-loop	strong	9	uncovered	—	—	not dispatchable here (no credential / not installed)
loop-vertex-build	key-loop	strong	9	uncovered	—	—	not dispatchable here (no credential / not installed)

Currently dispatchable fleet — the only rows that can move

availableBuildWorkerIds() (the same call cyberine route prints as its available: line) reports 4 of 41 profiles as dispatchable from this machine right now. Coverage can only ever move for these; every other row needs a credential, a funded account, or an installed CLI first, so re-running eval passes cannot close them.

Profile	Evidence	OK runs	Verdict
claude	direct	57	covered by real recorded runs
codex	direct	24	covered by real recorded runs
loop-deepseek-build	direct	238	covered by real recorded runs
loop-mimo-build	direct	51	covered by real recorded runs

Every profile of a currently-available worker is covered by real recorded runs. There is no eval pass left to run that would raise this number: the remaining uncovered rows are all workers this machine cannot dispatch, and the retired ones can never be dispatched again. The honest state of the untested majority is "unreachable", not "untested".

Limitations

• Matching is exact-only: a profile is credited on an exact profile-id match, or an exact provider prefix + model match. It never infers from a model family, so uncovered can under-count history whose run directories no longer exist on this machine.
• A profile whose model comes from an env override (modelEnvKey) is matched against the model resolved IN THIS PROCESS. Running this script with a different env can shift a row.
• Re-pointing a profile's model id resets its coverage to uncovered, because matching is against the CURRENT model string. That is correct (no run has exercised the new id yet), not a regression — expect it after any deliberate model bump. Live example: T691 repointed loop-openrouter-build from the undated deepseek/deepseek-v4-pro (a frozen 2026-04-23 alias on OpenRouter) to the dated current GA deepseek/deepseek-v4-pro-0813, so its historical provider-path evidence no longer matches. The old ids stay visible in the "Seen as / why" column, which records what past runs actually ran.
• Only state: 'done' counts as evidence and only state: 'error' counts as a failure. A 'pending' or 'cancelled' worker run never executed (its step was cancelled upstream, or the engine was killed mid-run) and is ignored entirely rather than scored either way.
• Coverage says a worker RAN, not that it ran WELL. Ranking quality needs npm run eval:router (routerEval replay) or a recorded head-to-head, which is the second half of T678's brief.
• Only the WORKFLOW ENGINE records runs. A npm run cli / cyberine run dispatch goes through orchestrate(), which never touches RunStore, so it leaves NO run.json and cannot raise coverage here however many adapters it fans out to. Coverage moves come from npm run wf -- src/engine/implementTaskWorkflow.ts with CYBERINE_BUILD_FLEET set.
• A profile can be structurally unreachable regardless of credentials: the implement-task workflow's per-step workers whitelist decides who competes, so a worker enabled via CYBERINE_BUILD_FLEET but absent from that list is built and then skipped (the engine says so: "built ... but step X whitelist excludes it — not competing").