Quota cooldown — automatic circuit-breaker after 402/429

When a worker fails with a quota signal (HTTP 402 payment required / insufficient credits, or HTTP 429 rate-limit / too-many-requests), Cyberine records a temporary cooldown for that worker ID in ~/.cyberine/worker-cooldown.json. On the next dispatch, that worker is skipped automatically — the router never re-selects a cooling-down worker. This prevents hammering an exhausted provider across repeated runs.

The gate lives in cliWorkerAvailable() (src/orchestrator/buildAdapters.ts), so it applies identically to the auto-detected default fleet, the router's candidate pool (availableBuildWorkerIds()), AND an explicit CYBERINE_BUILD_FLEET list — naming a cooling-down CLI worker there skips it with the same note rather than re-spawning it on a still-exhausted account (kzhyc).

Cooldown kinds and their default durations:

• rate-limit (transient, will recover): 30 minutes (CYBERINE_QUOTA_COOLDOWN_RATELIMIT_MS).
• out-of-credits (hard, won't self-heal): 24 hours (CYBERINE_QUOTA_COOLDOWN_CREDITS_MS).
• auth-failure (needs re-auth / key rotation): 24 hours (CYBERINE_QUOTA_COOLDOWN_AUTH_MS).
• cli-unavailable (CLI worker out of quota / not logged in): 24 hours (CYBERINE_QUOTA_COOLDOWN_CLI_MS), but reprobed before every dispatch and by cyberine doctor — a fresh CLI login check that now passes clears the entry immediately instead of waiting out the 24h TTL (knwps).
• connection-failure (refused/unreachable endpoint — a local gateway or Ollama that is not running, DNS miss, timeout): 5 minutes (CYBERINE_QUOTA_COOLDOWN_CONNECTION_MS). Short by design so the worker rejoins quickly once the endpoint returns, but a configured-but-down endpoint stops erroring on every dispatch after its first failure.

The cooldown is recorded at the adapter level — it fires whenever run() returns status === 'error' and the error message matches a known quota pattern. No pre-checking is done; the signal surfaces naturally from the real API response.

A CLI usage limit that names its reset ends the cooldown at that reset (ktwg5). Codex refuses with "You've hit your usage limit ... try again at 9:16 AM" (or "try again at Sep 24th, 2026 9:16 AM" when the reset is another day). Any CLI worker whose refusal carries such a "try again at " or "try again in " phrase gets a rate-limit cooldown that expires exactly then, instead of the 24-hour cli-unavailable TTL. When a codex refusal names no time, the reset is taken from the newest rate_limits reading codex itself writes to $CODEX_HOME/sessions/**/rollout-*.jsonl (any window at 100% used). Verify with npm run selftest:cli-usage-limit, then cyberine cooldown (or cyberine doctor) to see the entry.

Known Limitations and Deferred Work: muse publishes no reset time (only tier_limit / rate_limited 429 events, see docs/providers/muse-cli.md), so a muse limit keeps the generic TTL. The codex rollout reading is only as fresh as the last codex session on this machine. Pre-dispatch ordering on these readings is the job of the failover chain (ksxky), not of this cooldown.

Exhausted Claude seat (k7em9). A subscription seat whose 5-hour or weekly window is gone does NOT fail: the claude CLI retries internally (up to max_retries, sleeping until the window resets) and never exits, emits no result event, nothing on stderr, and no rate_limit_event — that event rides on a successful reply. The only signal is a system / api_retry line with error_status: 429, error: "rate_limit" and retry_delay_ms (the time until the reset, recomputed on every attempt). The Claude adapter watches for that line on both the streaming and the detached path, kills the child on first sighting, returns status: 'error' with a message that names the rate limit and the minutes to reset (raw.quotaExhausted carries the parsed event), and records a rate-limit cooldown for that worker id expiring at now + retry_delay_ms instead of the flat 30-minute TTL. A seat-pinned lane (claude-<seat>) therefore cools down alone; its sibling seats keep dispatching. Measured live 2026-09-13 on a seat at 100% of its five-hour window; the two captures replay under npm run selftest:seat-exhaustion.

Autopilot drivers and the quota preflight also abandon an exhausted seat (k0vyh). probeQuota() (src/autopilot/quotaPreflight.ts), the print session driver (src/autopilot/sessionDriver.ts) and the streaming session driver (src/autopilot/streamSessionDriver.ts) all spawn through the same claude -p invocation the adapter above uses, so all three now watch their child's stdout for the api_retry line as it streams, kill the child on first sighting, and turn it into a reading/turn the caller already knows how to handle instead of a mystery timeout: probeQuota() returns a ClaudeRateLimitInfo with status: 'rejected' and resetsAt = now + retry_delay_ms (the same "rejected" field evaluateQuotaGate(), livenessQuota.ts and seatHeadroom.ts already key off), and both session drivers resolve the turn with rateLimit: { status: 'rejected', resetsAt } and isError: true instead of throwing a stall/cap timeout — autopilotLoop.ts's classifyTurnFailure() reads that status: 'rejected' exactly as it reads a real rate_limit_event, so the autopilot loop's existing --on-quota wait|stop|api handling takes over rather than retrying the same exhausted seat. Each of the two session drivers also records a rate-limit cooldown (worker id: the driver's sessionName, or the claude binary name when no session name was given) so the worker-cooldown store reflects the same seat other Cyberine surfaces read. Replayed by npm run selftest:seat-exhaustion (sections 6-9).

Known limitations: only error_status: 429 + error: "rate_limit" abandons the spawn — a 529 overload or any other api_retry is left to the CLI's own retry. CLAUDE_CODE_MAX_RETRIES does not help: a value set in ~/.claude/settings.json overrides the spawn env, and 0 still waits for the reset. Preemptive rotation off a seat on allowed_warning, before the window fully closes, is still a follow-up (k9cw8).

Loop workers honour retry-after too (k2vdg). LoopAgentAdapter's OpenAI-wire error path reads the failed response's retry-after header — delta-seconds or an HTTP-date, per RFC 9110 — via retryAfterToUntilMs() (src/store/workerCooldown.ts) and cools down at that exact reset instant instead of the flat CYBERINE_QUOTA_COOLDOWN_RATELIMIT_MS default: the per-model fallback cooldown (recordModelCooldown, an opts.untilMs override) when a fallback model rate-limits mid-chain, and the whole-worker cooldown (recordRateLimitCooldownUntil) when a rate-limit is the terminal error. A header that is absent, empty, or unparsable falls back to the flat default unchanged. retryAfterToUntilMs()'s parsing (delta-seconds, HTTP-date, absent, garbage) and the recordQuotaCooldown untilMs override are covered by npm run selftest:cooldown-until.

Known limitations (k2vdg): only the HTTP retry-after response header is read — an error body carrying a reset hint in its own vocabulary rather than a header is not parsed, since no funded provider on this fleet's OpenAI-wire endpoints was observed doing that. CLI workers other than the Claude seat above (Codex, Gemini, Muse) still cool down for the flat 30-minute rateLimitTtlMs() on a rate-limit signal: their failure surface is stdout/stderr text, not HTTP response headers, so there is no structured reset to read. src/judges/fallbackCompletion.ts and src/engine/WorkflowEngine.ts also call classifyQuotaError() on an error string, but neither records a cooldown from that call (judge-backend switching and credit-ledger zeroing, respectively) — there is nothing to wire there.

Exception (kc98t): loop-deepseek-build and loop-deepseek-direct are ALSO gated proactively, from a free /user/balance probe rather than a failed dispatch — a reactive out-of-credits cooldown expires after 24h and the account does not self-heal the way a rate limit does, so a still-unfunded account would otherwise become a candidate again as soon as the cooldown ticks out. cyberine doctor --liveness (and any other caller of collectLiveness(), e.g. the fleet TUI overview or GET /api/adapters/liveness) re-checks the balance and re-arms the same out-of-credits cooldown on every call when the account is still unfunded (src/core/deepseekCreditsGate.ts). Plain cyberine doctor stays offline-safe: it only reads back whatever cooldown that probe already wrote, under its own "DeepSeek API credits:" section.

# List active cooldowns (worker id, kind, time remaining, reason)
cyberine cooldown

# Clear the cooldown for one worker (e.g. after topping up credits)
cyberine cooldown clear loop-deepseek-build

# Clear all cooldowns
cyberine cooldown clear

# Disable the feature entirely (both recording and filtering)
CYBERINE_QUOTA_COOLDOWN=0 cyberine run "..."

Active cooldowns also appear in cyberine auth status under a "Quota cooldowns:" line.