Provider circuit breaker and bounded retry — one dead provider degrades the fleet, not the run
Distinct from the quota cooldown above, which reacts only to a 402/429 quota signal: the breaker
counts consecutive terminal failures of any shape at the Orchestrator.dispatch() seam, so it
also catches the failure modes no pattern recognises. State machine per provider: closed ->
open (fast-fail for a cooldown window) -> half-open (exactly one probe admitted) -> closed on
a successful probe, or straight back to open on a failed one.
A failure is classified before anything is retried. Only transient is retried (HTTP 5xx/529, bad
gateway, service unavailable, overloaded, ERR_STREAM_PREMATURE_CLOSE, EPIPE, timeouts, plus
rate-limit and connection failures). A hard failure is never retried, because paying twice for it
buys nothing: model-not-found, 400/404/422, invalid_request_error, context-length, ENOENT /
"command not found", 402 out-of-credits and 401/403 auth. Anything unmatched is unknown — not
retried, but still counted toward the breaker.
# Default behaviour: breaker on, 3 consecutive failures to trip, 5 min cooldown, at most one retry
cyberine run "some task"
# Trip faster and cool down briefly (useful when probing a flaky endpoint)
CYBERINE_BREAKER_FAILURE_THRESHOLD=2 CYBERINE_BREAKER_COOLDOWN_MS=30000 cyberine run "some task"
# Turn the whole feature off — restores the exact pre-breaker dispatch behaviour, no retry either
CYBERINE_BREAKER=0 cyberine run "some task"
# An open circuit is visible in the same places the quota cooldown is, as kind `circuit-open`
cyberine cooldown
cyberine doctor
curl -s localhost:41700/metrics | grep circuit-openReading a degraded fan-out. A skipped provider is never a silent absence — it shows up twice:
- as an error
Resultwhose message beginsprovider circuit open: <id>and carries the state, the consecutive-failure count, the remaining window and the last failure's reason; - as an
adapterNotesline in CLI/SDK output, the same channelbuildDefaultAdapters()already uses to explain a missing worker.
A real failure that has not yet tripped the circuit gets a [breaker: N/3 consecutive failures, class=hard] suffix, so progress toward a trip is readable rather than hidden.
Spend-cap ordering is preserved in both directions, which is the invariant selftest:orch-resilience
gates: the breaker can only ever refuse a dispatch, never admit one, so every call that reaches a
provider still passed the dispatch guards, the daily reservation and the hard-cap reservation; and a
fast-failed call reserves nothing and is charged nothing. The hard-cap reservation is taken per
attempt rather than once per dispatch, so a retry can never spend budget the cap did not admit.
Known Limitations and Deferred Work. The breaker gates Orchestrator.dispatch() only — not
WorkflowEngine.runWorker(), so a multi-step DAG workflow's own worker calls are unaffected. State
lives in <CYBERINE_HOME>/provider-breaker.json and there is no CLI subcommand to reset it:
cyberine cooldown clear <workerId> clears the operator-visible projection but not the state
machine, whose only reset is programmatic (clearBreaker()) or deleting that file. Because the
dominant entry point is a fresh process per dispatch, the counter is persisted rather than in-memory;
two concurrent dispatches can lose an increment, which delays a trip by one dispatch and never causes
an early one.
Blocker visibility — waitingFor on stalled workers
When a worker cannot make progress, a waitingFor field explains why. It appears in:
Fallback model chain — per-worker model degradation
When a LoopAgentAdapter worker's primary model is overloaded or unavailable (HTTP 429 / 402 / 5xx / unexpected overload 400), Cyberine retries the same request with the next model in a configured...