cyberine
UsageRun recipes

Provider circuit breaker and bounded retry — one dead provider degrades the fleet, not the run

Distinct from the quota cooldown above, which reacts only to a 402/429 quota signal: the breaker counts consecutive terminal failures of any shape at the Orchestrator.dispatch() seam, so it also catches the failure modes no pattern recognises. State machine per provider: closed -> open (fast-fail for a cooldown window) -> half-open (exactly one probe admitted) -> closed on a successful probe, or straight back to open on a failed one.

A failure is classified before anything is retried. Only transient is retried (HTTP 5xx/529, bad gateway, service unavailable, overloaded, ERR_STREAM_PREMATURE_CLOSE, EPIPE, timeouts, plus rate-limit and connection failures). A hard failure is never retried, because paying twice for it buys nothing: model-not-found, 400/404/422, invalid_request_error, context-length, ENOENT / "command not found", 402 out-of-credits and 401/403 auth. Anything unmatched is unknown — not retried, but still counted toward the breaker.

# Default behaviour: breaker on, 3 consecutive failures to trip, 5 min cooldown, at most one retry
cyberine run "some task"

# Trip faster and cool down briefly (useful when probing a flaky endpoint)
CYBERINE_BREAKER_FAILURE_THRESHOLD=2 CYBERINE_BREAKER_COOLDOWN_MS=30000 cyberine run "some task"

# Turn the whole feature off — restores the exact pre-breaker dispatch behaviour, no retry either
CYBERINE_BREAKER=0 cyberine run "some task"

# An open circuit is visible in the same places the quota cooldown is, as kind `circuit-open`
cyberine cooldown
cyberine doctor
curl -s localhost:41700/metrics | grep circuit-open

Reading a degraded fan-out. A skipped provider is never a silent absence — it shows up twice:

  • as an error Result whose message begins provider circuit open: <id> and carries the state, the consecutive-failure count, the remaining window and the last failure's reason;
  • as an adapterNotes line in CLI/SDK output, the same channel buildDefaultAdapters() already uses to explain a missing worker.

A real failure that has not yet tripped the circuit gets a [breaker: N/3 consecutive failures, class=hard] suffix, so progress toward a trip is readable rather than hidden.

Spend-cap ordering is preserved in both directions, which is the invariant selftest:orch-resilience gates: the breaker can only ever refuse a dispatch, never admit one, so every call that reaches a provider still passed the dispatch guards, the daily reservation and the hard-cap reservation; and a fast-failed call reserves nothing and is charged nothing. The hard-cap reservation is taken per attempt rather than once per dispatch, so a retry can never spend budget the cap did not admit.

Known Limitations and Deferred Work. The breaker gates Orchestrator.dispatch() only — not WorkflowEngine.runWorker(), so a multi-step DAG workflow's own worker calls are unaffected. State lives in <CYBERINE_HOME>/provider-breaker.json and there is no CLI subcommand to reset it: cyberine cooldown clear <workerId> clears the operator-visible projection but not the state machine, whose only reset is programmatic (clearBreaker()) or deleting that file. Because the dominant entry point is a fresh process per dispatch, the counter is persisted rather than in-memory; two concurrent dispatches can lose an increment, which delays a trip by one dispatch and never causes an early one.