Always-on operations — graceful drain, health, queue intake, maintenance

The API-first 24/7 model keeps cyberine web supervised and pokes it (or the CLI) with thin external triggers instead of running a fat internal daemon. The pieces below harden that path.

Graceful drain. cyberine web and cyberine mcp register SIGTERM/SIGINT handlers that stop accepting new connections, let in-flight requests / SSE streams finish within a bounded grace window, then exit 0 — crash-only safe so a supervised restart never corrupts state. Detached run children are spawned unref'd, so they are not children of the server and are neither owned nor orphaned by it.

cyberine web --port 41700 &
kill -TERM %1     # -> "[web] SIGTERM received — draining..." -> "[web] drained cleanly — exiting 0"

Health / readiness / metrics (poll these from launchd KeepAlive / systemd / a load balancer):

curl -s 127.0.0.1:41700/health    # {"status":"ok","uptimeSec":N}  — dependency-free, cheap (<5ms)
curl -s 127.0.0.1:41700/ready     # {"status":"ready","checks":{"registry":true,"bus":true}} (503 when not)
curl -s 127.0.0.1:41700/metrics   # run counts by state, active cooldowns, queue depth, concurrency utilization, spendCaps (daily+hard, T636)

Autonomous task intake. Append a task with no human action; a tick consumer drains it under CYBERINE_MAX_CONCURRENT backpressure (wire cyberine queue tick into a launchd/cron timer):

cyberine enqueue "refactor the auth module"      # appends to <CYBERINE_HOME>/queue.jsonl (status: pending)
curl -s -X POST 127.0.0.1:41700/api/enqueue \
  -H 'Content-Type: application/json' -d '{"task":"refactor the auth module"}'   # 202 {id,status}
cyberine queue list                              # show queued tasks + status
cyberine queue tick                              # dispatch pending up to CYBERINE_MAX_CONCURRENT free slots

Enqueue never loses a task to a concurrent tick (T585). tickQueue() rewrites the whole queue.jsonl from an in-memory snapshot to flip statuses; a task appended between that read and the rewrite used to be silently discarded (the tick's atomic rename made the discard atomic, not absent). enqueueTask() now blocks on the same advisory lock tickQueue() holds for its full read-rewrite cycle, so an append can only land before the snapshot (and gets rewritten back) or after the rewrite (and lands on top of it) — never inside the window. Guard: npm run selftest:queue (scripts/t585-smoke.ts) drives the real lock + writeQueue() across two real OS processes and proves both halves: the fixed, lock-protected append survives a concurrent stale rewrite, and a raw unlocked append (the pre-fix behavior) is discarded by that same rewrite.

Completion webhooks (no polling — complements SSE). Set CYBERINE_WEBHOOK_URL (or pass webhookUrl per-run on POST /api/runs) to receive a bounded, retrying POST on run started/done/failed/cancelled with {event, runId, state, workflowName, detail, durationMs, summary}.

Credit watch + starvation alerts. cyberine maintenance (wire into a launchd timer) polls provider balances: it clears an out-of-credits cooldown early once the provider is topped up above the credit threshold, and — when EVERY available build worker is cooled — fires an alert via both the Cyberine Stop hook surface (where a telegram notifier hook is wired), CYBERINE_ALERT_WEBHOOK_URL, and the first-party Telegram channel (CYBERINE_TELEGRAM_BOT_TOKEN + CYBERINE_TELEGRAM_CHAT_ID, k6p4c).

cyberine maintenance   # clear replenished cooldowns + alert if the fleet is starved
cyberine status        # provider balance + cooldown state per worker

Env knobs:

24/7 supervisor — one process, one supervised unit (launchd on macOS, systemd --user on Linux). cyberine serve is the consolidated foreground runtime: a SINGLE process that boots the web server AND runs the cron-tick, reap, and maintenance loops internally (no child processes, no separate timer plists/timers). It is single-instanced via a pidfile (<CYBERINE_HOME>/serve.pid, with stale-PID detection) and drains gracefully on SIGTERM/SIGINT.

cyberine serve                       # run the consolidated process in the foreground (web + ticker)
cyberine serve --port 41700          # same, custom port/host (defaults 127.0.0.1:41700)
cyberine serve install               # macOS: install ONE launchd agent com.cyberine.loop
                                     #        (KeepAlive + RunAtLoad) and PURGE any legacy 4-plist install;
                                     # Linux: install ONE systemd --user unit cyberine-loop.service
                                     #        into ~/.config/systemd/user/, run
                                     #        `systemctl --user daemon-reload` then
                                     #        `systemctl --user enable --now`, and print its
                                     #        `is-active` status;
                                     # other platforms: write a systemd .service + pm2 config under
                                     #        <CYBERINE_HOME>/supervisor/ and print activation steps
                                     #        (nothing is auto-installed there)
cyberine serve uninstall             # macOS: unload + remove com.cyberine.loop AND all legacy
                                     #        labels (idempotent);
                                     # Linux: `systemctl --user disable --now cyberine-loop`, remove
                                     #        the unit file, `systemctl --user daemon-reload`

After serve install, launchd (macOS) or systemd --user (Linux) respawns the process on crash and starts it at login/session-start. macOS shows a single background item. (A code-signed Cyberine.app for a friendly "Cyberine Loop" display name is tracked separately and needs an Apple Developer ID cert.) The cyberine shim forwards SIGTERM/SIGINT to the runtime, so a supervisor stopping it triggers the graceful drain (not an orphaned child).

Linux systemd --user unit. cyberine-loop.service runs <cyberine> serve with Type=simple, Restart=on-failure / RestartSec=5, WantedBy=default.target (the user-manager analogue of multi-user.target), and logs to <CYBERINE_HOME>/supervisor/logs/serve.log / serve-err.log via StandardOutput=append:... / StandardError=append:.... Check it directly with the normal systemd tools:

systemctl --user status cyberine-loop     # active/inactive/failed + recent log tail
journalctl --user -u cyberine-loop -f     # follow its stdout/stderr

Known limitations (systemd path). serve install targets a per-user unit only — it never writes a system-wide unit under /etc/systemd/system/ (no root required, but also no before-first-login start). A --user unit stops when the invoking user's last login session ends unless lingering is enabled; serve install prints the command but does NOT run it for you (opt-in on purpose — enabling lingering is a separate, machine-wide privilege this CLI should not spend silently):

loginctl enable-linger $(whoami)          # keep cyberine-loop running after you log out
loginctl disable-linger $(whoami)         # revert

A host with no systemd user manager at all (some minimal containers, non-systemd distros such as Alpine/OpenRC or Void) is not detected separately from a normal Linux host — serve install still attempts systemctl --user ... there and the shell-out fails with whatever error systemctl itself reports; use the on-disk pm2 template (<CYBERINE_HOME>/supervisor/cyberine.pm2.config.js, written on every serve install regardless of platform) as the cross-distro fallback in that case.

Crash-resume + checkpointing. On cyberine serve startup (and as a cyberine reap step) the registry is scanned for runs stuck in running whose PID is dead: a checkpointed run is requeued so a re-dispatch skips its completed steps, a non-checkpointed one is cancelled — nothing is left dangling. WorkflowEngine checkpoints each step's output under the run dir, so a resumed run continues from the last completed step instead of restarting.

# (logged at startup) [cyberine serve] crash-resume: 1 requeued, 0 cancelled
#                     [cyberine serve] crash-resume: <runId> resumed — Requeued with 1/3 checkpointed step(s)

Heartbeat + health. Each serve tick writes <CYBERINE_HOME>/daemon-status.json (heartbeat ts, uptime, tick count, last cron/reap/maintenance ts, active runs, cooled providers, queue depth). cyberine status renders a Supervisor-health block from it; a missing or stale heartbeat (older than CYBERINE_HEARTBEAT_STALE_MS, default 3x the cron-tick interval) is reported as DOWN.

cyberine status   # ... Supervisor health: UP — heartbeat 4s ago / uptime / ticks / active runs / queue depth

Idempotent dispatch / enqueue. Pass an Idempotency-Key header (or an idempotencyKey body field) on POST /api/runs or POST /api/enqueue; a retried trigger within the TTL returns the FIRST request's response (same runId / enqueue result) instead of double-dispatching. Keys persist under <CYBERINE_HOME> and survive a restart; requests with no key behave exactly as before.

A stored entry addresses the request, not the key string alone: the key is scoped to the HTTP method plus the resolved API path, so the same key sent to /api/runs and to /api/enqueue is two independent entries and neither replays the other's response. Only a response that actually committed something (status < 400) claims the key — a 4xx/5xx leaves it unclaimed, so a retry under the same key with a corrected body runs normally instead of replaying the earlier failure for the rest of the TTL.

KEY=$(uuidgen)
curl -s -X POST 127.0.0.1:41700/api/runs -H "Idempotency-Key: $KEY" \
  -H 'Content-Type: application/json' -d '{"task":"build the login page"}'   # dispatches -> {runId}
curl -s -X POST 127.0.0.1:41700/api/runs -H "Idempotency-Key: $KEY" \
  -H 'Content-Type: application/json' -d '{"task":"build the login page"}'   # SAME runId, no second run