Lane grouping — when lanes help vs flat fan-out

The T269 lane runner is the default swarm executor in --mode swarm. It runs the plan through clusterChunks + packLanes BEFORE workers fan out, so the chunks a worker can see on disk follow the dependency graph instead of being a textual convention the planner must restate in every prompt. Three knobs (T268) control the lane layer end-to-end:

The combined-graph lane invariant (T263 / T268 / src/engine/swarmPlan.ts:248). Build an undirected graph over the chunk ids whose edges are the UNION (or selection — see CYBERINE_SWARM_GROUP_BY) of:

• file-overlap edges — two chunks whose ownsFiles overlap on at least one path (reusing areClaimSetsDisjoint from src/core/scheduler.ts; an empty ownsFiles is "unknown scope" and conflicts with everything, matching the scheduler's claim-set semantics).
• dependsOn edges — the chunk DAG edges declared in the plan.

Connected components of that graph become one SwarmLane each. Within a lane:

• the chunks share a single worktree (provisioned by runSwarmLaneRunner via provisionWorktree);
• the chunks run serially in DFS post-order topological sort of dependsOn — the dependent chunk ALWAYS starts after its upstream chunk's commit is in the worktree;
• the lane emits a SwarmLaneDiffSummary to RunStep.laneDiffs whose diff field is the cumulative diff from the lane's base ref to its HEAD (the union of every per-chunk commit in the lane), commitShas is the per-chunk commit SHAs in execution order, and gaps is the hard-failed/skipped chunk ids (always loud, never silent).

Across lanes, no two lanes touch the same files and no two lanes have a happens-before edge — so workers in different lanes can run in parallel under the cap. The end-of-run LaneMergeJudge (T267) applies the per-lane cumulative diffs sequentially to a clean integration worktree off HEAD, gates on npm run typecheck, and emits ONE integrated diff as the step's winnerOutput (consumed byte-for-byte by cyberine apply <runId> / cyberine land <runId>).

When lanes help vs flat fan-out. The T263+T264 grouping layer is almost always a win — a plan that does not benefit from grouping still works (it just collapses to one component per worker), and a plan that does benefit (any two chunks that share a file or have a dependsOn edge) gets a sound execution model without the planner having to restate the dependency context in every prompt. The three times to consider a knob:

• CYBERINE_SWARM_GROUP_BY=files — chunk scopes are disjoint AND the planner's dependsOn edges are mostly an implementation detail of one cluster (e.g. the planner chained "define AuthUser" -> "use AuthUser" inside the auth module). File-overlap-only groups them, and the false-positive deps edges go away. The smoke's 6-chunk / 3-cluster plan produces the same 3 lanes under files / deps / both; this knob only matters when deps would over-merge a plan.
• CYBERINE_SWARM_GROUP_BY=deps — chunk scopes are disjoint (no two chunks share a file) but the implementation must run sequentially (e.g. "create a new endpoint" then "wire it into the router"). With files-only the clusterer would put them in different lanes and race the apply; with deps they stay in one lane and run serially. Rare; use only when scopes are provably disjoint and order matters.
• CYBERINE_SWARM_LANES=K — when the natural lane count is larger than the fleet (K < lanes.length), packLanes snake-packs the disjoint lanes under the cap (balanced by chunk count, output bins pairwise disjoint on ownsFiles — the LaneMergeJudge's disjointness gate still passes). When K > lanes.length it's a no-op. Default 0 = auto cap = fleet size. Override when you need fewer worktrees than the fleet has workers (small fleet, big worktree budget), or when you want to squeeze the bin packing yourself.

Failure semantics (T266). When a chunk hard-fails AFTER its single retry, the lane aborts per failureMode: 'stop' (default): every remaining chunk in the lane is reported in lane.gaps; the committed prefix diff (chunks that ran successfully before the hard fail) is kept. failureMode: 'skip-non-descendants' aborts the descendants of the failed chunk (reported in gaps) but attempts to run non-descendant chunks that have no dependsOn chain into the failed chunk. Surfacing is always loud — any chunk the runner chose to skip ends up in gaps, never silently dropped. The downstream LaneMergeJudge re-validates the declared ownsFiles disjointness across the surviving lane diffs and surfaces any out-of-lane stray as a failure verdict (not a silent drop).

Smoke evidence. The non-TUI smoke scripts/t271-grouped-swarm-smoke.ts exercises the full grouped pipeline (plan -> cluster -> lanes -> lane-merge) on a synthetic 6-chunk / 3-cluster plan with a MOCKED worker dispatch (no live provider needed). It asserts: 3 lanes formed under all 3 groupBy values, each lane's chunks committed in order, the end-of-run merge produces ONE integrated diff that applies cleanly + is internally consistent (every chunk's marker in the right file, in commit order, on a fresh worktree re-apply). The fully-LIVE end-to-end run (real LLM workers) is creds-gated; this smoke verifies the orchestration with a mocked dispatch. Evidence lives under .local/dogfood/t271-evidence/.

Chunk dependsOn — auto-routing and the plan-time DAG gate

A dependsOn edge between two swarm chunks is only real on the lane-runner path: it is the lane runner that puts the two chunks in one worktree and commits the upstream chunk before the dependent one starts. Reaching that path used to require the def author to ALSO set swarmUseLaneRunner: true; a plan that declared edges without the flag went down the classic per-chunk path, where every chunk is dispatched in one Promise.all and the edges do nothing. That was reachable both from a hand-authored WorkflowDef and from a YAML recipe (swarm_chunks[].depends_on with no swarm_use_lane_runner).

Two gates close it, both in the engine and neither needing anything from the def author:

• Auto-routing. resolveSwarmRunnerRouting (src/engine/swarmLaneRunner.ts) sends any plan that declares a dependsOn edge to the lane runner. Precedence: swarmUseLaneRunner: true > CYBERINE_SWARM_LANE_RUNNER=1 > the plan declares edges (auto-route) > the classic per-chunk fan-out. A plan with NO edges is unaffected — same path and same behaviour as before. The engine logs one line naming the decision whenever the plan has edges.
• Plan-time DAG validation. WorkflowEngine.execute runs validateChunkDag (src/engine/swarmPlan.ts) over every step's swarmChunks, alongside the step-level topoSort and BEFORE any worker or run record exists. It throws on a cycle (naming the path, e.g. a -> b -> a), a self-dependency, a duplicate chunk id, and a dependsOn pointing at a chunk that is not in the plan.

The A/B escape hatch is an explicit swarmUseLaneRunner: false: it keeps the classic per-chunk path even for a plan with edges, and the engine prints a WARNING: ... declares swarm chunk dependsOn edges that will NOT order execution line so the choice is never silent.

Verify recipe.

# The ordering guard: a 3-chunk graph (chunk-a -> chunk-b, chunk-c) with DISJOINT ownsFiles
# run through the real lane runner against a throwaway git repo, plus the routing truth
# table, the DAG-rejection cases, and the engine wiring. No LLM, no network.
npm run selftest:swarm-dep-order

# The neighbours it shares a path with
npm run selftest:swarm-lane-runner
npm run selftest:swarm-wiring

Known limitations and deferred work.

• Ordering is delivered by the lane runner, not by wave-based parallel dispatch. Chunks joined by a dependsOn edge land in ONE lane and run SERIALLY in that lane's worktree, so a fan-out like a -> b, c runs b and c one after the other rather than in parallel once a is done. That is the price of the in-tree guarantee (a dependent sees its upstream's commit); a wave scheduler that ran b and c concurrently off a's commit is not implemented.
• The DAG gate is fail-fast for the whole run: one cyclic or dangling chunk graph in ANY step aborts execute before the first worker starts. Notably, a SwarmPlanner LLM that hallucinates a dependsOn label now fails the run with a named error instead of silently dropping the edge (src/engine/swarmPlanner.ts deliberately preserves the unresolved label for exactly this).
• Auto-routing keys on the plan's edges only. It does not inspect the chunk prompts, so a plan that expresses an ordering in prose instead of dependsOn still fans out in parallel — declare the edge.
• {{chunkId}} splicing of a completed dependency's OUTPUT into a dependent chunk's prompt is still not implemented; the dependent sees its upstream's work through the lane worktree and the lane digest, not through prompt substitution.

Swarm run recipe — cyberine run --mode swarm with the grouping knobs

# Plain swarm — let the auto lane cap (fleet size) do the work
cyberine run --mode swarm --bg "refactor the auth module across all files"

# Tighter lanes — 3 lanes max (forces packLanes snake-pack if the clusterer
# produced more disjoint components)
CYBERINE_SWARM_LANES=3 \
  cyberine run --mode swarm --bg "split the audit subsystem"

# File-overlap-only grouping — use when chunk dependsOn edges are an
# implementation detail of one cluster
CYBERINE_SWARM_GROUP_BY=files \
  cyberine run --mode swarm --bg "add JSDoc to src/utils/*.ts"

# Audit knob — warn when any lane has more than 4 chunks (likely
# serialising too much per worktree)
CYBERINE_SWARM_MAX_LANE_SIZE=4 \
  cyberine run --mode swarm --bg "refactor the routing layer"

# Stack — pick the workers, the planner cap, the grouping, and the lane cap
CYBERINE_SWARM_FLEET="opencode-deepseek,claude,codex" \
CYBERINE_SWARM_MAX_CHUNKS=6 \
CYBERINE_SWARM_GROUP_BY=both \
CYBERINE_SWARM_LANES=4 \
  cyberine run --mode swarm --bg "refactor the audit subsystem"

# Or run the workflow directly (no `cyberine run` parent) for dogfooding:
CYBERINE_TASK="Add a sandbox-pinned grep_files tool as src/tools/grep.ts" \
CYBERINE_SWARM_FLEET="opencode-deepseek,claude,codex" \
CYBERINE_SWARM_GROUP_BY=both \
CYBERINE_SWARM_LANES=4 \
  npm run wf -- src/engine/swarmTaskWorkflow.ts