Long-term memory for Cyberine — survey + design

• Status: W1 (store + lineage plumbing) IMPLEMENTED 2026-06-07 via dogfood run 3eeac0ff — src/store/MemoryStore.ts, RunLineage on Run/GlobalRunRecord, env threading in launchBg.ts/cyberine-bg-run.ts, engine writes at all 3 terminal paths; E2E-verified (record + parent/chat lineage + auto previousRunId chain). W2 (outcome extraction + issue ledger writers) IMPLEMENTED 2026-06-07 via dogfood run 7c48bdb4 — parseOutcomeSections + issue-event writers in MemoryStore.writeRecord, OUTCOME_TRAILER contract on the implement step of implementTaskWorkflow.ts; E2E-verified (trailer parsed from verdict winner text, opened events deduped, fold lists open issues). W3 (retrieval + injection + cyberine memory CLI) IMPLEMENTED 2026-06-07 via dogfood runs b0a376cb/4ac190d3/2438c7a5/4e11277d — memoryRetrieval.ts scorer + preamble builder + strip markers, dispatch injection in launchBg.ts, chat session-start injection in ChatApp.tsx, cyberine memory subcommand in cyberine.ts; E2E-verified (live dispatched run received scored preamble, record task stripped clean, all 6 CLI verbs exercised). W4 (optional layers) IMPLEMENTED 2026-06-07 via dogfood runs 1bd75f39/310188fc — ADD-only LLM distillation (CYBERINE_MEMORY_MODEL) + ollama embeddings with cosine axis (CYBERINE_MEMORY_EMBEDDINGS) in MemoryStore.ts/memoryRetrieval.ts, lineage chain line in RunDetail.tsx (locked by selftest scenario 11); E2E-verified against local ollama (llama3.1:8b distillation refined a record summary; nomic-embed-text vectors re-ranked a semantic query).
• Researched/verified: 2026-06-07 — deep-research harness, 23 sources fetched, 25 claims adversarially verified (14 confirmed / 11 killed), grounded against the Cyberine codebase at HEAD ca89b7e.
• Goal (user ask): Cyberine phải có long-term memory, luôn traceable previous / current / next / issues — both a provenance trail AND retrievable cross-run memory.

Part 1 — Survey: how the field does agent memory (2024-2026)

1.1 Taxonomy

Agent memory is organized into five cognitive categories — episodic (experiences situated in time), semantic (abstract facts), procedural (skills, policies), working (in-context manipulation), sensory (perceptual buffering). The practical value lies in how memory is designed and used, not cognitive-science fidelity (arXiv:2602.06052, verified 3-0).

External memory substrates fall into four concrete types: vector index (embedding + ANN), text-record (sequential logs), structural store (graph/relational), hierarchical store. The same survey warns plain vector retrieval "can be unreliable" and injects noise when ranking is weak (arXiv:2602.06052, verified 3-0) — a reason NOT to make embeddings the only retrieval axis.

Mapping to Cyberine today:

1.2 Reference architectures (verified findings)

1.3 Two verified warnings that shape the design

1. Summarization drift (arXiv:2603.07670, verified 2-1): each compression pass silently discards low-frequency details; after enough passes the agent "remembers" a sanitized, generic history that fails on edge cases. → Cyberine must keep raw episodic records append-only and non-lossy; only indexes get compacted, never the records themselves.
2. Checkpoint ≠ provenance (arXiv:2603.21692, verified 3-0): LangGraph-style checkpoints capture execution state but not why the agent acted, what it concluded, how that shaped strategy. Cyberine run.json has exactly this gap — it stores winnerOutput (the diff) but not the conclusions/issues/next-steps. → The memory record must capture structured outcome fields, not just artifacts.

1.4 What was NOT verified (do not build on these)

• Zep's headline benchmark numbers (94.8% DMR, 90% lower latency) — refuted 0-3 in verification.
• "Multi-signal scoring (recency + relevance + importance) outperforms pure similarity" (Generative Agents) — refuted 0-3 as a claim; used below only as a design heuristic, not a research-backed fact.
• Bi-temporal invalidation may only pay off at multi-user scale (open question in research) — Cyberine is single-user, so we use append-only + supersededBy instead of 4 timestamps.

Part 2 — Ground truth: what Cyberine has and lacks (HEAD ca89b7e)

Existing persistence surface:

• RunStore — .local/runs/<runId>/run.json + events.jsonl (src/store/RunStore.ts:32); Run/RunStep/WorkerRun shapes in src/store/runTypes.ts:58-79.
• GlobalRegistry — $CYBERINE_HOME/runs/<runId>.json, GlobalRunRecord (src/store/GlobalRegistry.ts:23-41), tombstone delete.
• ChatSessionStore — $CYBERINE_HOME/chat-sessions/<id>.json incl. full apiMessages (src/store/ChatSessionStore.ts:51-61).
• Hooks — PreToolUse | PostToolUse | PreDispatch | RunStart | RunEnd with CC exit-code contract (src/core/hooks.ts:90).

Gaps (verified by codebase sweep 2026-06-07):

1. Zero run-to-run linkage — no parentRunId/previousRunId/sourceSessionId anywhere; runs are isolated UUIDs.
2. Task.context (src/core/types.ts) is an unused stub — natural injection point exists but nothing populates it.
3. No outcome summary — winnerOutput holds raw diff/text; nothing stores "what was accomplished, what issues remain, what's next".
4. No cross-run retrieval — listRuns() sorts by updatedAt only.
5. Chat↔run link is in-memory only — chatRunsRef Map in ChatApp.tsx is never persisted; after resume, a session cannot say which runs it dispatched.
6. RunEnd hook payload lacks step outcomes (needs RunStore.get(runId) to read them).
7. No $CYBERINE_HOME/memory/ namespace; CYBERINE_HOME resolution is copy-pasted in 5 files (extract cyberineHome() helper first).

Part 3 — Design

3.0 Principles

• Episodic base is non-lossy and append-only (Graphiti episode-subgraph pattern; avoids summarization drift). One memory record per run, never rewritten — corrections happen via supersededBy.
• Lineage is first-class, retrieval is layered on top. Traceability (previous/current/next/issues) must work with grep alone; embeddings are an optional enhancement, never a dependency.
• No vector DB. Single-user scale (<10k records): brute-force cosine over JSON-stored vectors is milliseconds; BM25-style keyword + recency works with zero infra. Multi-axis retrieval is the verified pattern (Zep, 2-1).
• File-based, human-readable — same philosophy as .local/runs/.

3.1 Storage layout

$CYBERINE_HOME/memory/
  <project-slug>/              # slug = sanitized basename of cwd (per-project isolation)
    records/<runId>.json       # episodic record, one per run, append-only
    issues.jsonl               # issue ledger, append-only state transitions
    index.json                 # compact derived index (rebuildable from records/)
    embeddings.jsonl           # optional: {runId, model, vector[]} one line per record

index.json is a derived cache (like GlobalRegistry vs run dirs) — cyberine memory rebuild regenerates it from records/, mirroring the existing backfillRegistry pattern.

3.2 Schemas

MemoryRecord (episodic, one per run — written at run end):

interface MemoryRecord {
  runId: string;
  cwd: string;
  workflowName: string;
  task: string;                  // full task prompt
  state: 'done' | 'error' | 'cancelled';
  startedAt: number;
  endedAt: number;
  // --- lineage (PROV-AGENT-inspired: wasInformedBy / wasGeneratedBy) ---
  lineage: {
    parentRunId?: string;        // run that spawned this run (wasInformedBy)
    chatSessionId?: string;      // chat session that dispatched it
    previousRunId?: string;      // latest prior run in same project at dispatch time
    rootRunId?: string;          // first run of the chain
  };
  // --- structured outcome (closes the reasoning-provenance gap) ---
  outcome: {
    summary: string;             // 2-4 sentences: what was attempted, what happened
    accomplished: string[];      // concrete deliverables ("shipped X", "fixed Y")
    issues: string[];            // ids into issues.jsonl opened by this run
    next: string[];              // recommended next steps
  };
  workers: { agentId: string; provider?: string; state: string }[];
  verdictSummary?: string;       // judge verdict one-liner, if any
  supersededBy?: string;         // runId of a correcting record (invalidate, never delete)
}

IssueRecord (issues.jsonl, append-only transitions — bi-temporal-lite):

interface IssueEvent {
  issueId: string;               // stable slug
  at: number;
  event: 'opened' | 'resolved' | 'reopened' | 'wontfix';
  byRunId: string;               // provenance: which run changed the state
  title: string;
  detail?: string;
}

Current issue state = fold over events (last event wins). previous/current/next/issues traceability falls out directly: previous = lineage.previousRunId chain, current = the record, next = outcome.next, issues = open-issue fold filtered by cwd.

3.3 Write path

Primary write point: WorkflowEngine run-end (all three terminal paths already converge on the RunEnd hook dispatch — src/engine/WorkflowEngine.ts). The engine calls MemoryStore.writeRecord(run) directly (product feature, not a user hook; the RunEnd hook stays for user extensions).

outcome generation, two modes:

• Extractive (default, zero-cost): summary from task + final state + verdict + winner-output head; issues/next parsed from worker output sections when the implementer-fleet prompt template asks workers to emit an ISSUES: / NEXT: trailer (prompt-contract change in implementTaskWorkflow.ts).
• LLM-distilled (opt-in, CYBERINE_MEMORY_MODEL): one cheap call (default deepseek-v4-flash) that maps run.json → outcome JSON. Keep it single-shot ADD-only — Mem0's verified lesson is that LLM consolidation degrades sharply with conflict count (61%→25%), so we never let the LLM UPDATE/DELETE existing records.

Update 2026-09-22: there is no built-in default model — distillConfig() returns null and distillation is skipped entirely unless CYBERINE_MEMORY_MODEL is set, see src/store/MemoryStore.ts:356.

Lineage capture:

• launchBackgroundRun() (src/launchBg.ts:71) gains optional { parentRunId, chatSessionId }; threads them via env (CYBERINE_PARENT_RUN_ID, CYBERINE_CHAT_SESSION_ID) to the detached child, same mechanism as CYBERINE_RUN_ID today.
• ChatApp passes its session id on dispatch — this also fixes gap #5 by persisting dispatched run ids into ChatSessionRecord (new field dispatchedRuns: string[]).
• previousRunId = latest record in the project's index.json at dispatch time.
• Run (runTypes.ts) and GlobalRunRecord (GlobalRegistry.ts) gain the same optional lineage block so the TUI can render chains without opening memory files.

3.4 Read path (retrieval + injection)

Multi-axis scoring, Zep-style but file-based (no DB):

score(record, query) =
    w_kw  * keywordScore(query, record)     # BM25-ish over task+summary+issues
  + w_rec * exp(-ageDays / halfLife)        # recency decay
  + w_lin * lineageBoost                    # same chain / same chat session
  [+ w_vec * cosine(qVec, record.vector)]   # only when embeddings enabled

Injection points:

1. Dispatch-time (automatic): launchBackgroundRun builds a MEMORY preamble — top-K records (default 3) + open issues for the cwd — and prepends it to the task prompt (or fills Task.context, the existing unused stub). Capped (default ~1500 tokens), disabled with CYBERINE_MEMORY=0.
2. Chat planner (automatic): system prompt gains the same block at session start — the planner can answer "what were we doing / what's open".
3. Explicit: cyberine memory subcommand — list, show <runId>, search <query>, issues [--all], chain <runId> (walk lineage both directions), rebuild.

Embeddings (optional, OFF by default): CYBERINE_MEMORY_EMBEDDINGS=ollama uses local nomic-embed-text via Ollama (/v1/embeddings — pairs with the Ollama provider integration, docs/providers/ollama.md); any OpenAI-compatible embeddings endpoint also works. Vectors stored as JSON lines; brute-force cosine at query time.

3.5 Compaction & pruning

• Records are never rewritten (drift-avoidance). index.json keeps per-record one-liners; if it exceeds a cap (default 500 entries) the oldest index entries collapse into a per-project archive-summary.md — the underlying records stay on disk.
• Resolved issues older than N days (default 90) drop out of the default issues view (--all shows everything).
• supersededBy chains: cyberine memory show follows them and renders the newest truth, marking superseded records.

3.6 Config (new env vars)

3.7 Implementation plan (dogfood waves)

Per .claude/rules/dogfood-first.md — each wave is a file-disjoint slice runnable through the orchestrator:

1. W1 — store + lineage plumbing: cyberineHome() helper; src/store/MemoryStore.ts (records + issues + index); lineage fields on Run/GlobalRunRecord; env threading in launchBg.ts; engine write at run end. Verify: run a task, record appears, chain fields populated.
2. W2 — outcome extraction + issue ledger: extractive outcome builder; ISSUES:/NEXT: prompt-contract in implementTaskWorkflow.ts; issue fold logic. Verify: dogfood run emits issues, cyberine memory issues lists them.
3. W3 — retrieval + injection: scorer, MEMORY preamble in dispatch + chat system prompt; cyberine memory CLI. Verify: second run on a related task receives the first run's summary; PTY selftest scenario for the chat block.
4. W4 — optional layers: LLM distillation; Ollama embeddings; TUI lineage rendering (chain view in run detail).

Each wave: npm run typecheck before integration; recipes land in USAGE.md per feature-docs.md.

Sources

• arXiv:2602.06052 — Rethinking Memory Mechanisms of Foundation Agents (taxonomy, substrates)
• arXiv:2310.08560 — MemGPT
• arXiv:2603.07670 — agent-memory survey (summarization drift)
• arXiv:2501.13956 — Zep/Graphiti (bi-temporal graph, tri-tier, multi-axis retrieval)
• arXiv:2504.19413 — Mem0 (4-op consolidation, degradation limits)
• arXiv:2602.14038 — FluxMem (learnable memory selection)
• arXiv:2603.21692 — Reasoning Provenance for Autonomous AI Agents
• arXiv:2508.02866 — PROV-AGENT (W3C PROV extension, IEEE e-Science 2025)
• Cyberine codebase sweep 2026-06-07 (file:line anchors above)