Belay: An experience layer for coding agents.

AI coding agents can solve difficult problems, but practical learning is often trapped inside one session. A developer corrects an approach, explains an important constraint, and verifies the final solution—then repeats the same explanation days later.

Belay addresses that gap by separating observed work, deterministic analysis, optional AI interpretation, developer approval, and future delivery. A Go process reads native Claude Code and Codex transcripts, joins them to a minimized canonical event stream, encrypts retained content in local SQLite, and derives turn-cited issues and reusable experience candidates. Belay adds no hosted model and does not upload coding sessions to Doplex Labs.

In our first evaluation across new software and machine-learning tasks, Belay-assisted agents met the acceptance criteria on 11 of 16 tasks where an earlier lesson was useful. Agents without additional guidance did so on 6 of 16; agents given a concise human-written instruction did so on 7 of 16.

The evaluation also showed that the receiving agent and the limits of each lesson matter. Those findings shape Belay's next design priorities: agent-specific delivery, explicit scope, focused verification, and real-world longitudinal measurement.

1. The recurring learning problem.

Software development contains judgment that is difficult to capture in general documentation: which tests must run after a certain change, why an earlier implementation was rejected, how an unusual edge case differs, and when a normally useful rule should not apply.

Developers teach this knowledge naturally while working with agents. They correct mistakes, reject approaches, explain project constraints, and verify successful repairs. Most of that teaching remains buried in session history.

Transcript search can recover information, but retrieval alone does not determine which lesson matters, whether the original work was correct, where the lesson applies, or how the next agent should verify it.

Teach an important lesson once, then help future agents use it correctly.

2. Belay's approach: verified experience.

Belay turns selected moments from past agent work into approved guidance for future work. It keeps each lesson connected to its supporting evidence and preserves the developer as the authority.

1. A coding agent attempts a task.
2. The developer corrects an approach or establishes a rule.
3. The final solution is checked with tests or another concrete result.
4. Belay links the lesson to the original conversation and verification.
5. The developer reviews, scopes, and approves the lesson.
6. A future agent receives it while working on a related task.

We call this verified experience: a reusable lesson that remains connected to the work that produced it and the evidence used to check it.

3. Technical architecture.

Belay Local is a single-machine Go application with an encrypted SQLite store, an embedded browser interface, and a bounded stdio MCP server. It adds a content-bearing transcript sidecar alongside Belay's existing canonical event pipeline rather than replacing it.

Two evidence representations, one session identity

Numbat supplies a minimized canonical stream of session, command, tool, file, approval, and safety observations. Native readers independently parse the richer harness transcripts needed for verbatim evidence and conversational analysis:

Both paths derive the same ses_ session key from harness identity and native session ID. This lets Belay join minimized operational events to transcript turns without making either representation carry the other's responsibilities. Projects group by sanitized Git remote when available and by local project path otherwise.

Incremental ingestion

The live worker polls every two seconds. It resumes each JSONL file from a durable byte offset, processes only complete records, and detects truncation or rotation using file identity and a prefix hash. Historical backfill is drained separately so a large archive does not prevent an active session from appearing.

Encrypted local schema

Belay stores queryable metadata separately from sensitive content. The transcript_sessions table contains project identity, coverage, duration, token totals, cost totals, and turn counts. The transcript_turns table indexes session, order, time, role, tool, model, token counts, and known cost.

The encrypted turn payload contains retained user and assistant text, tool input and result, raw command, working directory, branch, subagent parent, parser and price-table versions, source-file identity, and the original JSONL byte offset used for citation. Tool input and output are capped at 16 KiB; truncation preserves both the head and tail.

Separation of concerns

Belay separates layers that ordinary memory systems often blend together:

4. Detection, evidence, and cost attribution.

Belay's issue layer is deterministic. It analyzes transcript turns per project, aggregates matching fingerprints across sessions, and emits a cost-ranked record containing the affected sessions, first and last occurrence, an eight-week trend, two to five cited excerpts, and a proposed fix target.

Normalization and fingerprints

Command and error signatures normalize volatile values such as numbers, hashes, temporary paths, absolute paths, and ports. Fingerprints combine the project with the normalized behavior, allowing the same failure mode to be recognized in later sessions without comparing full transcript text.

Traceable cost estimates

Each detector defines the exact span of turns attributable to the issue. Belay sums input, output, cache-read, and cache-write tokens in those spans and estimates dollars from a versioned in-repository price table keyed by exact model ID. An unknown model produces an unknown dollar value; Belay does not substitute or guess a price.

When issue spans overlap, each turn is counted once in aggregate waste. Wall time uses observed turn intervals and excludes gaps longer than 30 minutes. A lower-bound flag remains attached when coverage or pricing is incomplete. Every resulting number can therefore be traced back to a finite set of stored turns.

Bounded semantic interpretation

Some useful lessons—especially repeated developer corrections—require semantic judgment. belay analyze sends only bounded deterministic candidates and cited excerpts to the user's installed harness. Claude Code runs headlessly with tools disabled and session persistence disabled; Codex runs ephemerally in a read-only sandbox.

The harness must return JSON conforming to a closed schema containing candidate clusters, one-line rule text, target file, and confidence. Belay validates referenced IDs and approved targets before persistence. Each result records the harness, reported model, prompt version, input hash, and generation time. The model proposes; it does not activate guidance.

5. From observation to approved experience.

Belay represents reusable guidance as a lifecycle rather than an undifferentiated memory. Deterministic evidence can produce a candidate. Optional harness analysis can produce a proposal. The proposal begins with no instruction authority.

Experiences are versioned and project-scoped. Their state can be approved, active, paused, contradicted, superseded, or expired. Transitions are recorded with actor and reason. Approval and lifecycle mutations use short-lived, content-bound action tokens so a stale preview cannot authorize a changed proposal.

Mission Packs

A future agent receives a short Mission Pack rather than an entire transcript. The pack is generated for an exact project, harness, and task intent; it can include known traps, approved operating rules, verification commands, and evidence references. At most three approved experience items are included.

Mission Pack output is explicitly marked as an inactive proposal with untrusted evidence until the developer approves its use for that session. Acceptance creates a local receipt but does not edit project files. Belay can then bind that receipt to the compatible destination session and report delivery and deterministic evaluation status.

Local interfaces and failure behavior

The browser binds only to numeric loopback and requires a fresh launch token. Writes additionally validate origin and host. The MCP interface runs over bounded stdio, uses closed input and output schemas, and labels transcript-derived strings as untrusted observations. If acquisition, analysis, or Belay itself fails, the coding agent continues operating; Belay is not on the execution path of the user's work.

6. Evaluation design.

We evaluated whether lessons established in earlier coding-agent work could improve unfamiliar tasks. The final evaluation used eight purpose-built, compact Python code-repair families modeling software-systems and machine-learning correctness problems. They were held back from product development.

Each family began with an earlier source task that established a correct rule and one important limit. It then supplied two new tasks: a related task where the rule should help, and a boundary task where applying the rule too broadly should cause a correctness failure.

For every target, the agent received a fresh workspace, was instructed to edit only task.py, and could run python3 -m unittest -v. These tasks tested transfer of a specific correctness lesson under repeatable conditions; they were not intended to reproduce the size or ambiguity of a production repository.

What the agents were asked to solve

Comparison conditions

Claude Code and Codex attempted every new task under three conditions:

1. No additional guidance: the task and workspace only.
2. Human-written instruction: the task plus the exact concise rule and boundary.
3. Belay: the task plus a developer-approved Mission Pack created from the earlier source session.

The human instruction contained the rule and exception. The Mission Pack carried the same core lesson plus its applicability, a plan-before-editing instruction, separate proof obligations for the rule and exception, a narrow-change boundary, and the required verification command.

The eight source lessons were split evenly between Claude Code and Codex. Every target task was then attempted by both agents, so the study included same-agent and cross-agent transfer.

Scoring and evaluator checks

A run counted as accepted only when the agent finished execution, changed only the permitted implementation file, and passed a private correctness check after the agent exited. A message claiming “done” or “tests pass” did not count as success.

Before agent execution, each evaluator was tested against four implementations: the failing start state, a correct reference, a plausible incomplete repair, and a deliberately over-broad repair. Agents could run visible unit tests, but could not access the private acceptance checks.

Task families were hash-partitioned before implementation into development and held-back sets. Every run began in a fresh workspace with isolated agent and Belay state. Runs were not repeated or replaced based on outcome, and product changes stopped before the final evaluation began. All 96 registered cells remain in the record; one was interrupted after provider start by local disk exhaustion.

Development versus final evidence

Before the final evaluation, seven separate development families produced 84 runs. On their related tasks, Belay met the acceptance criteria on 11 of 14, the human instruction on 10 of 14, and no additional guidance on 7 of 14. Those results were used to freeze the product and procedure, so they are reported for context but excluded from the final product evidence.

Agents and models

7. Results.

When an earlier lesson was useful

Task by task, Belay recorded six additional successes and one fewer success relative to no additional guidance, for a net difference of five tasks. Relative to the concise human instruction, the net difference was four tasks.

Paired related-task outcomes

Relative to no guidance, Claude Code recorded five Belay-only successes and no losses. Codex recorded one Belay-only success and one loss, for no net difference. The evaluation was not designed to rank the agents; the split shows that one guidance format does not affect every receiving agent equally.

Preserving lesson boundaries

These tasks required the correct alternative behavior; doing nothing was not automatically sufficient. One Codex Belay run in the calibration family was interrupted by local disk exhaustion. In one Claude Code run, queue-update guidance was applied to current-batch positive pairs, breaking the differentiability that the boundary required.

That failure provides a concrete design signal: reusable guidance needs explicit conditions, required behavior on both sides of the condition, and focused checks—not instruction text alone.

Download the machine-readable aggregate results and paired comparisons, or inspect the task-level result matrix.

8. What we learned.

9. Privacy, trust, and developer control.

Belay is designed for private engineering work:

• Session content is stored in an encrypted local database.
• Belay does not upload coding sessions to Doplex Labs.
• Belay requires no account and ships no general-purpose model.
• Optional interpretation uses the developer's configured Claude Code or Codex provider.
• Conversation history and model-generated proposals cannot become active guidance without developer approval.
• Approved lessons remain reviewable, replaceable, pausable, and removable.
• If Belay is unavailable, the coding agent continues working normally.

Read the complete trust and privacy boundary.

10. Interpretation and scope.

The first evaluation provides a concrete product signal: evidence-backed lessons from prior work can improve later acceptance outcomes under the tested conditions.

The results also show where deeper product work matters. The observed benefit was strongest with Claude Code, neutral overall with Codex, and one boundary case required more precise scoping.

The downloadable data above provides aggregate and task-level outcomes. A companion preprint will provide the complete experimental protocol, task-construction record, evaluator interfaces, study history, and additional reproducibility materials after privacy and security review.

11. Where Belay goes next.

Explicit lesson boundaries

Represent each lesson with conditions, required behavior, alternative behavior, and focused checks.

Agent-specific delivery

Learn which guidance format helps each supported coding agent and when additional context is unnecessary.

Real developer workflows

Measure repeated corrections, time to useful progress, review effort, and accepted work in consenting developer environments.

Experience integrity

Strengthen protection against outdated, conflicting, incorrect, or adversarial lessons while preserving traceability and developer control.

12. Conclusion.

Coding agents should not begin every session with no benefit from the work that came before.

Belay is building a private experience layer that connects observed work, approved lessons, and future agent guidance. Its first evaluation demonstrates the potential of experience transfer and identifies the next engineering requirements: precise boundaries, agent-specific delivery, and outcome-based measurement.

The long-term measure of Belay will not be how many memories it stores. It will be how often developers need to teach the same important lesson twice.

References.

1. Ouyang, S., Yan, J., Hsu, I-H., et al. “ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory.” ICLR 2026. Source
2. Yan, J. and Lee, C-Y. “ReasoningBank: Enabling Agents to Learn from Experience.” Google Research, 2026. Source
3. He, Z., Wang, Y., Zhi, C., et al. “MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks.” 2026. Source
4. Fang, R., Liang, Y., Wang, X., et al. “Memp: Exploring Agent Procedural Memory.” Findings of ACL 2026. Source
5. Zhou, Y., Guo, K., Zhuang, H., et al. “Getting Better at Working With You: Compiling User Corrections into Runtime Enforcement for Coding Agents.” 2026. Source
6. Lopopolo, R. “Harness Engineering: Leveraging Codex in an Agent-First World.” OpenAI. Source
7. Young, J. “Effective Harnesses for Long-Running Agents.” Anthropic, 2025. Source
8. Shadwell, T. and Spânu, A. “Designing AI Agents to Resist Prompt Injection.” OpenAI. Source