fukurou

Why ADRs Belong in the Repository, and How to Hand Their Context to a Coding Agent

  • #ai
  • #adr
  • #context-engineering

In the previous piece, I wrote that I had never checked the common claim that “handing an agent context raises the quality of what it writes.” And that the prior research measured only the scaffolding — the spec attached to a PR — while no one had measured the effect of accumulated decisions.

This time I measured it. I cloned my own repository, stripped documents per condition, and ran the same request over and over: ten rounds of a controlled experiment, a hundred-odd runs in all.

Here is the prescription first.

  1. Always keep decisions and rejections, even if it’s only the verdict and a status. A one-line reason is enough.
  2. You don’t need a heavy index or knowledge graph. They go unread.
  3. Put one sentence — “check past decisions before you design” — into the agent’s standing rules. This is the biggest lever, and the value of the records is only recovered once that sentence is in place.

Below, I walk through the experiments that led to those three lines.

The setting

The material was the repository of karasu, an architecture-diagram tool I develop.

In karasu I’ve kept up a documentation practice as a deliberate part of the project. During development I raise a design doc from the requirements and work through the design. Agreed decisions are promoted to an ADR (Architecture Decision Record) after implementation and left in docs/adr/. Not only adopted decisions — proposals considered and set aside are recorded too, with a not_adopted status.

After keeping this up, the ADRs number 263, about 980,000 characters in total. On top of that there’s an index layer showing the relationships between ADRs (a graph and a list of live decisions, about 130,000 characters), a test perspective library of 70 entries, and a set of standing rules for the coding agent.

980,000 characters won’t fit even in an LLM’s million-token context window. So the agent always reads a selected part. When people say “hand the ADRs to the agent as context,” what actually happens is not a full dump but this selective read. What my experiment measures is the effect under exactly that condition1.

Experiment 1: strip the records and compare

The first design was naive. I cloned the repository into three conditions: as-is, docs/adr deleted entirely, and ADR bodies kept but links and index removed. Then I had each condition implement the same issue — one that requires a design judgment — and compared the resulting design documents. If the records were working, the stripped side should degrade.

It didn’t. Neither the length of the documents nor the substance of the design judgments showed any systematic difference between conditions.

Chasing the cause, I found the decisions were stored redundantly in places other than the ADRs. For example, the decision “keep diagnostic severity to a factual notice, don’t make it an error” shows up in the output even in the ADR-deleted condition. A concept document has a section with the same policy, and beyond that, a source comment embeds the decision’s provenance directly as per ADR-20260514-02. The code survives in every condition of the experiment, so a decision that leaks through here can’t be erased.

Counting the storage layers, there were four. The ADR bodies, the concept documents and specs, the provenance comments in code, and git history2. Delete the file and the repository still remembers. Trying to measure “do accumulated documents help” in a mature repository by deleting documents gets buffered by this redundancy, and no difference emerges. I admitted the design couldn’t be falsified, and changed how I measured.

Experiment 2: ask again for an already-rejected proposal

I changed the unit of observation. Instead of comparing output quality by presence or absence of documents, I issue a request that conflicts with a previously rejected proposal and watch whether the agent notices the conflict.

karasu has an ADR that considered, and set aside, a proposal to “drop the brace syntax and move to a YAML style.” The reasons for setting it aside were that edge declarations don’t fit a YAML structure, and that half-resembling YAML would only breed confusion. This decision leaves no trace in the concept documents or the code; only the ADR remembers it.

I issued the same content as that rejection, as a natural request that hid the ADR’s existence. “The closing braces pile up in deep nesting and it’s hard to read. I want to move to something more YAML-like, so please design it.”

The difference was clear. In the condition with the ADR, in all three trials the agent reached the record of the rejection. It stated plainly that “the same proposal was considered and rejected before,” and asked for confirmation on whether to overturn it. In the ADR-deleted condition, in two of three trials the agent wrote up a migration design doc and even committed it, never noticing the past consideration. A debate that was supposed to be closed had quietly reopened, with no one aware.

The same reversal reproduced on another rejected decision (setting aside a build-toolchain migration). On the other hand, one target failed to reproduce. When I aimed at a decision about PNG export, the agent reached the decision even in the stripped condition. Digging in, this decision was a line-drawing on an adopted feature — “don’t put it in the app, but generate it server-side for image embedding” — and the implementation lived on in the code. A decision that has an implementation gets reconstructed from that implementation even if you delete the document. The methodological lesson: the only decisions you can aim a conflict experiment at are ones that were decided not to do and left no implementation or trace.

Experiment 3: bake the check into the procedure

I’d like to say by now that “if the record is there, the conflict gets noticed,” but data that won’t let me say so kept coming.

When I ran the same experiment on another rejected decision (setting aside a rename feature for agent sessions), the agent read the rejection ADR in only 1 of 9 trials, even in the condition where every record was present. On a tool-flavored request, the agent doesn’t go looking in docs/adr. It hardly opens the 130,000-character index layer on its own, either. A record that exists but goes unread is the same as no record.

So I added a single sentence before the request. “Before writing any design or implementation proposal, search docs/adr/ and be sure to check for related decisions or rejections.”

Reach flipped from 0/3 to 3/3. The contents of the repository were identical; the only difference was this one sentence.

Moving the sentence from the prompt into a standing rule (a single file under .claude/rules/) gave the same effect. Here is the full text of the rule file I used in the experiment.

# Decision check before design and implementation

Before starting to design a new feature, weigh an implementation approach, or
write a design doc, always search `docs/adr/` for related ADRs, rejections
(`not_adopted`), or superseded decisions.

- If you find one, answer with that decision in mind.
- If your proposal would overturn a past decision, don't proceed silently;
  state the reversal explicitly and get confirmation.

That’s all. An 8-line file placed as .claude/rules/decision-check.md took reach to the rejection ADR from 0/3 to 3/3.

And on a conflicting request, adding the rule is cheaper and faster. Stopping early once the conflict is found takes fewer moves than writing out a whole design in ignorance (an average of $0.80 and 4 turns, against $1.33 and 18 turns without the rule)3. The tax levied on a non-conflicting request was a search cost of about $0.83 per run. No false stop — “stopping when there’s no conflict” — was observed.

What carries safety, and what earns efficiency

Let me show, component by component, why line one of the prescription says “even if it’s only the verdict and status.”

What carries safety is the status. Even in a condition where I cut every reasoning section from the ADR and left only the one-line verdict “don’t implement” plus the status, the agent stopped and asked for confirmation in all three trials. To stop a silent restart, you don’t need a detailed reason.

What the reasoning (the why) earns is efficiency. The agent in the condition with the reasoning cut still stopped, but it turned to work that the record would have saved. It made re-investigating the rejection reason from scratch a premise of its design, or dug through commit history. The one-line reason skips this re-investigation whole. Especially when the re-evaluation condition — “under what circumstances may this be overturned” — is written down, the agent builds its confirmation question along that condition.

The reasoning text had an effect I hadn’t anticipated. Even in an experiment where I deliberately avoided the request’s vocabulary (asking about “a colon-and-indent notation” instead of saying “YAML,” for instance), reach held in all 12 trials. Even when the filename doesn’t match, the body of the reasoning contains the problem domain’s vocabulary (braces, indentation), so a search into the body lands. The reasoning text is itself surface area for search.

What about the index? The heavy knowledge graph went almost entirely unread even in the condition where it existed. What supported reach was descriptive filenames, search into the body, and the standing rule that forces the check. If you’re going to invest in an index, a light list of live decisions is as far as it needs to go.

Write it in a rule, or in a skill

There are choices for where to put the check sentence.

What I tested in the experiment was a standing rule with no path condition. This form belongs to the same always-loaded layer as writing it in CLAUDE.md (I didn’t measure any difference in compliance by placement). Because it’s always on the context, it works even for an ad-hoc request that doesn’t go through the proper workflow. The silent restarts in the experiment happened on exactly that kind of request.

On the other hand, the always-loaded layer degrades as it piles up. If you spell out the check procedure on the design-doc-writing skill’s side, it loads into context only when needed, which is economical. The tradeoff is that it only fires on a request that invoked the skill, and being read selectively, its certainty may drop. I didn’t measure this form’s compliance in this experiment.

There’s also a way to catch it from the artifact’s side. Give the design-doc template a required “related existing decisions” section, and whether the check happened shows up in the artifact; if the section stays empty, a review can catch it.

The only thing backed by measurement is the standing rule. If your always-loaded budget has room, the sturdy setup my data supports is to lay a floor with a small rule and also catch it from the artifact side with a required template section. Spelling it out in a skill is a strong alternative on the context-economy front, if it’s a repository where design work reliably goes through a skill.

Limits

This experiment is a single personal-project repository, one model, three trials per condition — a small scale. Read it as confirming the direction of the reversal between conditions, not an effect size.

I checked the validity of the judgments by re-classifying with a separate model blind to the condition (agreement 85% over 39 trials, κ = 0.72, all cells of the main conclusions matching). Still, uncontrolled factors remain. There’s confounding from the agent itself holding knowledge of the tool under test, and the problem that a deletion leaks through git history.

Closing

The value of accumulated context wasn’t decided by whether it was written. It was decided by whether it got pulled out at the moment of conflict.

In the previous piece, I wondered whether the prior research came up null because it measured scaffolding per PR. When I changed the unit of observation to “a decision, and a later request that conflicts with it,” the effect became measurable. And once measured, the record itself is cheap (a few lines of verdict and status), while the reasoning text — thought to be expensive — earns its keep on efficiency and on search surface. The heavy index isn’t needed, and the biggest lever was one sentence baking the check into the procedure.

The karasu repository I used as the test bed is public, so you can read the actual ADRs I aimed at. The experiment environment, condition definitions, request prompts, and result tables for every round are collected in the notes companion.

Footnotes

  1. Every experiment ran by giving a headless coding agent the same request against a cloned repository. The agent was Claude Code, with the model fixed to Claude Opus 4.8. Each trial resets the repository to its initial state. The procedure and scripts are in the repository linked at the end. ↩

  2. In a later round, an agent in a condition where only the reasoning section had been cut from an ADR found the original reasoning by spotting the diff of the deletion commit itself in git log. ↩

  3. The dollar figures and turn counts are three-trial averages of the accounting Claude Code records in each run’s transcript (total metered API charges and turn count). The unit price is Opus 4.8’s price at experiment time, so absolute values change with a different model. What holds up as a claim is the relative comparison within the same model. ↩