I believed context improved quality. I never checked.

I keep design decisions and verification notes in the repositories for karasu and hato. The reasoning behind each design decision goes into an ADR¹, the viewpoints for verification into a test perspective library², and the lessons from incidents into postmortems³. Coding conventions don’t go in CLAUDE.md; they sit in .claude/rules when I need them. ADRs and the test perspective library were written for people first. But since a coding agent can also read them, I assumed the quality of the code it produces would go up too.

The other day it hit me. I have never once checked that effect. I believe it worked, and if you asked me for evidence, I would have nothing to show. This piece is the record of the re-examination that started from that realization.

What “it got easier” was made of

I haven’t checked, but the feeling is real. Before and after I started stacking context⁴ in the repository, the experience changed.

The biggest change: I no longer re-explain past decisions every session. An agent rebuilds the context it needs for a good generation each session. If the repository doesn’t hold the right information, a human has to hand over that material in the prompt every time. That is what I used to do, and when I forgot, I didn’t get the feature I wanted. Now I barely need to repeat myself.

The second change: the agent stops when a change would overturn a decision. Ask for a change that conflicts with a past decision, and it checks first: “ADR-XXXX chose option A, but this change goes with option B. Should ADR-XXXX be marked superseded?” Superseded is one of the ADR statuses; it means the decision has given its seat to a later one. So the agent doesn’t only flag the conflict. It keeps the record of decisions consistent. This has saved me more than once.

I feel a similar effect from the test perspective library. With ADRs, the agent tries to keep the spec consistent. With the test perspective library, it tries to keep the spec from missing a viewpoint. Both behaviors happen because the record is in the repository.

So are these the same thing? The first is about my effort going down. The second is about accumulated decisions and verification viewpoints no longer slipping through in silence. They look alike, but they differ in kind. When I dug into the research, that distinction turned out to matter far more than I thought.

What the controlled studies on instruction files measured

Controlled studies on the effect of instruction files like AGENTS.md and CLAUDE.md started appearing in early 2026.

One compared Codex and Claude Code across 10 repositories and 124 pull requests⁵. With an instruction file, median runtime was 28.64% shorter and output tokens 16.58% fewer. Task completion, though, didn’t change. Faster and cheaper, not more correct.

The other, from ETH Zurich, lands harder⁶. Give the agent a context file generated by an LLM, and the task success rate drops (0.5 to 2 points, depending on the setup), while inference cost rises by more than 20%. Success rose only when the file was one a developer had committed to the repository, and even then by about 4 points. “Committed by a developer” doesn’t mean a human wrote it. A draft written by AI and approved by a human can sit in that bucket too. The dividing line may not be who wrote it, but whether it passed human approval.

So at least the first part of my “it got easier” feeling, the drop in repeating myself, is explained by this efficiency gain. Fewer seconds and fewer tokens, of course it feels easier. And gaining efficiency and gaining quality are, in measurement, separate stories. I may have mistaken an efficiency gain for a quality gain.

What guards which quality

So is the second feeling, the confirmations and the caught omissions, also an illusion? Let me fix the words first. What I want to talk about is less code quality than system quality. And system quality has a standard map: the quality characteristics model in ISO/IEC 25010⁷.

Let me restate the question. Not which characteristic matters. What guards each one. There are three guardians.

Static checkers (lint, types, CI): reject what can be judged mechanically.
Tests: confirm the code runs as the defined spec says.
Documents (ADRs, the test perspective library, postmortems): hold the judgment, verification viewpoints, and past failures a spec can’t fully carry.

Assign the quality characteristics to these three, and you can see which document works where.

Quality characteristic	Main guardian	Effective document
Maintainability	Static checkers	(leave to static analysis)
Functional correctness	Tests	Postmortems (prevent recurrence)
Functional appropriateness	Documents	ADR
Functional completeness	Documents	ADR, test perspective library
Security	Documents	ADR, postmortems

The bottom three, functional appropriateness and down, are hard for either static checks or tests to catch. That is where ADRs can be expected to help.

Maintainability I push to static checkers by default. Naming rules, of course, but structural agreements like module boundaries or how logical and physical layers split can often be enforced by static analysis, depending on the language and tools. Having a machine reject it beats writing “follow this” in prose. So I don’t hand maintainability to documents. Confirming functional correctness has long been called verification (are we building it right). Confirming functional appropriateness is validation (are we building the right thing). That split lines up with the boundary between tests and documents.

The static-checker territory isn’t where I want context to enforce things. In karasu and hato I used to point out coding-convention violations every session. Now I let static checkers reject them where I can. Only the agreements a checker can’t express go into .claude/rules, when needed. Writing “follow this” in prose for something a check could reject hands a deterministic check to probabilistic generation. Do statically what can be done statically. What context takes on starts after that.

What those controlled studies measured was functional correctness. Task success rate, whether the tests pass. And the instruction files didn’t improve it. That may be expected. Tests are the first guardian of correctness, and it isn’t a job an instruction file can take over.

Where I felt “helped” is the documents’ territory. Checking consistency against an ADR guards functional appropriateness. Catching a missed viewpoint against the test perspective library guards functional completeness. And ADRs also record what was deliberately not adopted. karasu has ADRs that decided, on purpose, how to handle runtime authorization and how to sanitize input. A change that passes the tests but silently reintroduces a risk sealed off there passes on functional correctness and counts as an incident on security.

There is a measurement that backs this split. A study pulled design constraints out of real pull-request review discussions and built a benchmark⁸. Among issues that passed the tests and counted as “resolved,” fewer than half satisfied every design constraint. Functional correctness and design compliance had almost no correlation.

Whether tests pass and whether the code follows accumulated decisions don’t show up unless you measure them apart. What the research calls “no effect” is the former; what I felt working is the latter. But no study has measured the latter yet. My feeling isn’t refuted, and it isn’t supported.

Does writing specs help?

If you want to move appropriateness or completeness, write the spec and the decisions. There is a measurement suggesting it isn’t that simple.

A study tested the effect of spec-driven development across 119 open-source repositories and about 100,000 pull requests⁹. Pull requests that came with a spec artifact had a slightly higher defect rate and 5 points more rework. The author reads the presence of a spec not as a sign of better quality, but as a proxy for the task being complex.

I think it is early to conclude from this that writing specs is pointless. What got measured is “did the PR have a spec attached,” not how the spec was used in later work. A spec gets used in two ways. As scaffolding that guides the current implementation and goes unread once it is done. And as something that accumulates in the repository and lives on as a constraint on later tasks.

The “confirm before overturning” that helped me can, by definition, only happen with the second kind. The decision is recorded, a later session reads it, and the conflict gets noticed. Scaffolding-style specs have nothing after the implementation ends. Even among spec-driven-development tools, the two are treated differently. Some enforce a structure that discards change proposals after implementation and accumulates only the settled spec in a separate directory; others officially allow freezing the per-feature spec directory¹⁰. Left alone, the second turns the spec folder into a pile of old PR descriptions.

So I don’t read that observational study as denying the effect of specs. Lump together specs left as scaffolding and specs accumulated into constraints, and of course the effect looks diluted. But no study has isolated and measured the accumulated side yet.

Where does the missing input come from

Before the measuring, let me put into words why it should work.

When the input to an agent is short, generation doesn’t stop. The model fills the gap. Filling itself isn’t the problem; that is how generation works. The problem is the material. If nothing is left in the repository, the material is only the general knowledge the model learned in advance. Output filled from there can drift from what my service specifically assumes. That, I think, is the real source of output I never asked for in the prompt.

With decision records in the repository, it changes. Even without my writing it in the prompt, the agent can read the records. The material for filling shifts from general knowledge to “judgments approved in this repository.” For the characteristics documents guard, functional appropriateness, completeness, and security, the chance of generating something that contradicts a past decision goes down. Context is a device that swaps the material for filling. That is how I understand it now.

The mechanism gets sharper at scale. karasu has about 250 ADRs now, roughly 980,000 characters in total. I don’t remember all of them myself. They won’t fit in an agent’s million-token context by full read either. The repository remembers more than I do. I can’t list the relevant decisions from memory. Even so, when I see the proposal in front of me, I notice the hole. So alongside recording decisions, I need an index to follow them.

The mechanism guarantees no correctness, though. Even when the code aligns with a past decision, the decision itself can be wrong. When the situation changes, the rejected option can become the right one. The final judgment of whether I am building the right thing is mine. Context working on system quality doesn’t mean guaranteeing correctness. It means the chance to judge correctness isn’t taken away from the human. That is how I frame it now.

How I could check

So far this has been feeling and mechanism. It should work, I can say. But I haven’t checked. So I designed an evaluation to check it. Three steps. Narrow to the characteristics likely to move, compare against a repository with the records stripped out, and measure each characteristic.

Narrow to the characteristics likely to move

Documents guard only part of system quality. On the quality characteristics model, I sort out the ones where ADRs are likely to help.

Functional appropriateness: does the implementation follow the intended requirement and decision? The center where ADRs should help most.
Functional completeness: does it miss a needed viewpoint? ADRs and the test perspective library help.
Security: does it reintroduce a risk that was deliberately sealed off? The ADRs recording rejections help.

These three are the characteristics I expect an effect on. Functional correctness I treat separately. The research reported that giving records doesn’t move correctness. So I measure correctness as a negative control, where no effect is expected. If appropriateness and security improve while correctness stays flat, I have reproduced the research’s result inside my own repository. Maintainability I leave to static checkers, so it is out of scope. Compatibility and portability have decisions behind them but are hard to score in this experiment, so I set them aside for now.

Compare against a repository with the records stripped out

With the characteristics chosen, I compare on whether the records are present. I clone karasu into three states.

Current: ADRs and the index left as they are.
Strip frontmatter: keep the ADR bodies, but remove the frontmatter so the links and statuses between decisions are lost.
Remove the records: delete docs/adr entirely.

I have the same issue implemented in each of the three states. The gap between current and removed is the effect of the records themselves. The gap between current and stripped frontmatter is the effect of the connections. If “writing helps” is wrong and “keeping it in a followable form helps” is right, the two gaps should appear separately.

I watch out for symmetry. I clone all three separately, turn off cross-session memory, and run each issue in a fresh session. Otherwise a decision I thought I had removed leaks in through another path.

Measure each characteristic

I score the output per characteristic.

Functional appropriateness: the number of places that contradict a relevant ADR decision. I back the judgment with the history of which ADRs the agent actually read.
Functional completeness: counted against the test perspective library, the number of missed viewpoints.
Security: whether a previously rejected risk was reintroduced.
Functional correctness (negative control): whether the tests pass.

One run each can’t separate generation variance from the difference between conditions. I run the same issue and condition several times and compare distributions. I score blind to which condition produced the output, and cross-check my own judgment against a machine’s.

The numbers won’t be exact. Judgment uses an agent, and the number of issues is small. Still, it moves me past “I believe it worked.”

Where this leaves me

One thing got settled in this piece. I believed context raised system quality, and I held no evidence for it. Many of the moments I felt it working may, when measured, turn out to be efficiency gains. The quality side I haven’t measured yet.

But I know where to measure now. Functional appropriateness, completeness, and security, rather than functional correctness. The territory documents guard, rather than the territory tests guard. I will measure it by comparing against a repository with the records stripped. When I have results, I will write again.

ADR (Architecture Decision Record). A document that records one design decision at a time, with its reason and background. ↩
A reusable collection of the viewpoints worth checking in tests. In karasu I call it a test perspective library (TPL). ↩
A document written after an incident: what happened, why, and how to prevent it next time. ↩
I use context for the information that sits in the repository and the agent can read: CLAUDE.md, .claude/rules, ADRs, the test perspective library, postmortems, and so on. ↩
arXiv:2601.20404. A January 2026 preprint; replication is still thin. ↩
arXiv:2602.11988. Measured over 138 instances (SWE-bench Lite and others) across 12 repositories. ↩
An international standard for software quality. It sorts product quality into characteristics like functional suitability, maintainability, security, and reliability, and keeps a separate model for quality in use. ↩
arXiv:2604.05955. 6 repositories, 495 issues, 1,787 verified constraints. Design guidance reduces violations, but plenty of non-compliance remains. ↩
Hill (2026), SSRN preprint. Pre-review, and observational, so it can’t claim causation. ↩
The first is OpenSpec, which archives changes/ after implementation and merges them into specs/. The second is Spec Kit, whose official guide accepts freezing the archive (Flow-Forward) as an option. ↩