I looked at 558 AGENTS. md files: here's a 5-minute check for yours
A study of 558 public agent charter files shows most forbid actions and list build commands, but few note pitfalls. The author offers a quick checklist to audit your own file and compares coverage against the corpus baseline.
In a recent audit of 558 public AGENTS.md files, a developer mapped each file to a nine‑category taxonomy. The analysis revealed that 85.7% of the files contain sections that forbid certain actions, while 82.8% list build and test commands. Only 13.6% include a “gotcha” – a warning about a pitfall that can’t be deduced from the code itself. Two categories, gotchas and agent_meta (rules about the agent’s own behavior), are almost empty across the entire corpus, with only 25.8% of files addressing the latter.
What the Numbers Tell Us
The base rates were measured on 516 substantive files (the remaining 42 were one‑line pointers). The categories and their frequencies are:
- Boundaries – 85.7% (what must never be done)
- Build/Test – 82.8% (commands CI runs)
- Workflow – 67.1% (commit format, branches, release steps)
- Structure – 59.1% (layout, where new code belongs)
- Style – 54.5% (naming, formatting or a pointer to the config that enforces it)
- Environment – 45.0% (toolchain versions, required env vars)
- Overview – 32.2% (one paragraph: what this is, what it deliberately is not)
- Agent_meta – 25.8% (rules about the agent: tone, when to ask first)
- Gotchas – 13.6% (pitfalls that are not derivable from the code)
These figures match the genre of an AGENTS.md file: a defensive document that lists what not to break. The file that actually saves time is the one that is rarely written – the gotcha section.
Why Gotchas Are Scarce
Writing a gotcha requires a developer to have encountered the issue first. By the time the problem is understood, the fix is usually already in the code, making the warning redundant. Moreover, maintaining a gotcha is a liability; if the issue changes, the warning can become misleading. A second failure mode is the inclusion of generic advice that isn’t specific to the agent. In a sample of 347 entries from the Gotchas/Common Pitfalls section, 58% were readable from the repository itself, 34% were generic (“remember to install dependencies”), and only 8% were experience‑only pitfalls that can’t be derived from the code. That 8% is the most valuable content for an agent charter.
A 5‑Minute Checklist for Your File
No special tools are needed. Ask yourself these five questions:
- Are the commands copy‑pasteable? The file should state the exact command that CI runs, including the working directory.
- Does it name what must never be committed or touched? Beyond obvious rules like “don’t commit secrets,” look for boundaries that prevent accidental data loss.
- Does it describe the agent’s own behavior? Tone, when to ask for confirmation, and what must not leave the machine.
- Is there at least one sentence that can’t be derived from the code? This is the gotcha test.
- Do the referenced paths exist? About half the files point to another file, and 15% point to a knowledge store. A dangling reference can mislead an agent.
For a deeper dive, the author provides a command‑line tool that compares your file’s coverage against the 558‑file baseline and highlights gaps. The tool also checks for external pointers and dangling references.
What the Study Isn’t About
The analysis is not a quality assessment of the files themselves. Coverage is a process metric, not a quality metric. The classifier used to label the files achieved 92% precision and 70% recall on an English held‑out set, and 88% precision and 73% recall on a Chinese set. The weakest slots are still gotchas and agent_meta in both languages. The study also does not claim that filling all nine slots guarantees a good file; it merely provides a prompt for the necessary questions.
How to Contribute
If you have an AGENTS.md, CLAUDE.md, or cursor.rules file on a real project, run the comparison tool and let the author know if the tool mislabels your file or misses a slot. Feedback on the baseline rates is also welcome. The repository is available at https://github.com/janzong/agent-charters.
Why it matters
Understanding how agent charter files are written helps teams create clearer, safer documentation that reduces bugs and improves agent reliability.
Key points
- 558 public files were analyzed across nine categories
- 85.7% forbid actions, 82.8% list build commands
- Only 13.6% contain gotchas, the most valuable section
- A 5‑minute checklist can audit your own file quickly
- The study offers a command‑line tool for comparison
Frequently asked questions
What is an AGENTS.md file?
It is a charter that defines an agent’s behavior, boundaries, and environment, often used in AI projects.
Why are gotchas rare?
They require real-world experience to write and are hard to maintain once the issue is fixed in code.
