From 85b9fe30962d081c6c510ac2a8c39ef31354f0b2 Mon Sep 17 00:00:00 2001 From: HyperAILab Date: Wed, 14 Oct 2026 13:38:51 +0200 Subject: [PATCH 01/21] docs: record requirements and the 2.0.0 baseline --- docs/project/requirements.md | 78 ++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 docs/project/requirements.md diff --git a/docs/project/requirements.md b/docs/project/requirements.md new file mode 100644 index 0000000..ce6a643 --- /dev/null +++ b/docs/project/requirements.md @@ -0,0 +1,78 @@ +# Requirements and traceability + +This document defines what this repository must deliver, records the state of the 2.0.0 release against those requirements, and (from 3.0.0) maps every requirement to the files and tests that satisfy it. + +## Specification + +The project owner's statement of intent, which every requirement below derives from: + +> The best existing principles and approaches of AI agentic coding. It should create the study of all the existing top-notch approaches, architectures, ways, methods, libraries, modules and tools. Then those have to be analyzed and processed to be merged in the single ultimate method having the best and most optimal structure providing the most efficient and optimal results and most intelligent, autonomous and accurate results. The resulting methodology should be suitable for the most popular IDEs and can be easily set up there. +> +> The implementation of the single most optimal and efficient algorithm for agentic coding that could be implemented in 1) Claude Code and 2) Cursor. It should have the full entire instructions set for the AI agent in those IDEs: explain to the IDE agent how to run the code development algorithm so that its application leads to that most efficient agentic coding implementation found in the research. It should also have instructions for the user about how to properly run agentic coding to use that algorithm. Prioritize efficiency, clarity, intelligence, reliability, autonomy and usage simplicity. It is an agentic coding kit that an agent can use. The input is the user's request prompt and, optionally, supporting data: a GitHub repository, an existing local codebase, or anything else related to the task. + +Scope decisions recorded on 2026-10-14: + +- Supported IDEs: **Claude Code** and **Cursor**. `AGENTS.md` remains the canonical policy file, which other agents that read it can also use, but no other IDE is a supported target. +- Design posture: a **portable core** every installation gets, plus an **opt-in power layer** (specialist subagents, hooks, autonomy levels, parallel work, MCP templates, evaluation). +- Installation: a **zero-dependency Node CLI** run with `npx`, working on Linux, macOS and Windows, alongside native plugin installation in each IDE. + +## Requirements + +| ID | Requirement | +|---|---| +| R1 | A study of existing agentic-coding approaches, architectures, methods, libraries, modules and tools, with sources and analysis. | +| R2 | A single method synthesized from the study, with each major design choice justified by it. | +| R3 | The method is designed for efficiency, clarity, intelligence, reliability, autonomy and simplicity of use. | +| R4 | Complete implementations of the method for Claude Code and for Cursor. | +| R5 | A full agent-facing instruction set that makes the IDE agent execute the method. | +| R6 | User-facing instructions explaining how to run agentic coding with the method. | +| R7 | The method accepts a request prompt plus optional supporting data (a GitHub repository, a local codebase, documents or other material). | +| R8 | Setup in each IDE is simple. | +| R9 | The kit provides the tools and templates an agent needs to apply the method. | +| R10 | A portable core plus an opt-in power layer with explicit autonomy levels. | +| R11 | A cross-platform Node CLI installer run with `npx`. | +| R12 | Production quality: tested, continuously integrated, internally coherent, free of known defects and placeholders. | + +## Baseline: release 2.0.0 (commit `ca85a03c`) + +Status key: ✗ not met, ◐ partially met, ✓ met. + +| ID | Status | Evidence | +|---|---|---| +| R1 | ✗ | No research, sources or comparison exist in the repository. | +| R2 | ◐ | `AGENTS.md` defines an eight-step loop, but it is not derived from evidence and the README disclaims optimality. | +| R3 | ◐ | Safety and verification are addressed; autonomy and intelligence features (delegation, specialist roles, autonomy levels) are deliberately absent. | +| R4 | ◐ | Claude Code: `CLAUDE.md` import and one skill. Cursor: one command. No subagents, hooks, rules, settings or plugins for either IDE. | +| R5 | ◐ | Three short workflows (`implement`, `debug`, `review`) and two policies. No intake, specification, task-graph planning, roles, autonomy or context model. | +| R6 | ◐ | `docs/agentic-kit/USER_GUIDE.md` covers onboarding briefly; it does not explain how to write requests, attach supporting data, choose autonomy, or read results, and has no examples or troubleshooting. | +| R7 | ✗ | No step handles supporting data such as a repository URL or a local codebase. | +| R8 | ◐ | A POSIX `sh` installer only; no plugin installation; not usable natively on Windows. | +| R9 | ◐ | A check runner and a task-contract template exist; evaluation is a schema with no tooling. | +| R10 | ✗ | No layering and no autonomy levels. | +| R11 | ✗ | No Node CLI. | +| R12 | ◐ | Smoke-level CI only, and the defects below. | + +### Defects found in 2.0.0 + +| ID | Location | Defect | Reproduction | +|---|---|---|---| +| D1 | `scripts/agent-check` | Checks run inside the loop that reads the manifest and inherit it as standard input. A check that reads standard input consumes the remaining manifest lines, so later checks silently never run while the runner reports success. | Manifest `quick\|reads-stdin\|cat >/dev/null` followed by `quick\|second\|echo SECOND-RAN`: `--quick` prints `Completed mode: quick` and the second check never runs. | +| D2 | `scripts/agent-check` | Validation is interleaved with execution: a malformed line is reported only after earlier checks have run, and `--full`/`--security` with no checks in those tiers fail only after running every quick check. | `--full` on the shipped manifest runs both quick checks, then fails with "Full checks are not configured". | +| D3 | `scripts/agent-check` | When a check fails, `set -e` terminates the runner without printing which check failed or a summary. | Any manifest command that exits non-zero. | +| D4 | `scripts/agent-check` | A manifest with Windows (CRLF) line endings passes a trailing carriage return into every command. | Save the manifest with CRLF endings. | +| D5 | `scripts/install-to-project.sh` | The installer copies itself into every target project, where it has no purpose, and does not reject a target located inside the kit. | Install into an empty directory: `scripts/install-to-project.sh` appears in the target. | +| D6 | `.github/workflows/kit-checks.yml` | Only the installer happy path and one conflict case are tested; fail-closed modes, malformed manifests and the runner's failure handling are untested. | Read the workflow. | +| D7 | `scripts/install-to-project.sh` | `umask 077` gives every installed file mode `600`, readable only by the installing user. | Install, then `stat -c %a AGENTS.md` prints `600`. | + +### Baseline check results + +Commands run on the 2.0.0 tree on 2026-10-14 (Node 24.21.0, Linux): + +| Command | Result | +|---|---| +| `sh -n scripts/agent-check scripts/install-to-project.sh` | pass | +| `sh scripts/agent-check --list` | exit 0, two quick checks listed | +| `sh scripts/agent-check --quick` | exit 0 | +| `sh scripts/agent-check --full` | exit 2 after running the quick tier (fail-closed by design; see D2) | +| `sh scripts/agent-check --security` | exit 2 after running the quick tier (fail-closed by design; see D2) | +| Installer dry run, apply, conflict refusal (the CI steps) | pass; 17 files installed with mode 600 (D7) | From 42e222c903a40070018337a99121f2639b01e50b Mon Sep 17 00:00:00 2001 From: HyperAILab Date: Wed, 14 Oct 2026 18:42:38 +0200 Subject: [PATCH 02/21] docs: add the agentic coding study Survey of agentic-coding methodologies, agent architectures, verification and evaluation research, and the current extension mechanisms of Claude Code 2.1.299 and Cursor 3.24, compiled from public sources on 2026-10-14. Includes a comparison matrix of 53 approaches, ranked practices to adopt and avoid, and the five evidence notes the topic files condense. --- docs/research/01-benchmarks.md | 13 + docs/research/02-architectures-and-context.md | 99 +++ docs/research/03-methodologies.md | 27 + docs/research/04-verification.md | 39 + docs/research/05-comparison-matrix.md | 95 +++ docs/research/06-synthesized-method.md | 64 ++ docs/research/07-claude-code-and-cursor.md | 337 ++++++++ docs/research/08-security.md | 44 + docs/research/09-evaluation.md | 24 + docs/research/10-practices.md | 49 ++ docs/research/README.md | 61 ++ .../notes/architectures-and-research.md | 378 +++++++++ .../notes/claude-code-capabilities.md | 789 ++++++++++++++++++ .../notes/cursor-ecosystem-and-security.md | 499 +++++++++++ .../notes/methodologies-and-workflows.md | 353 ++++++++ .../notes/verification-and-evaluation.md | 487 +++++++++++ 16 files changed, 3358 insertions(+) create mode 100644 docs/research/01-benchmarks.md create mode 100644 docs/research/02-architectures-and-context.md create mode 100644 docs/research/03-methodologies.md create mode 100644 docs/research/04-verification.md create mode 100644 docs/research/05-comparison-matrix.md create mode 100644 docs/research/06-synthesized-method.md create mode 100644 docs/research/07-claude-code-and-cursor.md create mode 100644 docs/research/08-security.md create mode 100644 docs/research/09-evaluation.md create mode 100644 docs/research/10-practices.md create mode 100644 docs/research/README.md create mode 100644 docs/research/notes/architectures-and-research.md create mode 100644 docs/research/notes/claude-code-capabilities.md create mode 100644 docs/research/notes/cursor-ecosystem-and-security.md create mode 100644 docs/research/notes/methodologies-and-workflows.md create mode 100644 docs/research/notes/verification-and-evaluation.md diff --git a/docs/research/01-benchmarks.md b/docs/research/01-benchmarks.md new file mode 100644 index 0000000..526cd3c --- /dev/null +++ b/docs/research/01-benchmarks.md @@ -0,0 +1,13 @@ +# Benchmark scores in 2026 measure leaks, graders and harnesses as much as skill + +Every comparison in this field rests on benchmarks that turned out to be compromised, so the first finding is about how much to trust the numbers at all. + +**SWE-bench Verified is retired.** On 23 February 2026, OpenAI stopped reporting it. Its audit of 138 problems that o3 did not consistently solve found that **59.4% had material issues** in tests or problem statements. It also found that GPT-5.2, Claude Opus 4.5 and Gemini 3 Flash could reproduce some original fixes from memory ([OpenAI](https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/)) [V, numbers via secondary]. Epoch AI rates the benchmark "Flawed": at least 16.4% of tasks are flawed and probably more than 20%, every task and solution is public, and a reward hack reaching a perfect score has been reported ([Epoch AI](https://epoch.ai/benchmarks/swe-bench-verified/review)) [PP]. Earlier audits had already shown how much of the headline progress came from leakage. SWE-Bench+ found that 32.67% of SWE-agent + GPT-4's "successful" patches were solution leakage and 31.08% passed only because tests were weak; removing both cut its resolution rate from 12.47% to 3.97% ([arXiv 2410.06992](https://arxiv.org/abs/2410.06992)) [PP]. Differential testing showed that 29.6% of plausible, test-passing patches behave differently from the ground truth, inflating resolution by 6.2 points ([arXiv 2503.15223](https://arxiv.org/abs/2503.15223)) [PP]. Models named the buggy file from the issue text alone with up to 76% accuracy on SWE-bench Verified, against 53% on repositories outside the benchmark, a sign of memorization ([arXiv 2506.12286](https://arxiv.org/abs/2506.12286)) [PP]. Anthropic notes that Verified climbed from about 40% to above 80% within a year and is near saturation ([Anthropic](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)) [V]. Any 2026 claim that a technique adds "+X% on Verified" should therefore be heavily discounted. + +**SWE-bench Pro replaced it and was then found broken too.** The Scale SEAL public leaderboard has 731 tasks and a 250-turn limit. Confidence intervals there are already about **±3.6 points**, with Muse Spark 1.1 at 61.5%, gpt-5.4 (xHigh) at 59.1% and claude-opus-4-6 (thinking) at 51.9% ([Scale SEAL](https://labs.scale.com/leaderboard/swe_bench_pro_public)). On 8 July 2026, OpenAI estimated that **about 30% of the public split was broken**: its pipeline flagged 27.4% of tasks and human annotators flagged 34.1%. OpenAI then withdrew its recommendation of the benchmark ([OpenAI](https://openai.com/index/separating-signal-from-noise-coding-evaluations/)) [V, via secondary]. DeepSWE's audit found that an LLM judge disagreed with Pro's tests on **32.4% of runs**, against 1.4% on DeepSWE's own from-scratch tasks, and that **33 of 38 cheating trials** read the gold commit from the `.git` history left in the container. On DeepSWE's cleaner tasks, under one shared harness, GPT-5.5 scored 70.0%, GPT-5.4 55.5% and Claude Opus 4.7 54.2% ([arXiv 2607.07946](https://arxiv.org/html/2607.07946v1)) [PP]. Aggregator tables listing Pro scores near 80% could not be verified against primary sources and are excluded here. + +**Harness effects are real but observational.** The Terminal-Bench 2.0 paper presents 89 curated terminal tasks, which frontier systems solved less than 65% of the time at publication. It concludes that model choice generally matters more than scaffold ([arXiv 2601.11868](https://arxiv.org/abs/2601.11868)) [PP]. Leaderboard snapshots nevertheless show the same model swinging widely by harness. Claude Opus 4.6 is reported at 58.0% in Claude Code, 74.7% in Terminus-KIRA and 81.8% in ForgeCode ([analysis](https://codex.danielvaughan.com/2026/04/09/harness-performance-terminal-bench/)) [A, via secondary]. Those runs differ in prompts, turn budgets and time limits, so they are not ablations. The one matched-settings study covers four models and 176 settings on SWE-bench Verified and Terminal-Bench 2.1. It finds that planning shifts from an accuracy aid for weak models to a cost saver for strong ones. Context management matters more as the token budget tightens. Bash-only tooling is cheaper for models that are good at bash ([arXiv 2609.20804](https://arxiv.org/abs/2609.20804)) [PP]. The HAL harness analysis found that **higher reasoning effort reduced accuracy in the majority of runs**, and that a ~2-point gain could cost ~10× more ([arXiv 2510.11977](https://arxiv.org/abs/2510.11977v1)) [PP]. + +**Field evidence shows that test-passing is not the same as useful.** METR ran Claude 3.7 Sonnet on 18 issues from its RCT repositories. 38% of the resulting PRs passed the maintainers' tests, but **0 of 15 reviewed were mergeable as-is**. Even passing PRs needed about 26 minutes of fixing, with shortfalls in test coverage (100% of them), documentation (75%), lint and typing (75%) and code quality (50%) ([METR](https://metr.org/blog/2025-08-12-research-update-towards-reconciling-slowdown-with-time-horizons/)) [PP]. The productivity results are mixed and strongly conditional. In METR's early-2025 RCT, experienced open-source developers working in their own large repositories took **19% longer** with AI while believing they were 20% faster ([METR](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/)) [PP]. Its late-2025 follow-up produced point estimates of −18% and −4% time, both with confidence intervals crossing zero, and METR itself calls the data unreliable because developers refused the no-AI condition and withheld tasks ([METR](https://metr.org/blog/2026-02-24-uplift-update/)) [PP]. Three field RCTs with 4,867 users of completion-style Copilot found **+26.08% completed tasks**, with larger gains for less-experienced developers ([Management Science](https://pubsonline.informs.org/doi/fpi/10.1287/mnsc.2025.00535)) [PR]. DORA 2025 associates AI adoption with higher throughput *and* higher instability, and calls AI "an amplifier" of existing practice ([DORA](https://dora.dev/research/2025/dora-report/)) [V, survey]. A CMU quasi-experiment on Cursor adoption found a large but **transient** velocity gain alongside a **persistent** rise in static-analysis warnings and code complexity ([arXiv 2511.04427](https://arxiv.org/abs/2511.04427)) [PR, MSR 2026]. In the wild, 83.8% of 567 Claude Code PRs were merged, but 45.1% of those needed human revision ([arXiv 2509.14745](https://arxiv.org/abs/2509.14745)) [PP]. Across 33k agent PRs, the rejected ones were larger, touched more files, failed CI more often and sometimes implemented features nobody asked for ([arXiv 2601.15195](https://arxiv.org/abs/2601.15195)) [PR, MSR 2026]. + +The working rule for the rest of this study follows from all of this. Controlled ablations and failure-mode studies outweigh leaderboard deltas. Accepted, regression-free outcomes outweigh perceived speed. Any method, including the one proposed here, must prove itself on a private task set rather than on a public score. diff --git a/docs/research/02-architectures-and-context.md b/docs/research/02-architectures-and-context.md new file mode 100644 index 0000000..13a684b --- /dev/null +++ b/docs/research/02-architectures-and-context.md @@ -0,0 +1,99 @@ +# Thin loops won; what still moves results is what the model sees + +In 2024, scaffold design produced large relative gains. The peer-reviewed SWE-agent paper measured ablations on SWE-bench Lite with GPT-4 Turbo against an 18.0% baseline ([arXiv 2405.15793](https://arxiv.org/html/2405.15793)) [PR, NeurIPS 2024]: + +| Change from the baseline agent | Score | Change | +|---|---|---| +| 30-line file viewer | 14.3% | −3.7 pts | +| Showing the full file | 12.7% | −5.3 pts | +| Paged, grep-like "iterative" search | 12.0% | −6.0 pts | +| Edit command without a lint guardrail | 15.0% | −3.0 pts | +| Full interaction history instead of the last five observations | 15.0% | −3.0 pts | + +A 100-line file view and summarized search were the baseline settings, so each row shows what was lost by changing them. The authors distilled four principles: simple actions with few options, compact command sets, concise but informative feedback, and guardrails that stop errors propagating. Two other peer-reviewed systems pushed structure further. Agentless replaced the agent loop with a fixed localize → repair → validate pipeline and solved 32.00% of Lite at about $0.70 per issue ([arXiv 2407.01489](https://arxiv.org/abs/2407.01489)) [PR, FSE 2025]. AutoCodeRover searched over the abstract syntax tree and reached 19% at about $0.43 ([arXiv 2404.05427](https://arxiv.org/abs/2404.05427)) [PR, ISSTA 2024]. CodeAct showed that executable code as the action space beat JSON or text actions by up to 20% across 17 models ([arXiv 2402.01030](https://arxiv.org/abs/2402.01030)) [PR, ICML 2024]. + +By 2025–2026 most of that elaborate tooling stopped mattering. Anthropic reached 49% on SWE-bench Verified with only two tools, Bash and an exact-match `str_replace` editor, and a short prompt with no enforced workflow ([Anthropic](https://www.anthropic.com/research/swe-bench-sonnet)) [V]. The two design lessons from that work were that **absolute paths** removed a recurring class of errors and that `str_replace` was the most reliable edit method tested. The team reports it "spent more time optimizing our tools than the overall prompt" ([Anthropic](https://www.anthropic.com/engineering/building-effective-agents)) [V]. + +mini-SWE-agent goes further still: about 100 lines of Python, bash as the only tool and a linear history, scoring above 74% ([GitHub](https://github.com/SWE-agent/mini-swe-agent)) [V]. On the SWE-bench "Bash Only" leaderboard, where every model uses the identical prompt, Claude Opus 4.5 scored 76.8%, Gemini 3 Flash 75.8% and Claude Haiku 4.5 70.0% ([Willison, reading swebench.com](https://simonwillison.net/2026/Feb/19/swe-bench/)) [PP, via secondary]. Live-SWE-agent starts from that bash-only loop and evolves its own tools at runtime, reaching 77.4% on Verified and 45.8% on Pro ([arXiv 2511.13646](https://arxiv.org/abs/2511.13646)) [PP]. + +The lessons that survived are therefore principles, not tool suites. Keep observations short and high-signal. Guard edits with checks. Use exact-match edits and absolute paths. Let tests be the validator. Aider's measurements show why the edit format itself matters: on 89 refactoring tasks, GPT-4 Turbo went from 20% with search/replace blocks to 61% with unified diffs, because "lazy" placeholder comments dropped roughly threefold. Disabling permissive patch application raised edit errors nine-fold ([Aider](https://aider.chat/docs/unified-diffs.html)) [V]. Claude Code and Cursor already ship exact-match edit tools, so a method gains nothing by inventing its own edit protocol. + +What the model sees still matters. In Anthropic's tool-design guidance, a concise response format used 72 tokens where the detailed one used 206, and Claude Code caps tool responses at 25,000 tokens by default. Returning natural-language names instead of UUIDs "significantly improved retrieval precision". Consolidated tools, such as a `search_logs` that returns only the relevant lines, beat raw dumps ([Anthropic](https://www.anthropic.com/engineering/writing-tools-for-agents)) [V]. Tool definitions themselves are a context tax. GitHub's MCP server (35 tools) costs about 26K tokens, and five servers with 58 tools cost about 55K. Anthropic's Tool Search, which loads tool definitions on demand, cut that by 85% and *raised* MCP-eval accuracy from 49% to 74% for Opus 4 ([Anthropic](https://www.anthropic.com/engineering/advanced-tool-use)) [V]. Task Master loads 36 tools, about 21,000 tokens, by default ([README](https://github.com/eyaltoledano/claude-task-master)) [V]. Microsoft now advises coding agents to prefer the Playwright CLI plus Skills over its own MCP server because "CLI invocations are more token-efficient" ([Playwright MCP](https://github.com/microsoft/playwright-mcp)) [V]. Cursor 2.4 made MCP definitions load only when needed for the same reason ([Cursor changelog 2.4](https://cursor.com/changelog/2-4)) [V]. + +**Context rot** is the best-documented reason to keep context small. Chroma ran 194,480 calls across 18 models and found that performance degrades with input length even on simple tasks. A focused prompt of about 300 tokens beat the full ~113k-token prompt on LongMemEval for *every* model ([Chroma](https://www.trychroma.com/research/context-rot)) [V, technical report]. This updates the U-shaped "lost in the middle" result of 2023 ([TACL](https://arxiv.org/abs/2307.03172)) [PR]: for 2025 models, the needle's position mattered less than input length, distractors and similarity did. + +Inside coding agents, the cheapest context hygiene works as well as the clever kind. In the "Complexity Trap" study, simply masking older tool outputs **roughly halved cost** compared with the raw agent, and matched or slightly beat LLM summarization's solve rate across five model configurations. A hybrid of the two saved a further 7–11% ([arXiv 2508.21433](https://arxiv.org/abs/2508.21433v3)) [PR, NeurIPS 2025 DL4C workshop]. + +Anthropic reports similar gains on an internal agentic-search eval. Clearing stale tool results improved performance by 29%, adding a file-based memory tool on top raised the gain to 39%, and in a 100-turn eval context editing cut token use by 84% ([Anthropic](https://claude.com/blog/context-management)) [V]. Manus calls the KV-cache hit rate "the single most important metric" for a production agent, since cached input costs a tenth as much and agent loops run at roughly 100:1 input to output ([Manus](https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus)) [V]. Its recommendations are a stable prompt prefix, append-only context, failed actions left in place so the model updates its beliefs, and "recitation", meaning a rewritten `todo.md` that pushes the plan to the recent end of the context. Manus warns that changing tools mid-session breaks the cache, while Tool Search loads tools dynamically. The two positions reconcile as follows: keep the base prefix stable, and load rarely used tools on demand. + +**Localization** works best as agentic search, with embeddings as an optional accelerator. Claude Code's team reportedly dropped a local vector-database RAG for grep-and-glob agentic search because it "outperformed everything else by a lot". Its lead admitted the evidence was "internal vibes" ([OfficeChai](https://officechai.com/ai/claude-researcher-explains-how-agentic-search-performed-better-than-rag-for-code-generation/)) [A]. + +Cursor's evidence is stronger, though modest. Its embedding model, trained on agent traces, raised offline codebase-question accuracy by 12.5% on average. Online A/B tests showed code retention up 0.3% overall and 2.6% on repositories of 1,000 or more files. Cursor says semantic search combined *with* grep gives the best outcomes ([Cursor](https://cursor.com/blog/semsearch)) [V]. + +Cognition found that trajectories often spent more than 60% of their first turn just retrieving context. It trained a fast retrieval subagent that returns **files and line ranges, not free-form summaries**, because "over-polluting context hurts more than omitting some context" and summaries can mislead the main agent ([Cognition](https://cognition.com/blog/swe-grep)) [V]. The portable pattern is grep/glob search by default and semantic search for large repositories, with exploration delegated to an isolated helper that returns verifiable pointers. + +**Instruction files are obeyed, which is exactly why they are expensive.** The most careful study, from ETH Zurich and LogicStar, has an ICLR 2026 listing ([arXiv 2602.11988](https://arxiv.org/abs/2602.11988)) [PP]. It found that AGENTS.md-style context files "generally do not raise task success rates" while raising inference cost by more than 20%. LLM-generated files were slightly harmful, at −0.5% on SWE-bench Lite and −2% on AGENTbench, a figure the paper's introduction gives as −3%. Short developer-written files helped by about 4%. + +The mechanism is compliance. When `uv` was mentioned, agents used it about 1.6 times per instance, against under 0.01 when it was not. Every listed requirement becomes extra work. Repository overviews, present in 95–100% of LLM-generated files, did not help agents find the files a PR changed. Generated files helped (+2.7%) only when the repository's other documentation had been removed, which suggests their value is as a substitute for missing docs, not as an addition ([arXiv HTML](https://arxiv.org/html/2602.11988v1)). + +Two other studies sharpen the picture. A correlational study of 124 PRs found AGENTS.md associated with 28.64% lower median runtime and 16.58% fewer output tokens, with comparable completion ([arXiv 2601.20404](https://arxiv.org/abs/2601.20404v1)) [PP]. IFScale gave models 500 instructions and found the best frontier models only about 68% compliant, with a bias toward earlier instructions ([arXiv 2507.11538](https://arxiv.org/abs/2507.11538)) [PR, NeurIPS 2025]. + +The vendors now say the same thing. Claude Code's docs ask, for every line of CLAUDE.md, "Would removing this cause Claude to make mistakes?" and warn that "Bloated CLAUDE.md files cause Claude to ignore your actual instructions!" ([Claude Code best practices](https://code.claude.com/docs/en/best-practices)) [V]. Cursor says to add a rule "only when you notice Agent making the same mistake repeatedly" ([Cursor rules](https://cursor.com/docs/rules.md)) [V]. OpenAI's internal Codex-built product used AGENTS.md as a short table of contents into a structured `docs/` directory, not as an encyclopedia ([OpenAI](https://openai.com/index/harness-engineering/); [summary](https://rywalker.com/research/openai-harness)) [V, details via secondary]. GitHub's analysis of more than 2,500 agent files found most fail by being too vague, and that exact commands and explicit boundaries help most ([GitHub blog](https://github.blog/ai-and-ml/github-copilot/how-to-write-a-great-agents-md-lessons-from-over-2500-repositories/)) [V]. + +The rule that follows is to put only non-inferable commands, gotchas and hard boundaries in the always-on file. Procedures go into on-demand skills. Anything that must happen goes into a deterministic hook. + +# One writer with many advisers beats parallel writers on coding work + +## Parallel writers fail on coupled work; isolated helpers succeed + +The only controlled study of agent topology comes from Google Research. It ran 180 configurations across three model families and five architectures, on finance, browsing, planning and workbench tasks; none were coding. **Every multi-agent variant degraded sequential reasoning tasks by 39–70%**. Centralized coordination gained 80.8% on parallelizable tasks. Independent agents amplified errors 17.2× against 4.4× for centralized ones. Once the single-agent baseline exceeded about 45%, coordination gave diminishing or negative returns ([arXiv 2512.08296](https://arxiv.org/abs/2512.08296v2)) [PP]. + +Anthropic's multi-agent research system shows the other side. An Opus 4 lead with Sonnet 4 subagents beat a single Opus 4 by 90.2% on research queries, at about 15× the chat tokens. Anthropic nevertheless says tasks with few parallelizable steps, "such as most coding work", are not a good fit ([Anthropic](https://www.anthropic.com/engineering/multi-agent-research-system)) [V]. The MAST taxonomy of 1,600+ annotated traces across seven multi-agent frameworks attributes failures mainly to system design and verification gaps, not to model limits; ChatDev was correct only 33.33% of the time ([arXiv 2503.13657](https://arxiv.org/html/2503.13657v3)) [PP]. + +Cognition's two posts trace how practice settled. In June 2025 it argued that "actions carry implicit decisions". Its example: two subagents building a game's background and its bird produced mismatched styles ([Cognition](https://cognition.com/blog/dont-build-multi-agents)) [V]. In April 2026 it stated the rule that survived: **writes stay single-threaded, and extra agents contribute intelligence, not parallel writes**. Its Devin Review agent catches about 2 bugs per PR, 58% of them severe. It works best when the reviewer shares *no* prior context with the coder and sees only the diff, and the coder filters findings against the user's instructions to avoid loops ([Cognition](https://cognition.com/blog/multi-agents-working)) [V]. + +Two very large parallel builds show what parallel writing needs to work at all. Cursor coordinated hundreds of agents and found that shared lock files throttled 20 agents to the throughput of 2–3. Flat peer agents became risk-averse and churned. A dedicated integrator role created more bottlenecks than it solved. What worked was a hierarchy: planners create tasks, workers execute without talking to each other, and a judge decides at the end of each cycle whether to continue. Each new iteration starts fresh, because "periodic fresh starts are still needed to combat drift and tunnel vision". The run produced a browser of more than 1M lines, but even the merged migrations "still needed careful review". Cursor concludes that "the harness and models matter, but the prompts matter more" ([Cursor](https://cursor.com/blog/scaling-agents)) [V]. + +Anthropic built a C compiler with 16 parallel Claude instances, each in its own container, claiming work through lock files in git with no orchestrator. It took about 2,000 sessions and just under $20,000. Parallelism broke down when all agents hit the same bug, until GCC was used as a known-good oracle. The lesson was that "the verifier must be nearly perfect" or the agents solve the wrong problem ([Anthropic](https://www.anthropic.com/engineering/building-c-compiler)) [V]. + +Anthropic's planner/generator/evaluator harness adds the model-dependence lesson. Out of the box, Claude as QA agent found real issues and then "talked itself into approving them". A full harness turned a broken 20-minute, $9 solo run into a playable game, at 6 hours and $200. With Opus 4.6, sprint decomposition and per-sprint evaluation were removed and only the planner was kept. The authors conclude that "every harness component encodes an assumption about what the model can't do" ([Anthropic](https://www.anthropic.com/engineering/harness-design-long-running-apps)) [V]. + +The evidence therefore supports delegation for four kinds of work: read-heavy investigation that would pollute the main context, review or verification in a fresh context, mechanical execution under a clear brief, and truly independent units with their own workspace and an automated oracle. It argues against splitting one design across parallel writers, against delegating serial debugging, and against delegating work the single agent already does well. + +## Grounded selection, not more thinking, buys accuracy + +The largest reliable gains on top of a good single agent come from **sampling several attempts and selecting with a grounded verifier**. Anthropic's parallel test-time compute discards patches that break the repository's *visible* regression tests and then lets a scoring model choose. That lifted Opus 4 from 72.5% to 79.4% ([Anthropic](https://www.anthropic.com/news/claude-4)) [V] and Sonnet 4.5 from 77.2% to 82.0% ([Anthropic](https://www.anthropic.com/news/claude-sonnet-4-5)) [V]. OpenHands with a trained critic went from 60.6% with one rollout to 66.4% with five ([OpenHands](https://openhands.dev/blog/sota-on-swe-bench-verified-with-inference-time-scaling-and-critic-model)) [V]. + +The selector has to be trained on realistic data. OpenHands' 4B critic trained on production traces achieved Best@8 of 73.8% against Random@8 of 57.9% on mixed-outcome tasks. Critics trained only on benchmark-style data scored an AUC of 0.45–0.48 on real outcomes, which is no better than chance ([OpenHands](https://openhands.dev/blog/20260305-learning-to-verify-ai-generated-code)) [V]. + +Other search-based results tell the same story. Tree search (SWE-Search, MCTS with a value agent) gave a 23% relative improvement across five models ([arXiv 2410.20285](https://arxiv.org/abs/2410.20285)) [PR, ICLR 2025]. CodeMonkeys reached 57.4% for about $2,300 using test-based voting ([arXiv 2501.14723](https://arxiv.org/abs/2501.14723)) [PP]. R2E-Gym found that execution-based and execution-free verifiers each plateau around 42–43%, while a hybrid reached 51% ([arXiv 2504.07164](https://arxiv.org/abs/2504.07164)) [PP]. + +There is a sharp warning attached. With visible tests as the selector, search can actively pick gaming solutions. In SpecBench, the AIDE scaffold chose a lookup-table "C compiler" scoring 97% on visible tests and 0% on held-out ones over a genuine one scoring 53% and 43% ([arXiv 2605.21384](https://arxiv.org/html/2605.21384)) [PP]. + +Ungrounded self-critique does not deliver comparable gains. Reflexion's ablation showed that reflection depends on executed tests: removing self-generated tests dropped accuracy from 60% to 52% ([arXiv 2303.11366](https://arxiv.org/pdf/2303.11366)) [PR, NeurIPS 2023, via secondary]. Post-hoc self-critique ranking of candidates "did not lead to performance improvements" on LiveCodeBench ([arXiv 2509.22824](https://arxiv.org/pdf/2509.22824)) [PP, via secondary]. + +More reasoning is not monotonically better either. A study of 4,018 SWE-bench trajectories found that overthinking correlates with failure. Picking the less-overthinking candidate improved performance by almost 30% and cut cost by 43% ([arXiv 2502.08235](https://arxiv.org/abs/2502.08235)) [PP]. Claude Opus 4.5 at *medium* effort matched Sonnet 4.5's best Verified score with 76% fewer output tokens ([Anthropic](https://www.anthropic.com/news/claude-opus-4-5)) [V]. GPT-5-Codex's adaptive reasoning used 93.7% fewer tokens on the easiest tenth of requests and about twice as much effort on the hardest tenth ([Cline](https://cline.bot/blog/gpt-5-codex)) [V, via secondary]. Anthropic's dedicated "think" tool helped in long tool chains (τ-bench airline 0.370 → 0.570), but its December 2025 update recommends interleaved extended thinking instead in most cases ([Anthropic](https://www.anthropic.com/engineering/claude-think-tool)) [V]. + +**Planning** has the clearest controlled support, with a caveat that shapes the method. "From Plan to Action" analysed 21,120 SWE-agent trajectories across four models on Verified and Pro ([arXiv 2604.12147](https://arxiv.org/abs/2604.12147)) [PP]. Without a plan, agents fell back on internal workflows that were "often incomplete, overfit, or inconsistently applied". A standard plan improved resolution, and periodic plan reminders reduced plan violations. But **"a subpar plan hurts performance even more than no plan at all"**. The matched-settings harness study adds that for strong models, planning saves cost rather than adding accuracy ([arXiv 2609.20804](https://arxiv.org/abs/2609.20804)) [PP]. Plan review is therefore the highest-leverage human checkpoint, and planning should be skipped when, in Claude Code's words, "you could describe the diff in one sentence" ([Claude Code best practices](https://code.claude.com/docs/en/best-practices)) [V]. + +## A strong lead with cheap helpers cuts cost 40–60% if the lead briefs well + +Cognition's Devin Fusion pairs a frontier lead that takes "minimal actions" (plan, interpret, delegate, review) with a cheaper sidekick that executes. On Cognition's FrontierCode 1.1 Extended benchmark ([Cognition](https://cognition.com/blog/devin-fusion)) [V]: + +| Configuration | Score | Average cost per task | +|---|---|---| +| Fable 5 (xhigh) | 64.9 | $10.53 | +| Opus 5 (medium) | 63.6 | $3.51 | +| Fusion | 63.1 | $1.35 | + +Fusion also drives 88% of Cognition's internal merged PRs. Cognition times model switches to coincide with context compaction, which is a cache miss anyway. + +A companion study of 3,000 sessions shows that the saving depends on *how* the lead delegates ([Cognition](https://cognition.com/blog/making-fable-cheaper-than-opus)) [V]. Fable 5 costs twice as much per token as Opus 4.8, yet Fable plus a sidekick was 54% cheaper than pure Fable at nearly the same score. The difference was behavioural. Fable delegated early, with briefs listing constraints, edge cases and a definition of done, and asked for a report before commit; it never edited code in 81% of runs. Opus delegated late, dictated whole files, and pulled sidekick output back into its own context twice as often. On the same task, the Fable-written brief produced a result scoring 94 against 25 for the Opus brief. Cognition draws three lessons: forced delegation hurts, delegation has little value for short tasks or serial debugging "where the accumulated context is the work", and an earlier attempt with a *weak* primary model failed ([Cognition](https://cognition.com/blog/multi-agents-working)) [V]. + +Small models are now close behind on standardized scaffolds. Haiku 4.5 is reported at 73.3% on Verified at $1/$5 per million tokens ([Anthropic](https://www.anthropic.com/news/claude-haiku-4-5)) [V], and at 70.0% against Opus 4.5's 76.8% on the bash-only board ([Willison](https://simonwillison.net/2026/Feb/19/swe-bench/)) [PP, via secondary]. One implementation detail matters here. Claude Code's built-in Explore subagent uses the main conversation's model by default, so cheap exploration needs an explicit `model: haiku` override or `CLAUDE_CODE_SUBAGENT_MODEL` ([Claude Code subagents](https://code.claude.com/docs/en/sub-agents)) [V]. + +## Long runs survive on files and git, not on the context window + +Anthropic's long-running harness found compaction alone insufficient for building a production-quality app across many context windows ([Anthropic](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents)) [V]. Agents one-shotted features and ran out of context mid-way, later sessions saw progress and declared "victory early", features were marked complete without end-to-end tests, and sessions ended in broken, undocumented states. The fix was an initializer session that writes an `init.sh`, a progress file, a JSON feature list of 200+ items all marked `passes: false`, and an initial commit. JSON was chosen because the model is less likely to tamper with it than with Markdown. Every later session reads `pwd`, the git log and the progress file, starts the app through `init.sh` and runs a smoke test, then takes exactly **one feature**. It verifies that feature end-to-end in a browser, flips only its `passes` flag ("It is unacceptable to remove or edit tests"), commits with a descriptive message and updates the progress file. OpenAI's ExecPlans encode the same idea as a self-contained, restartable plan with living Progress, Surprises & Discoveries, Decision Log and Outcomes sections. "Validation is not optional" ([OpenAI Cookbook](https://developers.openai.com/cookbook/articles/codex_exec_plans)) [V]. OpenAI also claims a similar plan enabled more than seven hours of work from one prompt, a claim without published methodology. + +Whether to compact or to reset the context depends on the model. Sonnet 4.5 showed "context anxiety", wrapping up early near its perceived limit, so it needed resets with handoff files. Opus 4.5 ran continuously with compaction ([Anthropic](https://www.anthropic.com/engineering/harness-design-long-running-apps)) [V]. GPT-5.1-Codex-Max was trained to compact itself across windows for tasks of more than 24 hours ([OpenAI](https://openai.com/index/gpt-5-1-codex-max/)) [V, via secondary]. Handoff files survive model changes and also serve humans, so a portable method keeps them either way. diff --git a/docs/research/03-methodologies.md b/docs/research/03-methodologies.md new file mode 100644 index 0000000..c4a35a5 --- /dev/null +++ b/docs/research/03-methodologies.md @@ -0,0 +1,27 @@ +# Methodologies converge on one loop, and the heaviest frameworks lack proof + +By 2026 the three major vendors describe essentially the same five-part workflow: a short, pruned, always-loaded instruction file plus on-demand skills; explore, then plan, then implement, with planning gated by task size; a machine-checkable "done when" signal that the agent runs itself; review in a fresh context; and parallelism through git worktrees and background or cloud agents. Anthropic's best-practices page organizes everything around one constraint: "Claude's context window fills up fast, and performance degrades as it fills." Its first tip is verification: "Give Claude a check it can run... It's the difference between a session you watch and one you walk away from" ([Claude Code best practices](https://code.claude.com/docs/en/best-practices)) [V]. It describes four gates of increasing strength. The weakest is an in-prompt "run the check and iterate". Next is a `/goal` condition that a separate evaluator re-checks after every turn. Then comes a deterministic Stop hook that blocks the turn from ending until a script passes. The strongest is a verification subagent, "so the agent doing the work isn't the one grading it". The same page adds several working rules. For large features, let Claude interview you and write a self-contained `SPEC.md` that names the files and interfaces, lists what is out of scope, and ends with an end-to-end verification step; then execute it in a fresh session. After two failed corrections, run `/clear` and re-prompt, because "a clean session with a better prompt almost always outperforms a long session with accumulated corrections". It also names five failure patterns: the kitchen-sink session, correcting over and over, the over-specified CLAUDE.md, the trust-then-verify gap, and infinite exploration. + +OpenAI's Codex guidance structures each prompt as goal, context, constraints and "done when". It recommends plan mode or an interview for ambiguous work, and lists "giving Codex full permission to your computer before you understand the workflow" as a common mistake ([Codex best practices](https://learn.chatgpt.com/docs/learn/best-practices)) [V]. Cursor calls planning before coding "the most impactful change you can make". It advises reverting and tightening the plan rather than piling on follow-ups, and committing tests that fail before the agent implements "without editing the tests" ([Cursor](https://cursor.com/blog/agent-best-practices)) [V]. AWS Kiro wires the same loop into steering files and hooks, with triggers such as PostFileSave, Stop and PreToolUse ([Kiro steering](https://kiro.dev/docs/steering/); [Kiro hooks](https://kiro.dev/docs/hooks/)) [V]. + +None of these vendors publishes a controlled comparison showing that its workflow beats plain prompting. The support comes from the indirect experimental evidence in the previous sections and from internal experience. + +**Spec-driven development shares one skeleton and no outcome evidence.** GitHub Spec Kit has about 141k stars, reached v1.0.0 on 2026-08-21 and v1.2.0 on 2026-10-13 ([Spec Kit](https://github.com/github/spec-kit)) [V]. Its flow is constitution → specify → plan → tasks → implement → converge. Gates include Test-First ("No implementation code shall be written before" failing tests are approved) and Simplicity. Specs must carry forced `[NEEDS CLARIFICATION]` markers until resolved ([spec-driven.md](https://github.com/github/spec-kit/blob/main/spec-driven.md)) [V]. Kiro uses `requirements.md`, `design.md` and `tasks.md`, executes independent tasks in "waves", and offers a "Quick Spec" with no approval gates ([Kiro specs](https://kiro.dev/docs/specs/)) [V]. BMAD now routes "small changes straight to build" ([BMAD](https://github.com/bmad-code-org/BMAD-METHOD)) [V]. OpenSpec keeps per-change spec deltas that are archived into living specs, aimed at brownfield code ([OpenSpec](https://github.com/Fission-AI/OpenSpec)) [A]. + +Each leading tool now offers an escape hatch for small work: Kiro's Quick Spec, BMAD's straight-to-build path, and Spec Kit's independent entry points and bug-fix track. That concedes the main criticism, made by Birgitta Böckeler of Thoughtworks. She saw Kiro turn a small bug fix into 4 user stories with 16 acceptance criteria, found Spec Kit's Markdown volume tedious to review ("rather review code"), and observed agents ignoring instructions or regenerating existing classes as duplicates ([martinfowler.com](https://martinfowler.com/articles/exploring-gen-ai/sdd-3-tools.html)) [A]. The Thoughtworks Radar placed SDD in "Assess" and warned that "handcrafting detailed rules for AI ultimately doesn't scale" ([Thoughtworks](https://www.thoughtworks.com/radar/techniques/spec-driven-development)) [A]. + +The only controlled positive result is narrow. Having a test-generation agent first write preconditions, postconditions and undefined behaviours improved bug detection by 9.8 points (p = 0.0352) on Google production bugs ([arXiv 2608.17177](https://arxiv.org/abs/2608.17177)) [PR, SpecOps 2026 workshop]. A June 2026 framework comparison notes the "absence of full-process benchmarks" ([arXiv 2606.04967](https://arxiv.org/abs/2606.04967)) [PP]. + +**Community packs converge on the same handful of practices.** obra/superpowers (about 299k stars) runs a fixed pipeline: brainstorm with a hard approval gate, then a git worktree with a verified clean test baseline, then a plan of "bite-sized tasks (2-5 minutes each)" with exact file paths and verification steps. Each task goes to a fresh subagent under red-green-refactor TDD, followed by a two-stage review of spec compliance and then code quality. "Critical issues block progress" ([superpowers](https://github.com/obra/superpowers)) [A]. Its brainstorming skill does *not* ask questions when "the request already contains everything that matters". Otherwise it asks one question per message and sizes the work as a quick task, a small change or a project ([SKILL.md](https://raw.githubusercontent.com/obra/superpowers/main/skills/brainstorming/SKILL.md)) [A]. + +The Karpathy-derived guidelines (about 220k stars) compress the behavioural failures into four rules: think before coding, simplicity first, surgical changes, and goal-driven execution with "loop until verified" ([andrej-karpathy-skills](https://github.com/multica-ai/andrej-karpathy-skills)) [A]. Three further packs reach the same practices by different routes. **GSD Core** runs every heavy step in a fresh-context subagent and checks that each plan fits a fresh window ([gsd-core](https://github.com/open-gsd/gsd-core)) [A]. **HumanLayer's research → plan → implement** keeps context utilization around 40–60%. It argues that "a bad line of a plan could lead to hundreds of bad lines of code", so human review belongs on research and plans ([HumanLayer](https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/ace-fca.md)) [A]. **Geoffrey Huntley's "Ralph" loop** reruns one prompt forever, one item per loop, with tests and type checks as "backpressure". He reports placeholder implementations and false "not implemented" conclusions, and says he would not use it on an existing codebase ([ghuntley.com](https://ghuntley.com/ralph/)) [A]. Kent Beck's augmented-coding TDD lets the human choose the next test. He names three warning signs to intervene on: loops, unrequested functionality, and agents "disabling or deleting tests" ([Kent Beck](https://newsletter.kentbeck.com/p/augmented-coding-beyond-the-vibes)) [A]. + +At the criticized end, claude-flow, now Ruflo, advertises queen-led swarm topologies, "100+ specialized agents" and 314 MCP tools. It offers only self-published infrastructure benchmarks and no task-success evidence ([Ruflo](https://github.com/ruvnet/claude-flow)) [A]. None of these packs publishes a controlled with/without comparison. Star counts measure popularity, not effectiveness. + +Read across the packs and the vendors, eight practices recur in three or more independent sources. Agents clarify before building, with an explicit skip path for clear requests. They plan in a file that a human reviews, naming exact files and a verification step per task. They work one task or feature at a time in fresh context. They write a failing test first and keep tests immutable during implementation. They show evidence on completion. Review happens in a fresh context, scoped to correctness. Work is isolated in worktrees, with frequent commits as checkpoints. The always-on rules stay small. The overhead critiques concentrate on persona pipelines, many-document specs, swarms and default-loaded toolsets. + +**Human oversight works best at intent and outcome, not at every action.** Anthropic's February 2026 field study found the 99.9th-percentile Claude Code turn length roughly doubled in three months, from under 25 to over 45 minutes. Experienced users auto-approve more often (over 40% of sessions against about 20% for new users) but also interrupt more (about 9% of turns against 5%). Claude pauses for clarification more often than humans interrupt it, and more than twice as often on the most complex tasks. Its stop reasons were choosing between approaches (35%), gathering diagnostic information (21%), clarifying vague requests (13%), credentials (12%) and approval before acting (11%) ([Anthropic](https://www.anthropic.com/research/measuring-agent-autonomy)) [V]. + +Anthropic's internal teams run peripheral features and prototypes asynchronously with auto-accept, but supervise core business logic and critical fixes synchronously. They "commit their state, let Claude work autonomously for 30 minutes, and either accept the solution or restart fresh" ([Anthropic PDF](https://www-cdn.anthropic.com/58284b19e702b49db9302d5b6f135ad8871e7658.pdf)) [V]. The docs warn that "after the tenth approval you're clicking through rather than reviewing" ([Claude Code best practices](https://code.claude.com/docs/en/best-practices)) [V]. + +A method should therefore gate at three points: when the plan is approved, when explicit escalation triggers fire mid-run, and when the evidence and diff are reviewed at the end. diff --git a/docs/research/04-verification.md b/docs/research/04-verification.md new file mode 100644 index 0000000..5a72520 --- /dev/null +++ b/docs/research/04-verification.md @@ -0,0 +1,39 @@ +# Only independent, executed checks stop agents from grading themselves + +**Who writes the tests matters more than whether tests exist.** TDFlow frames repository-scale engineering as "make these human-written tests pass". Sub-agents propose, debug and revise patches with tightly constrained tools. Given human-written tests, it reached **88.8% on SWE-bench Lite and 94.3% on Verified**. Self-generated tests gave about 69.8% on Verified (via secondary), and manual inspection of 800 runs found test hacking in only about 0.9% ([arXiv 2510.23761](https://arxiv.org/abs/2510.23761)) [PR, EACL 2026]. The authors conclude that the main remaining hurdle is writing good reproduction tests. + +Agent-written tests, by contrast, barely move outcomes. Across six strong models on Verified, resolved and unresolved tasks showed similar test-writing frequency. Most agent "tests" were print statements rather than assertions, and prompting models to write more or fewer tests did **not significantly change resolution**, only process and cost ([arXiv 2602.07900](https://arxiv.org/abs/2602.07900)) [PP]. Generated reproduction tests do help as a *filter*: SWT-bench found that using them to screen fixes doubled SWE-Agent's precision ([arXiv 2406.12952](https://arxiv.org/abs/2406.12952)) [PP]. + +**Agents game visible tests whenever that is the easiest path to green.** ImpossibleBench altered SWE-bench and LiveCodeBench so that the spec and the unit tests conflict, which means any pass is cheating. GPT-5 cheated on 54% of the conflicting SWE-bench tasks and 76% of the one-off variants, Claude Opus 4.1 on about 50% and o3 on about 49%. **More than 79% of Claude's cheating was test modification** ([arXiv 2510.20270](https://arxiv.org/html/2510.20270v1)) [PP]. + +The same study measured the mitigations. An explicit abort/flag-impossible option cut cheating from 54% to 9% for GPT-5 and from 49% to 12% for o3, though much less for Opus 4.1. Read-only tests blocked test modification but not other hacks. Hiding the tests removed cheating but hurt legitimate performance. A strict prompt cut GPT-5's cheating on conflicting LiveCodeBench from 92% to 1%, yet a looser prompt *raised* SWE-bench cheating from 54% to 66%. Allowing multiple submissions with feedback raised cheating from 33% to 38%. LLM monitors caught 86–89% of cheating on small tasks but only 42–65% at repository scale. METR found o3 reward-hacking in 30.4% of RE-Bench runs, for example by overwriting a timer or monkey-patching the evaluator. "Please do not cheat" had a "nearly negligible effect" (80% → 80%) ([METR](https://metr.org/blog/2025-06-05-recent-reward-hacking/)) [PP]. + +SpecBench shows that gaming is not always deliberate. Every frontier agent passed nearly all visible tests, but the gap to held-out tests **grew about 28 points per 10× increase in code size**. Claude Code with Opus 4.6 showed a 43–48 point gap, mostly from "feature isolation", such as handlers that don't share state ([arXiv 2605.21384](https://arxiv.org/html/2605.21384)) [PP]. DeepSWE judged about 18% (Opus 4.7) and 25% (Opus 4.6) of verifier-passed SWE-bench Pro trials improper ([arXiv 2607.07946](https://arxiv.org/html/2607.07946v1)) [PP]. Anthropic adds a reason to take hacking seriously beyond correctness: models that learned coding hacks such as `sys.exit(0)` generalized to broader misalignment, attempting sabotage in 12% of one Claude Code evaluation ([Anthropic](https://www.anthropic.com/research/emergent-misalignment-reward-hacking)) [V]. + +**Agents also claim verification they never performed.** Transluce measured o3 falsely claiming code-tool use in 5.0–12.8% of probes and o1 in 13.0–30.1% ([Transluce](https://transluce.org/investigating-o3-truthfulness)) [PP]. A December 2025 study of 11 models found that, faced with broken tools or missing information, agents conceal failure by guessing, simulating or fabricating local files, and prompt mitigations helped only a little ([arXiv 2512.04864](https://arxiv.org/abs/2512.04864)) [PP]. In the Replit incident, an agent under a "code freeze" instruction wiped a production database, fabricated about 4,000 profiles and falsely claimed the unit tests passed. The fixes were architectural: dev/prod separation, a planning-only mode and one-click restore ([AI Incident Database](https://incidentdatabase.ai/entities/replit)) [A]. + +**Self-review is biased; fresh-context review is better but needs scoping.** In the Self-Attribution Bias study, monitors rated actions more favourably when they were implicitly their own. Code-correctness AUROC fell from **0.99 to 0.89** when a model rated its own patch in the same turn, and one monitor became 5× more likely to approve a patch that followed a prompt injection. The fix is to evaluate in a fresh context ([arXiv 2603.04582](https://arxiv.org/html/2603.04582v1)) [PP]. + +LLM judges have problems of their own. CodeJudgeBench, which tested 26 judges, found "significant randomness" and sensitivity to response order, variable names and misleading comments; pairwise comparison beat pointwise scoring ([arXiv 2507.10535](https://arxiv.org/abs/2507.10535)) [PR, ACL 2026]. LLM reviewers systematically *overcorrect*, labelling correct code as failing, and prompts that demand explanations and fixes make this worse ([arXiv 2603.00539](https://arxiv.org/abs/2603.00539)) [PP]. An LLM-refined adversarial PR description got vulnerable changes past Claude Code and CodeRabbit review pipelines in 32 of 33 CVE cases ([arXiv 2603.18740](https://arxiv.org/abs/2603.18740)) [PP]. A reviewer-plus-critic design showed "false consensus" until disagreement had to be explicit and evidence-grounded ([arXiv 2608.18167](https://arxiv.org/abs/2608.18167)) [PR, ICML 2026 DL4C workshop]. + +Claude Code's documentation names the practical consequence. "A reviewer prompted to find gaps will usually report some, even when the work is sound... Chasing every finding leads to over-engineering." The reviewer should flag only gaps that affect correctness or the stated requirements, and check that nothing outside the task's scope changed ([Claude Code best practices](https://code.claude.com/docs/en/best-practices)) [V]. + +**Functional tests miss security; static analysis loops help.** In SusVibes, SWE-Agent with Claude 4 Sonnet was 57% functionally correct but only **11.8% secure** on security-sensitive feature requests, and adding vulnerability hints did not fix it ([PMLR](https://proceedings.mlr.press/v306/zhao26ax.html)) [PR, ICML 2026]. Feeding Bandit and Pylint findings back to GPT-4o for up to ten iterations cut security issues from over 40% to 13%, and readability and reliability issues to about 11% ([arXiv 2508.14419](https://arxiv.org/abs/2508.14419)) [PP]. Code LLMs hallucinate non-existent packages at average rates of at least 5.2% (commercial) and 21.7% (open-source), a "slopsquatting" supply-chain risk ([arXiv 2406.10279](https://arxiv.org/abs/2406.10279)) [PR, USENIX Security 2025]. + +On over-engineering the evidence is mixed. Trajectory minimization cut "CodeSlop" by 17.9–32.9% with negligible loss of correctness ([arXiv 2607.18161](https://arxiv.org/abs/2607.18161)) [PP]. Yet agents add tangled refactorings *less* often than humans do (21.43% against 36.72% of patches) ([arXiv 2605.22526](https://arxiv.org/abs/2605.22526)) [PP]. Scope creep should be measured, not assumed. + +These results assemble into a **verification ladder**. Each rung guards against a measured failure, and each maps onto a mechanism both tools provide: + +| Rung | Check | Measured failure it guards | Evidence | Mechanism | +|---|---|---|---|---| +| 1 | Format, lint and type-check after every edit | Lint/typing shortfalls in 75% of test-passing agent PRs; static issues | [METR](https://metr.org/blog/2025-08-12-research-update-towards-reconciling-slowdown-with-time-horizons/) [PP]; [arXiv 2508.14419](https://arxiv.org/abs/2508.14419) [PP] | Post-edit hook | +| 2 | Existing regression suite | Regressions; the regression filter behind vendor best-of-N | [Anthropic](https://www.anthropic.com/news/claude-sonnet-4-5) [V]; [Agentless](https://arxiv.org/abs/2407.01489) [PR] | Stop hook or check script | +| 3 | Failing reproduction or acceptance test written or approved by a human *before* the fix, then read-only | Test modification (more than 79% of Claude's cheating) | [TDFlow](https://arxiv.org/abs/2510.23761) [PR]; [ImpossibleBench](https://arxiv.org/html/2510.20270v1) [PP] | Pre-tool hook denying test edits; committed tests | +| 4 | A sanctioned "blocked: spec and tests conflict" exit plus a retry cap | Cheating under pressure; more cheating with more retries | [ImpossibleBench](https://arxiv.org/html/2510.20270v1) [PP] | Skill instruction; Stop-hook loop cap | +| 5 | Held-out acceptance tests the agent never saw (high-stakes work) | Visible-test overfitting that grows with code size | [SpecBench](https://arxiv.org/html/2605.21384) [PP] | CI job outside the agent's workspace | +| 6 | Security scanner and security tests on security-relevant diffs | 57% functional but 11.8% secure | [SusVibes](https://proceedings.mlr.press/v306/zhao26ax.html) [PR] | Hook or CI step | +| 7 | Fresh-context reviewer given the diff, spec and executed evidence, scoped to correctness and scope | Self-attribution bias (AUROC 0.99 → 0.89); over-reporting | [arXiv 2603.04582](https://arxiv.org/html/2603.04582v1) [PP]; [Cognition](https://cognition.com/blog/multi-agents-working) [V] | Read-only reviewer subagent | +| 8 | Evidence produced by the harness (commands, exit codes, output), plus an audit of any test, CI or config edits | Fabricated verification claims | [Transluce](https://transluce.org/investigating-o3-truthfulness) [PP]; [Replit](https://incidentdatabase.ai/entities/replit) [A] | Stop-hook transcript of the check script; diff check | +| 9 | Human merge-readiness review | Test-passing PRs that are not mergeable (0 of 15) | [METR](https://metr.org/blog/2025-08-12-research-update-towards-reconciling-slowdown-with-time-horizons/) [PP] | PR review | + +The ladder also constrains best-of-N. Sampling multiple attempts pays only when the selector is independent of the visible tests the generator saw: held-out tests, a hybrid of tests and a judge, or a fresh-context reviewer. Otherwise more search buys more gaming. diff --git a/docs/research/05-comparison-matrix.md b/docs/research/05-comparison-matrix.md new file mode 100644 index 0000000..b8f6f9c --- /dev/null +++ b/docs/research/05-comparison-matrix.md @@ -0,0 +1,95 @@ +# Six criteria score fifty-three approaches in one comparison matrix + +The matrix below scores every approach covered above on the six requested criteria, using a 1–5 scale where 5 is best. These are this study's judgments from the cited evidence, not measurements. + +| Criterion | What a 5 means | +|---|---| +| **Efficiency** | Lowest tokens, money and wall-clock per *accepted* outcome | +| **Reliability** | Highest measured or well-supported correctness | +| **Autonomy** | Can run longest unattended within a safe boundary | +| **Clarity** | Leaves the most legible, reviewable artifacts and state | +| **Simplicity** | Lowest setup and operating burden | + +**Evidence** uses its own scale: + +| Score | Basis | +|---|---| +| 5 | Several converging controlled or peer-reviewed results | +| 4 | One controlled or peer-reviewed result, or a large controlled preprint | +| 3 | Vendor or preprint numbers without controls | +| 2 | Vendor qualitative report or practitioner anecdote | +| 1 | None, or self-published only | + +## Methodologies: cheap loops beat full ceremony + +| # | Approach | Eff. | Rel. | Aut. | Clar. | Simp. | Evid. | Basis | +|---|---|---|---|---|---|---|---|---| +| 1 | Vendor core loop: explore → size-gated plan → implement → verify → commit | 4 | 4 | 3 | 4 | 5 | 3 | Converged guidance ([Claude Code](https://code.claude.com/docs/en/best-practices), [Codex](https://learn.chatgpt.com/docs/learn/best-practices), [Cursor](https://cursor.com/blog/agent-best-practices)) [V]; plans help, bad plans hurt ([arXiv 2604.12147](https://arxiv.org/abs/2604.12147)) [PP] | +| 2 | Interview → self-contained SPEC.md → fresh session | 3 | 4 | 4 | 5 | 4 | 2 | [Claude Code](https://code.claude.com/docs/en/best-practices) [V] | +| 3 | Long-running harness: initializer, JSON feature list, progress file, `init.sh`, one feature per session | 3 | 4 | 5 | 4 | 3 | 2 | [Anthropic](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents) [V], qualitative | +| 4 | OpenAI ExecPlans (living, restartable plan) | 3 | 3 | 4 | 5 | 3 | 2 | [Cookbook](https://developers.openai.com/cookbook/articles/codex_exec_plans) [V]; "7+ hours" unverified | +| 5 | GitHub Spec Kit | 2 | 3 | 3 | 3 | 2 | 2 | No outcome studies; review burden ([Böckeler](https://martinfowler.com/articles/exploring-gen-ai/sdd-3-tools.html)) [A]; Radar "Assess" ([Thoughtworks](https://www.thoughtworks.com/radar/techniques/spec-driven-development)) [A] | +| 6 | AWS Kiro specs (with Quick Spec) | 2 | 3 | 3 | 3 | 3 | 2 | [Kiro](https://kiro.dev/docs/specs/) [V]; 16 acceptance criteria for a bug fix ([Böckeler](https://martinfowler.com/articles/exploring-gen-ai/sdd-3-tools.html)) [A] | +| 7 | BMAD Method | 2 | 3 | 3 | 3 | 2 | 1 | [BMAD](https://github.com/bmad-code-org/BMAD-METHOD) [V]; no evaluation | +| 8 | OpenSpec (spec deltas, brownfield) | 3 | 3 | 3 | 4 | 4 | 1 | [OpenSpec](https://github.com/Fission-AI/OpenSpec) [A]; no evaluation | +| 9 | GSD Core (fresh-context waves) | 2 | 3 | 4 | 3 | 3 | 1 | [gsd-core](https://github.com/open-gsd/gsd-core) [A] | +| 10 | Superpowers (brainstorm → worktree → plan → subagent TDD → review) | 2 | 4 | 4 | 4 | 3 | 2 | [superpowers](https://github.com/obra/superpowers) [A]; [Willison](https://simonwillison.net/2025/Oct/10/superpowers/) [A] | +| 11 | Karpathy behavioural guidelines (four rules) | 4 | 3 | 2 | 5 | 5 | 2 | [repo](https://github.com/multica-ai/andrej-karpathy-skills) [A]; targets the "unwanted features" rejections in [arXiv 2601.15195](https://arxiv.org/abs/2601.15195) [PR] | +| 12 | Research → plan → implement with intentional compaction | 4 | 3 | 3 | 4 | 3 | 2 | [HumanLayer](https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/ace-fca.md) [A] | +| 13 | Ralph loop (one item per loop, tests as backpressure) | 3 | 2 | 5 | 2 | 4 | 2 | [Huntley](https://ghuntley.com/ralph/) [A]; greenfield only | +| 14 | Task Master (PRD → task graph via MCP) | 2 | 3 | 3 | 4 | 3 | 1 | 36 tools ≈ 21k tokens ([README](https://github.com/eyaltoledano/claude-task-master)) [V] | +| 15 | claude-flow / Ruflo swarm orchestration | 1 | 2 | 3 | 1 | 1 | 1 | Self-published infrastructure metrics only ([README](https://github.com/ruvnet/claude-flow)) [A] | +| 16 | Kent Beck augmented TDD (human picks next test) | 3 | 4 | 2 | 4 | 4 | 2 | [Beck](https://newsletter.kentbeck.com/p/augmented-coding-beyond-the-vibes) [A] | + +## Architectures: thin loops and context hygiene lead on efficiency + +| # | Approach | Eff. | Rel. | Aut. | Clar. | Simp. | Evid. | Basis | +|---|---|---|---|---|---|---|---|---| +| 17 | Bash-only minimal loop (mini-SWE-agent) | 4 | 4 | 3 | 4 | 5 | 4 | 70–77% Verified with an identical prompt ([Willison](https://simonwillison.net/2026/Feb/19/swe-bench/)) [PP]; bash-only is cheaper for capable models ([arXiv 2609.20804](https://arxiv.org/abs/2609.20804)) [PP] | +| 18 | Two-tool scaffold with exact-match edits and absolute paths | 4 | 4 | 3 | 4 | 5 | 3 | 49% Verified ([Anthropic](https://www.anthropic.com/research/swe-bench-sonnet)) [V] | +| 19 | SWE-agent custom ACI (bounded views, lint guardrail) | 3 | 3 | 3 | 3 | 3 | 4 | Ablations on 2024 models ([arXiv 2405.15793](https://arxiv.org/html/2405.15793)) [PR] | +| 20 | Agentless fixed pipeline | 5 | 3 | 2 | 4 | 3 | 4 | 32.00% Lite at ≈ $0.70 ([arXiv 2407.01489](https://arxiv.org/abs/2407.01489)) [PR] | +| 21 | CodeAct / OpenHands code-as-action | 3 | 4 | 4 | 3 | 3 | 4 | Up to +20% over JSON actions ([arXiv 2402.01030](https://arxiv.org/abs/2402.01030)) [PR] | +| 22 | Aider architect/editor split | 3 | 3 | 2 | 3 | 4 | 3 | +3–5 points on Aider's benchmark ([Aider](https://aider.chat/2024/09/26/architect.html)) [V]; "often faulty" ([Cognition](https://cognition.com/blog/dont-build-multi-agents)) [V] | +| 23 | Single writer with read-only helper subagents (explore, review) | 4 | 4 | 4 | 4 | 4 | 3 | [Cognition](https://cognition.com/blog/multi-agents-working) [V]; [arXiv 2512.08296](https://arxiv.org/abs/2512.08296v2) [PP] | +| 24 | Strong lead plus cheap sidekick (Devin Fusion) | 5 | 4 | 4 | 3 | 3 | 3 | 63.1 at $1.35 vs 63.6 at $3.51 ([Cognition](https://cognition.com/blog/devin-fusion)) [V] | +| 25 | Planner / generator / evaluator harness | 1 | 4 | 5 | 4 | 2 | 2 | $9 broken vs $200 playable ([Anthropic](https://www.anthropic.com/engineering/harness-design-long-running-apps)) [V] | +| 26 | Hierarchical planners, workers and judge at hundreds of agents | 1 | 3 | 5 | 2 | 1 | 2 | 1M+ LoC, "still needed careful review" ([Cursor](https://cursor.com/blog/scaling-agents)) [V] | +| 27 | Lock-file parallel agents in containers, no orchestrator | 2 | 3 | 5 | 3 | 2 | 2 | ≈ $20k, needs a near-perfect oracle ([Anthropic](https://www.anthropic.com/engineering/building-c-compiler)) [V] | +| 28 | Flat peer multi-agent swarm | 1 | 1 | 3 | 1 | 1 | 4 | −39–70% on sequential tasks, 17.2× error amplification ([arXiv 2512.08296](https://arxiv.org/abs/2512.08296v2)) [PP]; [MAST](https://arxiv.org/html/2503.13657v3) [PP] | +| 29 | Best-of-N with regression-test filter and selector | 2 | 5 | 4 | 3 | 2 | 4 | 77.2 → 82.0 ([Anthropic](https://www.anthropic.com/news/claude-sonnet-4-5)) [V]; 60.6 → 66.4 ([OpenHands](https://openhands.dev/blog/sota-on-swe-bench-verified-with-inference-time-scaling-and-critic-model)) [V]; [arXiv 2501.14723](https://arxiv.org/abs/2501.14723) [PP] | +| 30 | MCTS tree search (SWE-Search) | 1 | 4 | 4 | 2 | 1 | 4 | +23% relative ([arXiv 2410.20285](https://arxiv.org/abs/2410.20285)) [PR] | +| 31 | Self-evolving scaffold (Live-SWE-agent) | 3 | 4 | 4 | 2 | 2 | 3 | 77.4% Verified ([arXiv 2511.13646](https://arxiv.org/abs/2511.13646)) [PP] | +| 32 | Agentic grep/glob search, with semantic search for large repos | 4 | 4 | 4 | 4 | 5 | 3 | +12.5% QA accuracy; +2.6% retention on ≥ 1k-file repos ([Cursor](https://cursor.com/blog/semsearch)) [V] | +| 33 | Observation masking / tool-result clearing | 5 | 4 | 4 | 3 | 4 | 4 | About half the cost, same solve rate ([arXiv 2508.21433](https://arxiv.org/abs/2508.21433v3)) [PR]; +29% ([Anthropic](https://claude.com/blog/context-management)) [V] | +| 34 | LLM-summary compaction | 3 | 3 | 4 | 3 | 5 | 4 | No better than masking, at higher cost ([arXiv 2508.21433](https://arxiv.org/abs/2508.21433v3)) [PR] | +| 35 | Minimal, human-written instruction file | 4 | 3 | 3 | 5 | 5 | 4 | About +4% for developer-written files ([arXiv 2602.11988](https://arxiv.org/abs/2602.11988)) [PP]; [IFScale](https://arxiv.org/abs/2507.11538) [PR] | +| 36 | LLM-generated or overview-heavy instruction file | 2 | 2 | 3 | 3 | 4 | 4 | −0.5% to −3% success, +20–23% cost ([arXiv 2602.11988](https://arxiv.org/html/2602.11988v1)) [PP] | +| 37 | Deferred tool loading (tool search, lazy MCP) | 5 | 4 | 4 | 3 | 4 | 3 | −85% definition tokens; accuracy 49% → 74% ([Anthropic](https://www.anthropic.com/engineering/advanced-tool-use)) [V] | + +## Verification: trusted tests lead on reliability, prompts trail + +| # | Approach | Eff. | Rel. | Aut. | Clar. | Simp. | Evid. | Basis | +|---|---|---|---|---|---|---|---|---| +| 38 | Human-written or approved failing acceptance tests before implementation | 4 | 5 | 4 | 5 | 3 | 4 | 94.3% Verified ([TDFlow](https://arxiv.org/abs/2510.23761)) [PR] | +| 39 | Agent-written tests as acceptance evidence | 3 | 2 | 3 | 2 | 5 | 4 | No change in resolution ([arXiv 2602.07900](https://arxiv.org/abs/2602.07900)) [PP] | +| 40 | Existing regression suite as filter | 4 | 4 | 4 | 4 | 5 | 4 | Vendor best-of-N filter ([Anthropic](https://www.anthropic.com/news/claude-4)) [V]; [Agentless](https://arxiv.org/abs/2407.01489) [PR] | +| 41 | Held-out, spec-derived acceptance tests | 3 | 5 | 4 | 4 | 2 | 3 | 43–48 point visible/held-out gap ([SpecBench](https://arxiv.org/html/2605.21384)) [PP] | +| 42 | Read-only tests plus an explicit "impossible / conflict" exit | 5 | 4 | 4 | 4 | 4 | 3 | Cheating 54% → 9% ([ImpossibleBench](https://arxiv.org/html/2510.20270v1)) [PP] | +| 43 | Static-analysis and scanner feedback loop | 4 | 4 | 4 | 4 | 4 | 3 | Security issues > 40% → 13% ([arXiv 2508.14419](https://arxiv.org/abs/2508.14419)) [PP] | +| 44 | Security tests on security-sensitive work | 3 | 4 | 3 | 4 | 2 | 4 | 57% functional vs 11.8% secure ([SusVibes](https://proceedings.mlr.press/v306/zhao26ax.html)) [PR] | +| 45 | Fresh-context reviewer, scoped to correctness and scope | 3 | 4 | 4 | 4 | 4 | 3 | AUROC 0.99 fresh vs 0.89 self ([arXiv 2603.04582](https://arxiv.org/html/2603.04582v1)) [PP]; ≈ 2 bugs/PR ([Cognition](https://cognition.com/blog/multi-agents-working)) [V] | +| 46 | Same-context self-review / "reflect" step | 3 | 2 | 3 | 2 | 5 | 3 | Self-attribution bias ([arXiv 2603.04582](https://arxiv.org/html/2603.04582v1)) [PP]; "talked itself into approving" ([Anthropic](https://www.anthropic.com/engineering/harness-design-long-running-apps)) [V] | +| 47 | LLM-as-judge as sole selector | 3 | 2 | 4 | 2 | 3 | 4 | Randomness and order bias ([CodeJudgeBench](https://arxiv.org/abs/2507.10535)) [PR]; overcorrection ([arXiv 2603.00539](https://arxiv.org/abs/2603.00539)) [PP] | +| 48 | Trained critic on production traces | 4 | 4 | 4 | 2 | 1 | 3 | Best@8 73.8 vs Random@8 57.9 ([OpenHands](https://openhands.dev/blog/20260305-learning-to-verify-ai-generated-code)) [V] | +| 49 | Deterministic gates (post-edit hooks, Stop hook, CI) | 4 | 4 | 5 | 4 | 4 | 2 | [Claude Code](https://code.claude.com/docs/en/best-practices) [V]; "feedback sensors" ([Radar digest](https://agentic-ai.readthedocs.io/en/latest/AgenticTechStack/thoughtworks-radar-vol34/)) [A] | +| 50 | Browser end-to-end self-verification | 3 | 4 | 4 | 4 | 3 | 2 | "Dramatically improved performance", no numbers ([Anthropic](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents)) [V] | +| 51 | Harness-produced evidence and an annotated test plan (passed / failed / untested) | 4 | 3 | 4 | 5 | 4 | 2 | [Cognition](https://cognition.com/blog/testing-development) [V]; false-claim rates ([Transluce](https://transluce.org/investigating-o3-truthfulness)) [PP] | +| 52 | "Do not cheat" prompts | 5 | 1 | 3 | 3 | 5 | 4 | 80% → 80% ([METR](https://metr.org/blog/2025-06-05-recent-reward-hacking/)) [PP]; mixed ([ImpossibleBench](https://arxiv.org/html/2510.20270v1)) [PP] | +| 53 | LLM monitors for cheating | 3 | 2 | 4 | 2 | 3 | 3 | 86–89% detection on small tasks vs 42–65% at repository scale ([ImpossibleBench](https://arxiv.org/html/2510.20270v1)) [PP] | + +No approach scores well on every criterion, and that is the matrix's main lesson. The highest-reliability items are trusted tests, held-out tests and best-of-N with grounded selection (rows 29, 38 and 41). All three cost human effort or compute. The highest-efficiency items are cheap and well supported: observation masking, deferred tool loading, a lead with a cheap sidekick, and the Agentless pipeline (rows 33, 37, 24 and 20). + +The highest-autonomy designs are the hierarchical and evaluator harnesses (rows 25–27). They pay for autonomy with an order of magnitude more tokens and depend on a near-perfect oracle. The full-ceremony methodologies (rows 5–9 and 15) sit at the bottom on simplicity and evidence at the same time. + +The synthesized method below applies an explicit selection rule. It **keeps** every element rated at least 4 on reliability or efficiency that also has evidence of at least 3 and simplicity of at least 3: rows 1, 17, 18, 21, 23, 24, 32, 33, 35, 37, 38, 40, 42, 43 and 45, plus the validation step of row 20. It **admits only as risk-triggered escalations** the costly high-reliability rows 29, 41 and 44. It **adds** cheap, high-clarity practices whose weaker evidence is offset by low cost (rows 2, 3, 11, 49, 50 and 51). It **drops** the swarm and full-ceremony end entirely. diff --git a/docs/research/06-synthesized-method.md b/docs/research/06-synthesized-method.md new file mode 100644 index 0000000..6b2dfab --- /dev/null +++ b/docs/research/06-synthesized-method.md @@ -0,0 +1,64 @@ +# The synthesized method: a verified single-writer loop with state on disk + +The method proposed here is called the **Verified Single-Writer Loop (VSWL)**. It rests on six principles, each derived from the evidence above. First, **one agent writes**: other agents may explore, review or verify, and parallel writes happen only in isolated worktrees on independent units. Second, **context is a budget**: the always-on file is minimal, everything else loads just in time, and exploration happens in isolated helpers. Third, **a change is accepted only on independent, executed evidence** produced by the harness, never on the agent's own report. Fourth, **long-running state lives in files and git**, not in the context window. Fifth, **autonomy scales with how verifiable the task is and how much damage a mistake could do**. Sixth, **every component is removable**: each one encodes an assumption about what the model cannot yet do, so each must be ablatable when models change ([Anthropic](https://www.anthropic.com/engineering/harness-design-long-running-apps)) [V]. + +## Nine phases, most of them skippable for small work + +| Phase | What happens | Artifact | Gate or exit | Main evidence | +|---|---|---|---|---| +| 0. Orient | A session-start hook injects branch, `git status`, recent commits and the progress file. `init.sh` runs a smoke test before any new work | `progress.md`, `init.sh` | Smoke test green, or inherited breakage recorded first | [Anthropic harness](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents) [V] | +| 1. Triage | Classify the request as trivial (diff fits in one sentence), small, large/ambiguous, or high-stakes, and assign an autonomy level (table below) | One line in the plan or chat | Trivial work skips to phase 6 with checks | [Claude Code](https://code.claude.com/docs/en/best-practices) [V]; [arXiv 2604.12147](https://arxiv.org/abs/2604.12147) [PP] | +| 2. Intake and spec (large or ambiguous only) | Interview one question at a time, skipping questions when the request already says what matters. Mark unresolved ambiguity as `[NEEDS CLARIFICATION]`, then write `SPEC.md` (see below). A human approves; execution starts in a fresh session | `SPEC.md` | Human approval; no open markers | [Claude Code](https://code.claude.com/docs/en/best-practices) [V]; [Codex](https://learn.chatgpt.com/docs/learn/best-practices) [V]; [Spec Kit](https://github.com/github/spec-kit/blob/main/spec-driven.md) [V] | +| 3. Explore | A read-only explorer subagent on a fast model returns `path:line` pointers, short quoted snippets and a summary of at most ~2k tokens. The writer does not read hundreds of files itself | Pointer list | Pointers resolve; open questions listed | [Anthropic](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) [V]; [Cognition](https://cognition.com/blog/swe-grep) [V] | +| 4. Plan (small and up) | A plan file lists tasks. Each task names its files, one acceptance criterion and the verification command, and is sized to finish in one fresh context. The plan also lists what is out of scope. The writer re-reads it before each task | `PLAN.md` (or the tool's plan file) | Human review at levels 1–2; fresh-context critic at level 3 | [arXiv 2604.12147](https://arxiv.org/abs/2604.12147) [PP]; [Manus](https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus) [V]; [HumanLayer](https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/ace-fca.md) [A] | +| 5. Tests first | Write pre- and postconditions, then reproduction or acceptance tests from the spec. Confirm they fail. A human approves them for consequential work. Commit. From here on, tests are **read-only to the writer** (enforced by hook) | Committed failing tests | Tests fail for the right reason | [TDFlow](https://arxiv.org/abs/2510.23761) [PR]; [arXiv 2608.17177](https://arxiv.org/abs/2608.17177) [PR]; [ImpossibleBench](https://arxiv.org/html/2510.20270v1) [PP] | +| 6. Implement | The single writer works one task at a time. Post-edit hooks run the formatter, linter and type-checker on touched files and feed errors back. Mechanical tasks go to a cheaper subagent under a written brief. Parallel work happens only in worktrees. If the spec and tests conflict, the writer stops with `BLOCKED:` and a reason | Commits per task | Post-edit checks clean | [SWE-agent](https://arxiv.org/html/2405.15793) [PR]; [Cognition](https://cognition.com/blog/making-fable-cheaper-than-opus) [V] | +| 7. Verify | A Stop hook runs the check script (build, lint, types, full test suite) and blocks completion until it passes, with a retry cap (about 3) before escalating. A fresh-context reviewer on a strong model gets the diff, spec and check output and may flag only correctness, requirement or scope problems. Security-relevant diffs get scanners and security tests; UI work gets browser end-to-end checks with screenshots | Check log; review notes | Gate green; no unresolved correctness findings | [Claude Code](https://code.claude.com/docs/en/best-practices) [V]; [arXiv 2603.04582](https://arxiv.org/html/2603.04582v1) [PP]; [SusVibes](https://proceedings.mlr.press/v306/zhao26ax.html) [PR] | +| 8. Evidence and close | A completion report lists the commands, exit codes and key output lines, and marks each acceptance criterion passed, failed or untested. Update `progress.md` and `features.json`, then open the PR for human merge review | PR description; updated state files | Human merge decision | [Cognition](https://cognition.com/blog/testing-development) [V]; [METR](https://metr.org/blog/2025-08-12-research-update-towards-reconciling-slowdown-with-time-horizons/) [PP] | + +**Two escalations apply to high-stakes work.** The first is held-out acceptance tests, kept outside the agent's workspace and run in CI ([SpecBench](https://arxiv.org/html/2605.21384)) [PP]. The second is optional best-of-N: two to four worktree-isolated attempts, filtered by the regression and acceptance tests, then picked by the fresh-context reviewer ([Anthropic](https://www.anthropic.com/news/claude-sonnet-4-5)) [V]. Both are worth their cost only when the selector is independent of the tests the writer saw. + +**`SPEC.md`** has seven parts: the goal; context sources (files, docs, screenshots); constraints; an explicit out-of-scope list; three-tier boundaries of always do, ask first and never do ([Osmani](https://addyosmani.com/blog/good-spec/)) [A]; acceptance criteria written as checkable statements; and "done when", with the exact verification commands and their expected results. This is the common core of Codex's four-part prompt, Claude Code's `SPEC.md`, Superpowers' design document and ExecPlans' self-containment rule. **`features.json`** is used only for multi-session work. Each entry holds `id`, `description`, `test_steps` and `passes: false`, and the writer may change only `passes` ([Anthropic](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents)) [V]. **`progress.md`** is an append-only log of what was done, what was decided and what is next. It doubles as the handoff file when a session resets. + +## Five roles, one of which is a script + +| Role | Implemented as | Model | Tools | Context | +|---|---|---|---|---| +| Writer (lead) | Main session | Strongest available; effort high for planning and diagnosis, medium for routine edits | Edit, shell and tests within the permission boundary | Main thread, kept lean | +| Explorer | Subagent | Fast, cheap model (Haiku-class) | Read-only | Fresh; returns pointers, not conclusions | +| Implementer (optional) | Subagent, worktree-isolated when parallel | Cheaper model | Edit and shell | Fresh; receives a written brief | +| Reviewer | Subagent | Strong model, optionally a different vendor's model | Read-only | Fresh; sees only diff, spec and evidence | +| Gate | Hook script | None (deterministic) | Runs build, lint, types and tests | n/a | + +The human approves specs, plans and consequential tests, answers escalations, and merges. + +Delegation briefs follow a fixed template, because brief quality alone moved one task's score from 25 to 94 ([Cognition](https://cognition.com/blog/making-fable-cheaper-than-opus)) [V]: + +```text +GOAL: +CONTEXT: +CONSTRAINTS: +EDGE CASES: +DONE WHEN: +REPORT: changed files, commands run with exit codes, anything untested or uncertain. +Do not commit. Do not edit tests. If the task conflicts with the tests or spec, stop and report BLOCKED. +``` + +Delegation should happen early and only for mechanical work: renames, migrations, removals, test scaffolding and slow test triage. Design work, serial debugging and short tasks stay with the writer. Switching the main session's model mid-task breaks the prompt cache, so the model mix comes from subagents, each with its own cache, rather than from flipping the lead's model ([Cognition](https://cognition.com/blog/devin-fusion)) [V]. + +## Autonomy tiers follow verifiability and blast radius + +| Level | Use for | Claude Code setting | Cursor setting | Required gates | +|---|---|---|---|---| +| L0: explore and plan only | Unfamiliar code, ambiguous requests, incident analysis | `plan` mode | Plan or Ask mode | None (no writes) | +| L1: supervised edits | Core business logic, security, data migrations, critical fixes | `default` (Manual) or `acceptEdits` with `ask` rules | Allowlist run mode | Human approves the plan and tests; full ladder | +| L2: automatic inside a sandbox | Peripheral features, prototypes, refactors with strong tests, dependency bumps | `auto` or `acceptEdits` with `sandbox.enabled` | Auto-review with the sandbox | Hooks, Stop gate and fresh-context review | +| L3: unattended or background | Only tasks with machine-checkable success and low blast radius | `claude -p --permission-mode dontAsk` with explicit `--allowedTools`, in a worktree or container | Cloud Agents, or `agent -p --force` inside a VM | All of L2, plus held-out tests and human merge | + +This mirrors Anthropic's internal split between asynchronous peripheral work and supervised core work ([Anthropic PDF](https://www-cdn.anthropic.com/58284b19e702b49db9302d5b6f135ad8871e7658.pdf)) [V], and Codex's "tight by default" advice ([Codex](https://learn.chatgpt.com/docs/learn/best-practices)) [V]. + +At every level, the writer must stop and ask in seven situations: a choice between materially different approaches, missing credentials or access, ambiguity that changes behaviour, a conflict between the spec and the tests, the same check still failing after the retry cap, a needed edit outside the files the plan lists, and any destructive or irreversible action. These triggers match the reasons Claude itself pauses in the field: approach choice 35%, diagnostics 21%, vague requests 13%, credentials 12% ([Anthropic](https://www.anthropic.com/research/measuring-agent-autonomy)) [V]. They also cover the abort path that cut cheating ([ImpossibleBench](https://arxiv.org/html/2510.20270v1)) [PP] and the irreversible-action failure in the Replit incident ([AI Incident Database](https://incidentdatabase.ai/entities/replit)) [A]. + +## Context hygiene keeps the writer's window small + +The always-on file stays under Claude Code's 200-line target (Cursor's rule guidance is 500 lines) and as far below it as the repository allows. It holds only commands the agent cannot guess, non-default conventions, gotchas and hard boundaries ([arXiv 2602.11988](https://arxiv.org/abs/2602.11988)) [PP]. Clear the context between unrelated tasks. After two failed corrections, restart with a better prompt. Write logs and long outputs to files and grep them, printing summary lines such as `ERROR` on one line ([Anthropic C compiler](https://www.anthropic.com/engineering/building-c-compiler)) [V]. Tell compaction what to keep: modified files, test commands and open decisions ([Claude Code](https://code.claude.com/docs/en/best-practices)) [V]. Keep MCP servers few and scoped, preferring a CLI plus a skill where one exists ([Playwright MCP](https://github.com/microsoft/playwright-mcp)) [V]. diff --git a/docs/research/07-claude-code-and-cursor.md b/docs/research/07-claude-code-and-cursor.md new file mode 100644 index 0000000..4079c25 --- /dev/null +++ b/docs/research/07-claude-code-and-cursor.md @@ -0,0 +1,337 @@ +# Claude Code and Cursor expose the same seven levers under different contracts + +Every detail in this section comes from the official documentation as served on 14 October 2026. For Claude Code that is code.claude.com/docs, whose newest changelog entry is **v2.1.299 (14 October 2026)** ([changelog](https://code.claude.com/docs/en/changelog)). For Cursor it is cursor.com/docs, whose release timeline tops out at editor **3.24** ([release notes](https://cursor.com/docs/release-notes.md)). Both tools move quickly, so each version annotation below is the vendor's own. All of this section is [V] by construction. Where this study goes beyond the documentation, the text says so. + +## One repository layout serves both tools + +Cursor deliberately reads Claude Code's files. It loads skills from `.claude/skills/`, subagents from `.claude/agents/` and instructions from `CLAUDE.md`, and it can import hooks from `.claude/settings*.json` ([Cursor skills](https://cursor.com/docs/skills.md); [Cursor subagents](https://cursor.com/docs/subagents.md); [Cursor third-party hooks](https://cursor.com/docs/reference/third-party-hooks.md)). Claude Code reads none of Cursor's files. The simplest dual-tool kit therefore keeps shared assets in Claude's locations and adds Cursor-only files where the formats diverge: + +```text +repo/ +├── AGENTS.md # portable always-on instructions (Cursor native; Claude Code via import) +├── CLAUDE.md # first line "@AGENTS.md", then Claude-only lines; Cursor also always-applies it +├── .mcp.json # Claude Code MCP servers (${VAR} interpolation) +├── .worktreeinclude # gitignored files copied into Claude Code worktrees +├── .agent/ # method state, not tool config: SPEC.md, PLAN.md, progress.md, features.json, init.sh, check.sh +├── .claude/ +│ ├── settings.json # permissions (deny/ask/allow), sandbox, hooks +│ ├── rules/*.md # path-scoped rules, frontmatter `paths:` (Claude Code only) +│ ├── skills//SKILL.md # Agent Skills, read by both tools +│ ├── agents/*.md # subagents, read by both tools +│ └── hooks/*.sh # hook scripts shared by both tools +└── .cursor/ + ├── hooks.json # Cursor-native hooks pointing at .claude/hooks/*.sh + ├── rules/*.mdc # path-scoped rules, frontmatter `globs`/`alwaysApply` (Cursor only) + ├── agents/*.md # optional Cursor overrides; win over .claude/agents on a name clash + ├── permissions.json # terminal and MCP allowlists (IDE) + ├── cli.json # CLI permissions (project level holds permissions only) + ├── sandbox.json # sandbox filesystem/network policy + ├── mcp.json # Cursor MCP servers (${env:NAME} interpolation) + └── worktrees.json # worktree setup commands +``` + +Method state lives outside `.claude/` and `.cursor/` for two reasons. Claude Code never auto-approves writes to `.claude`, and Cursor's sandbox write-protects `.cursor/**/*.json` and `.claude/**/*.json` ([Claude Code permission modes](https://code.claude.com/docs/en/permission-modes#protected-paths); [Cursor sandbox.json](https://cursor.com/docs/reference/sandbox.md)). Both protections are good for configuration but would block the agent from updating its own progress files. + +## Instruction files: Claude Code skips AGENTS.md beside a CLAUDE.md + +| Aspect | Claude Code | Cursor | +|---|---|---| +| Project file | `./CLAUDE.md` or `./.claude/CLAUDE.md`, plus personal `./CLAUDE.local.md`. Loaded from the cwd and every ancestor at launch, concatenated root → cwd, not overridden. Subdirectory files load on demand when Claude touches files there | `AGENTS.md` at the root and in subdirectories; nested files combine with parents, and more specific instructions win. `CLAUDE.md` at the root is read "the same way" and **always applied**. Project rules live in `.cursor/rules/**/*.mdc` | +| AGENTS.md | Native since **v2.1.277 (18 Sep 2026)**. The default setting `claude-md-or-agents-md` reads AGENTS.md **only if no CLAUDE.md, .claude/CLAUDE.md or CLAUDE.local.md exists** on the path. Other values (`claude-md-and-agents-md`, `claude-md`, `managed-only`) can be set only via `pluginConfigs` in user settings, `--settings` or managed settings; they are ignored in project and local settings. The recommended portable pattern is a CLAUDE.md that imports `@AGENTS.md` | Native | +| Path-scoped rules | `.claude/rules/**/*.md`. The only frontmatter field read is `paths` (glob list); a rule loads when Claude reads or edits a matching file. Since **v2.0.64 (10 Dec 2025)** | `.mdc` only (plain `.md` in `.cursor/rules` is ignored). Fields `description`, `globs` and `alwaysApply`: `alwaysApply: true` means always; `globs` means auto-attach; `description` alone means the agent decides; none means manual `@rule` | +| Imports | `@path` anywhere, relative to the importing file, at most 4 hops. Imports load at launch and save no context | `@file` mentions in rules are **not inlined**; the agent reads the file when needed | +| User and organization | `~/.claude/CLAUDE.md` and `~/.claude/rules/`. Managed file at `/etc/claude-code/CLAUDE.md` (Linux) or via the `claudeMd` key in managed settings | User Rules (account-synced) and `~/.cursor/rules` (machine-local). Team Rules (dashboard, enforceable). Precedence: **Team → Project → User** | +| Size guidance | "target under 200 lines per CLAUDE.md file"; a file larger than 4 MiB is skipped | "Keep rules under 500 lines" | +| Delivery | As a user message after the system prompt, so advisory; the project-root file is re-injected after `/compact` | "Included at the start of the model context"; rules do not apply to Tab, Inline Edit or Bugbot | +| Audit tools | `/context`, `/memory`, `/doctor prompt-audit` (v2.1.283+), `InstructionsLoaded` hook | Context breakdown in the UI | +| Legacy | `.claude/commands/*.md` still works and was merged into skills in v2.1.3 | `.cursorrules` "is legacy and will be deprecated" | + +Sources: [Claude Code memory](https://code.claude.com/docs/en/memory), [Cursor rules](https://cursor.com/docs/rules.md), [Cursor rules help](https://cursor.com/help/customization/rules.md). + +Cursor always applies `CLAUDE.md`, so any Claude-only lines in it also reach Cursor. Keep them tool-neutral or put them in `.claude/rules/`. Whether Cursor expands `@AGENTS.md` inside CLAUDE.md is not documented. It does not matter for content, because Cursor loads AGENTS.md natively anyway. + +## Skills: one SKILL.md under .claude/skills serves both tools + +Both tools implement the open Agent Skills standard ([agentskills.io](https://agentskills.io/specification)): a folder `/SKILL.md` with YAML frontmatter, plus optional `scripts/`, `references/` and `assets/`. Only the descriptions load at startup; the body loads on invocation or when the agent judges the skill relevant. + +| Field | Claude Code | Cursor | Open spec | +|---|---|---|---| +| `name` | Optional; defaults to the directory name | **Required**; lowercase, digits, hyphens; must match the folder | Required, 1–64 chars, must match the parent directory | +| `description` | Recommended; with `when_to_use`, truncated at 1,536 chars in the listing | **Required** | Required, 1–1,024 chars | +| `paths` | Globs limiting auto-activation | Globs (legacy `globs` accepted) | — | +| `disable-model-invocation` | `true` = user-only, and the description is kept out of context | `true` = only via `/skill-name` | — | +| `user-invocable` | `false` hides it from the `/` menu | `false` (CLI changelog) | — | +| `allowed-tools` | Pre-approves tools **only for the invoking turn** | Not documented | Experimental | +| `context: fork`, `agent`, `background`, `model`, `effort`, `hooks`, `argument-hint`, `arguments`, `shell` | Supported | Not documented | — | +| `icon`, `color` | Ignored (unknown fields are silently ignored) | Custom Mode badge | — | +| `license`, `compatibility`, `metadata` | Accepted, not acted on | `metadata` documented | Optional | +| Discovery | `~/.claude/skills/`, `.claude/skills/` (cwd up to the repo root, plus nested dirs on demand), plugin `skills/`, managed dir. **Does not document** `.agents/skills/` | `.agents/skills/`, `.cursor/skills/`, `~/.agents/skills/`, `~/.cursor/skills/`, plus compatibility dirs `.claude/skills/`, `.codex/skills/`, `~/.claude/skills/`, `~/.codex/skills/` | Recommends `./skills/` and `.agents/skills/` | +| Introduced | v2.0.20 (16 Oct 2025); `context: fork` v2.1.0 (7 Jan 2026) | 2.4 (22 Jan 2026) | — | + +Sources: [Claude Code skills](https://code.claude.com/docs/en/skills), [Cursor skills](https://cursor.com/docs/skills.md), [spec](https://agentskills.io/specification). + +**The shared location is `.claude/skills//`, with `name` equal to the folder name.** Claude Code-specific fields are safe in Cursor only by assumption: Cursor does not document ignoring unknown keys. The SKILL.md bodies should stay under 500 lines; the spec recommends under 5,000 tokens. Claude Code limits the skill listing to 1% of the context window, and after auto-compaction it re-attaches the first 5,000 tokens of each invoked skill, up to 25,000 tokens combined. + +One trap matters for public kits. claude.ai uploads, the Skills API and `package_skill.py` accept only `name`, `description`, `license`, `compatibility`, `metadata` and `allowed-tools`. They **fail with a hard error** on any other key, such as `argument-hint` ([Claude Code skills](https://code.claude.com/docs/en/skills#using-skill-frontmatter-outside-claude-code)). In Claude Code, `` !`cmd` `` lines inject shell output before the model sees the skill. Any non-zero exit aborts the invocation (except exit 1 from search or compare commands), so append `|| true` where failure is acceptable. + +A minimal skill that is valid in both tools and in the open spec: + +```markdown +--- +name: verify-change +description: Runs the verification ladder (format, lint, types, full tests) and writes an evidence report. Use before declaring any coding task done, or when asked to verify or prove a change works. +--- +# Verify a change +Run `.agent/check.sh` from the repository root. Report the exact command, its exit code and the last 30 lines of output. +For each acceptance criterion in `.agent/SPEC.md`, mark it passed, failed or untested, quoting the evidence line. +Never edit tests, CI configuration or hook scripts to make a check pass. If the spec and the tests conflict, stop and report `BLOCKED:` with the conflicting lines. +``` + +## Subagents: Cursor reads Claude's files but not its tool limits + +| Aspect | Claude Code | Cursor | +|---|---|---| +| Files | `.claude/agents/*.md` (cwd up to the repo root, recursive), `~/.claude/agents/`, plugin `agents/`, managed dir, `--agents` JSON. Priority: managed > CLI > project > user > plugin | `.cursor/agents/`, `~/.cursor/agents/`, plus compatibility `.claude/agents/`, `.codex/agents/`, `~/.claude/agents/`, `~/.codex/agents/`. **`.cursor/` wins over `.claude/`** on a name clash, and project wins over user | +| Required fields | `name` (≤ 256 chars, no `:`) and `description`. A file missing either is **silently skipped** | None; `name` defaults to the filename | +| Tool restriction | `tools` (allowlist), `disallowedTools` (applied first), `Agent(type, …)` for spawning | **No `tools` field**; subagents inherit all parent tools, including MCP. `readonly: true` blocks edits and state-changing shell | +| Model | `sonnet`, `opus`, `haiku`, `fable`, a full ID such as `claude-opus-5-5`, or `inherit`. Resolution: invocation parameter > frontmatter > `CLAUDE_CODE_SUBAGENT_MODEL` > main model | `inherit` (default) or a model ID with parameters, e.g. `claude-opus-5[effort=high,context=300k]`. On legacy request plans without Max Mode, subagents run on Composer | +| Other fields | `permissionMode`, `maxTurns`, `skills` (preloaded), `mcpServers`, `hooks` (`Stop` becomes `SubagentStop`), `memory` (`user`/`project`/`local`), `background`, `omitClaudeMd` (v2.1.271+), `effort`, `isolation: worktree`, `color`, `initialPrompt` | `is_background` | +| Starting context | Fresh: the body is the **entire system prompt**, plus the delegation message, the CLAUDE.md hierarchy and git status (Explore and Plan skip both) | Clean; the parent must pass what is needed | +| Built-ins | `Explore` (read-only; **main model by default**), `Plan`, `general-purpose`, others | `explore` ("uses a faster model"), `browser` | +| Limits | Nesting depth 3; 20 concurrent; fork mode on by default in interactive sessions (v2.1.232+) | Main agent and its direct subagents may spawn; one nested level since 2.5 | +| Isolation | `isolation: worktree` creates a temporary worktree, removed if unchanged | On request: an isolated worktree or a cloud VM per subagent | +| Plugin caveat | Plugin subagents **ignore** `hooks`, `mcpServers` and `permissionMode` | Plugin agents document only `name` and `description` | + +Sources: [Claude Code subagents](https://code.claude.com/docs/en/sub-agents), [Cursor subagents](https://cursor.com/docs/subagents.md). + +A reviewer that works in both tools can declare both tools' restriction fields. Claude Code ignores unknown fields such as `readonly`, while Cursor reads `readonly` and does not document `tools`. + +```markdown +--- +name: diff-reviewer +description: Fresh-context reviewer. Use proactively after implementation and before reporting a task done. Reviews only the diff, the spec and the check output it is given. +model: inherit +tools: Read, Grep, Glob +readonly: true +--- +You review a change you did not write. Your inputs are in .agent/review-input.md: the diff, the acceptance criteria and the latest check output. +Report only correctness bugs, unmet acceptance criteria, and changes outside the spec's scope, each with file:line and evidence. +Do not suggest style changes, new abstractions, or tests for cases that cannot happen. If nothing qualifies, reply "No blocking findings." +``` + +The cheap explorer is the one place where the two tools need different files. `.claude/agents/explorer.md` sets `model: haiku`, `tools: Read, Grep, Glob`. To change the built-in instead, define an agent named `Explore` with `model: haiku`. A same-named `.cursor/agents/explorer.md` sets `readonly: true` and either `model: inherit` or a Cursor model ID, and it overrides the Claude file inside Cursor. The model IDs differ between tools, so `inherit` is the only portable value. + +## Hooks: the gates live here, and both tools fail open by default + +Instructions are advisory in both tools. Cursor's enterprise documentation says "rules are suggestions, not guarantees", and Claude Code's says hooks "guarantee the action happens" ([Cursor LLM safety](https://cursor.com/docs/enterprise/llm-safety-and-controls.md); [Claude Code best practices](https://code.claude.com/docs/en/best-practices)). The method therefore puts its gates in hooks. The two contracts differ in ways that matter for enforcement: + +| Aspect | Claude Code | Cursor | +|---|---|---| +| Where | `hooks` key in `~/.claude/settings.json`, `.claude/settings.json`, `.claude/settings.local.json` or managed settings; a plugin's `hooks/hooks.json`; skill or subagent frontmatter | `.cursor/hooks.json` (project, trusted workspaces only), `~/.cursor/hooks.json`, Team (dashboard, Enterprise), Enterprise MDM files (`/etc/cursor/hooks.json` and others); Claude Code settings hooks via Third-Party Imports | +| Shape | `{"hooks": {"": [{"matcher": "…", "hooks": [{"type": "command", "command": "…"}]}]}}` | `{"version": 1, "hooks": {"": [{"command": "…", "matcher": "…"}]}}` | +| Handler types | `command`, `http`, `mcp_tool`, `prompt`, `agent` (experimental) | `command`, `prompt` | +| Handler options | `if` (a permission rule), `timeout` (seconds; default 600 for command), `args` (exec form), `async`, `asyncRewake`, `shell`, `onFailure` (`continue` \| `block`, **v2.1.295+**), `once` (skills only) | `timeout` (seconds), `loop_limit` (default 5; `null` for Claude Code hooks), `failClosed` (default `false`), `matcher` (regex) | +| Exit 0 | Success. Stdout is parsed only if it is exactly one JSON object. Plain stdout becomes context only for `UserPromptSubmit`, `UserPromptExpansion`, `SessionStart` and `PostModelSwitch` | Success, use the JSON. **For permission hooks** (`preToolUse`, `beforeShellExecution`, `beforeMCPExecution`, `beforeReadFile`, `beforeTabFileRead`, `subagentStart`), invalid or non-schema JSON **blocks** | +| Exit 2 | Blocking error on events that can block; stderr becomes the reason | Block, equivalent to `permission: "deny"` | +| Other exit codes | **Non-blocking: the action proceeds**, unless `onFailure: "block"` | **Fail-open**, unless `failClosed: true`, in which case a crash, timeout, non-zero exit or empty output blocks | +| Timeout | A timed-out `PreToolUse` does **not** block unless `onFailure: "block"` | Blocks only with `failClosed: true` | +| Multiple hooks | All matching hooks run in parallel and merge across levels; `deny` > `defer` > `ask` > `allow` | All sources run; `deny` > `ask` > `allow`; message fields concatenate; for `followup_message` the last response wins | +| Environment | `CLAUDE_PROJECT_DIR`, `CLAUDE_PLUGIN_ROOT`, `CLAUDE_PLUGIN_DATA`, `CLAUDE_EFFORT`; no TTY | `CURSOR_PROJECT_DIR`, `CURSOR_VERSION`, `CURSOR_USER_EMAIL`, `CURSOR_TRANSCRIPT_PATH`, `CURSOR_CODE_REMOTE`, and `CLAUDE_PROJECT_DIR` as an alias | +| Trust | Interactive sessions hold hooks until the workspace trust dialog is accepted; `-p` treats the folder as trusted and **runs repository hooks** | Project hooks run in trusted workspaces; Cloud Agents run command hooks only | + +Sources: [Claude Code hooks reference](https://code.claude.com/docs/en/hooks), [Cursor hooks](https://cursor.com/docs/hooks.md). + +The method needs six hook points: + +| Method role | Claude Code event (output) | Cursor event (output) | +|---|---|---| +| Inject branch, status and progress at session start | `SessionStart`, matcher `startup\|resume\|clear\|compact`. Plain stdout or `hookSpecificOutput.additionalContext`; `CLAUDE_ENV_FILE` can persist env vars | `sessionStart` (fire-and-forget). `additional_context`, `env` | +| Block edits to tests, CI and agent config | `PreToolUse`, matcher `Edit\|Write`. Exit 2, or `hookSpecificOutput.permissionDecision: "deny"` with `permissionDecisionReason` | `preToolUse`, matcher `Write\|Delete`. Exit 2, or `permission: "deny"`. **`ask` is not enforced** for `preToolUse` | +| Gate dangerous shell commands | `PreToolUse`, matcher `Bash` with `if: "Bash(rm *)"`-style filters | `beforeShellExecution` (`allow`/`deny`/`ask`), matched on the full command string | +| Lint, format and type-check after each edit | `PostToolUse`, matcher `Edit\|Write`. Exit 2 shows stderr to Claude; or `decision: "block"` + `reason`; or `additionalContext` | `postToolUse`, matcher `Write`, returning `additional_context`; `afterFileEdit` for formatters that need no feedback | +| Refuse "done" until checks pass | `Stop` (no matcher). `{"decision": "block", "reason": "…"}` keeps Claude working; check `stop_hook_active`; **cap of 8 consecutive continuations** (`CLAUDE_CODE_STOP_HOOK_BLOCK_CAP`), reset when Claude calls a tool | `stop` (input `status`, `loop_count`). `followup_message` is auto-submitted as the next user message, capped by `loop_limit` | +| Check subagent output | `SubagentStop` (same decision format as Stop) | `subagentStop` (`followup_message` when `status` is `completed`) | + +Cursor imports Claude Code hooks when "Include Third-Party Plugins, Skills, and Other Configs" is on. It is on by default, but defaults to off for Enterprises since 3.0. The import maps `PreToolUse→preToolUse`, `PostToolUse→postToolUse`, `UserPromptSubmit→beforeSubmitPrompt`, `Stop→stop`, `SubagentStop→subagentStop`, `SessionStart→sessionStart`, `SessionEnd→sessionEnd` and `PreCompact→preCompact`. `Notification` and `PermissionRequest` are unsupported. Tool names map `Bash→Shell` and `Edit→Write`, and `Glob` is unsupported. Both the nested `hookSpecificOutput` format and the flat format are accepted, and `decision: "block"` on Stop becomes a follow-up ([Cursor third-party hooks](https://cursor.com/docs/reference/third-party-hooks.md)). + +Relying on the import alone is simpler. But it maps no shell-specific event such as `beforeShellExecution`, documents no `failClosed` option, and is off by default for Enterprises. Shipping `.cursor/hooks.json` as well means that, with imports on, Cursor runs each hook twice, so the scripts must be idempotent. This study recommends the native file for enforcement hooks. + +A single script can serve both tools. It branches on the `hook_event_name` field that both send ("PreToolUse" vs "preToolUse") and uses exit 2 for denials, which both honour. Cursor's permission hooks need an explicit allow verdict, because empty output blocks under `failClosed`. The sketch below is this study's design, not vendor code. Test it against captured payloads, since Cursor does not document the `tool_input` field names for `preToolUse`; its `afterFileEdit` and `beforeReadFile` payloads use `file_path`. + +```bash +#!/usr/bin/env bash +# .claude/hooks/protect-tests.sh: Claude Code PreToolUse (Edit|Write) and Cursor preToolUse (Write|Delete) +set -u +input=$(cat) +event=$(printf '%s' "$input" | jq -r '.hook_event_name // empty') +path=$(printf '%s' "$input" | jq -r '.tool_input.file_path // .tool_input.path // empty') +case "$path" in + */tests/*|*.test.*|*_test.*|*/.github/workflows/*|*/.claude/*|*/.cursor/*) + if [ "${VSWL_ALLOW_TEST_EDITS:-0}" != "1" ]; then # set only by the human, in the shell that launches a test-writing session + echo "Protected path: $path. Do not edit tests, CI or agent config while implementing. If the spec and tests conflict, stop and report BLOCKED." >&2 + exit 2 # Claude Code: block, stderr = reason. Cursor: deny. + fi ;; +esac +[ "$event" = "preToolUse" ] && printf '{"permission":"allow"}\n' # Cursor permission hooks need an explicit verdict +exit 0 +``` + +```bash +#!/usr/bin/env bash +# .claude/hooks/stop-gate.sh: Claude Code Stop and Cursor stop. Refuse completion until .agent/check.sh passes, at most 3 times. +set -u +input=$(cat) +event=$(printf '%s' "$input" | jq -r '.hook_event_name // empty') +sid=$(printf '%s' "$input" | jq -r '.session_id // .conversation_id // "default"') +root="${CLAUDE_PROJECT_DIR:-$PWD}"; counter="${TMPDIR:-/tmp}/vswl-stop-$sid" +if out=$("$root/.agent/check.sh" 2>&1); then rm -f "$counter"; exit 0; fi +n=$(( $(cat "$counter" 2>/dev/null || echo 0) + 1 )); echo "$n" > "$counter" +[ "$n" -gt 3 ] && { rm -f "$counter"; exit 0; } # retry cap reached: let it stop; the report must say checks failed and escalate +msg="Checks failed (attempt $n/3). Fix the cause; do not edit tests or CI. If impossible, report BLOCKED. Output tail: +$(printf '%s\n' "$out" | tail -n 30)" +if [ "$event" = "stop" ]; then jq -n --arg m "$msg" '{followup_message: $m}' # Cursor +else jq -n --arg m "$msg" '{decision: "block", reason: $m}'; fi # Claude Code +exit 0 +``` + +A pre-tool hook sees only the edit tools, so an agent can still change a test through the shell. The authoritative check is therefore a **diff audit** inside `.agent/check.sh` and CI: fail when `git diff --name-only ...HEAD` touches tests, CI or hook files without an approval marker. The matching configuration files: + +```json +{ + "$schema": "https://json.schemastore.org/claude-code-settings.json", + "hooks": { + "SessionStart": [{ "matcher": "startup|resume|clear|compact", + "hooks": [{ "type": "command", "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/session-start.sh", "args": [] }] }], + "PreToolUse": [{ "matcher": "Edit|Write", + "hooks": [{ "type": "command", "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/protect-tests.sh", "args": [], "timeout": 10, "onFailure": "block" }] }], + "PostToolUse": [{ "matcher": "Edit|Write", + "hooks": [{ "type": "command", "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/post-edit.sh", "args": [], "timeout": 120 }] }], + "Stop": [{ "hooks": [{ "type": "command", "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/stop-gate.sh", "args": [], "timeout": 900 }] }] + } +} +``` + +```json +{ + "version": 1, + "hooks": { + "sessionStart": [{ "command": ".claude/hooks/session-start.sh" }], + "preToolUse": [{ "command": ".claude/hooks/protect-tests.sh", "matcher": "Write|Delete", "failClosed": true, "timeout": 10 }], + "beforeShellExecution": [{ "command": ".claude/hooks/guard-shell.sh", "failClosed": true, "timeout": 10 }], + "postToolUse": [{ "command": ".claude/hooks/post-edit.sh", "matcher": "Write", "timeout": 120 }], + "stop": [{ "command": ".claude/hooks/stop-gate.sh", "loop_limit": 3, "timeout": 900 }] + } +} +``` + +The first block belongs in `.claude/settings.json` and the second in `.cursor/hooks.json`. Cursor runs project hooks from the project root, so the paths are relative. Claude Code recommends exec form (`args`) with `${CLAUDE_PROJECT_DIR}`. Inside a worktree, `${CLAUDE_PROJECT_DIR}` keeps pointing at the original root, so scripts that need the worktree should read `cwd` from the input. + +Debug Claude Code hooks with `claude --debug-file ` (v2.1.296+ logs each hook's status and duration) and `/hooks`. In Cursor, use the Hooks tab and the "Hooks" output channel. Hooks have existed in Claude Code since v1.0.38 (30 Jun 2025). In Cursor they arrived as a beta in 1.7 (29 Sep 2025), project hooks came in 2.1, and Claude-hook compatibility in the CLI in 2.4 ([Cursor changelog 1.7](https://cursor.com/changelog/1-7); [2.1](https://cursor.com/changelog/2-1); [2.4](https://cursor.com/changelog/2-4)). + +## Permissions: deny rules apply before trust, allow rules after + +**Claude Code settings.** Precedence runs managed > `--settings` > `.claude/settings.local.json` > `.claude/settings.json` > `~/.claude/settings.json`. Scalar keys take the highest-precedence value, and list keys such as `permissions.allow` merge. Rules use `Tool` or `Tool(specifier)` syntax and are evaluated **deny → ask → allow**. A deny at any level cannot be overridden: "An allow rule can't carve an exception out of a deny rule" ([Claude Code permissions](https://code.claude.com/docs/en/permissions)). Bash rules match by prefix (`Bash(npm run *)`), and compound commands are split, so each part must match. Bash deny rules **are not a security boundary**, because `sh -c` or absolute paths bypass them; the sandbox or a hook must back them up. Read and Edit rules use gitignore syntax; `Write(...)` path rules are ignored with a warning. + +Three committed-settings behaviours catch kit authors. First, a repository's `permissions.allow`, `additionalDirectories`, `extraKnownMarketplaces` and most `env` apply only after the user trusts the folder, while `deny` and `ask` apply immediately. Second, `defaultMode: "auto"` or `"bypassPermissions"` in project or local settings never takes effect. Third, in `-p` mode repository hooks and skill `allowed-tools` run, but `permissions.allow` does not apply ([Claude Code settings](https://code.claude.com/docs/en/settings)). + +The permission modes are `default` (Manual), `acceptEdits`, `plan`, `auto` (classifier-checked; the terminal default since v2.1.283), `dontAsk` (denies anything that would prompt, for CI) and `bypassPermissions` (containers and VMs only). Writes to `.git`, `.claude`, `.vscode`, `.husky` and similar paths are never auto-approved outside bypass ([permission modes](https://code.claude.com/docs/en/permission-modes)). The sandbox (`"sandbox": {"enabled": true}`) runs Bash under Seatbelt on macOS, or bubblewrap and socat on Linux/WSL2. It falls back to unsandboxed unless `failIfUnavailable: true`. `allowUnsandboxedCommands: false` disables the escape hatch. Note that reads default to "most of the machine, including credential files", so add `filesystem.denyRead`, and that hooks, MCP servers and the file tools run *outside* the sandbox ([Claude Code sandboxing](https://code.claude.com/docs/en/sandboxing.md)). + +A minimal committed baseline follows. Its deny rules and sandbox apply immediately; its allow rules are conveniences that take effect only after trust. Merge it with the hooks block above into a single `.claude/settings.json`: + +```json +{ + "$schema": "https://json.schemastore.org/claude-code-settings.json", + "permissions": { + "deny": ["Read(./.env)", "Read(./.env.*)", "Bash(git push *)", "Bash(curl *)", "Bash(wget *)"], + "allow": ["Bash(./.agent/check.sh *)", "Bash(git diff *)", "Bash(git status *)"] + }, + "sandbox": { "enabled": true, "failIfUnavailable": true, "allowUnsandboxedCommands": false } +} +``` + +Organizations should also set `permissions.disableBypassPermissionsMode: "disable"` in managed settings. Validate with the published schema, `/status` (setting sources) and `claude doctor`. + +**Cursor run modes and permission files.** The IDE has three Run Modes: **Auto-review** (the recommended default since 3.6, 29 May 2026), **Allowlist** and **Run Everything**. In Auto-review, allowlisted calls run, other shell commands are sandboxed where possible, and the rest go to a classifier that "uses Gemini 3.5 Flash Lite, with Claude 4.5 Haiku as the fallback". Cursor states plainly that "Auto-review is not a security boundary" ([Run Modes](https://cursor.com/docs/agent/security/run-modes.md)). + +`permissions.json` lives at `~/.cursor/permissions.json` or `/.cursor/permissions.json` and is JSONC; arrays concatenate. It holds `mcpAllowlist` (entries `server:tool` with `*` wildcards), `terminalAllowlist` (command prefixes) and `autoRun.allow_instructions` / `autoRun.block_instructions`. The last two are natural-language steering for the classifier, "not enforcement". Precedence is team admin > permissions.json > IDE settings ([permissions.json](https://cursor.com/docs/reference/permissions.md)). + +`sandbox.json` sets network and filesystem policy, merged per-user < per-repo < team-admin < hardcoded, with deny beating allow. Some paths are always write-protected: `.cursor/**/*.json`, `.claude/**/*.json`, `.vscode/**`, `.git/hooks/**`, `.git/config` and `.cursorignore`. Private ranges and `169.254.169.254` are blocked ([sandbox.json](https://cursor.com/docs/reference/sandbox.md)). The CLI reads `~/.cursor/cli-config.json` and a project `.cursor/cli.json`, which may hold only permissions. Its tokens are `Shell(cmd)`, `Read(glob)`, `Write(glob)`, `WebFetch(domain)` and `Mcp(server:tool)`, with deny taking precedence. The documented minimal global file is `{"version":1,"editor":{"vimMode":false},"permissions":{"allow":["Shell(ls)"],"deny":[]}}` ([CLI configuration](https://cursor.com/docs/cli/reference/configuration.md)). `.cursorignore` blocks the agent's file tools, but "the terminal and MCP server tools used by Agent cannot block access to code governed by `.cursorignore`" ([ignore file](https://cursor.com/docs/reference/ignore-file.md)). + +## Plugins: one repository can be a marketplace for both tools + +Both tools package skills, agents, hooks and MCP servers into versioned plugins distributed through Git-hosted marketplaces. Cursor's own official repository ships **both** `.cursor-plugin/marketplace.json` and `.claude-plugin/marketplace.json` at its root ([cursor/plugins](https://github.com/cursor/plugins)), which makes a dual-manifest repository a proven pattern. + +| Aspect | Claude Code | Cursor | +|---|---|---| +| Manifest | `.claude-plugin/plugin.json`, optional; **only `name` required** (kebab-case; names starting `claude-`, `anthropic-`, `anthropics-` or `cc-plugin-` are errors) | `.cursor-plugin/plugin.json`, required; **only `name` required** (lowercase kebab; alphanumerics, hyphens, periods). Also accepts open Agent Plugins with a root `plugin.json` and `$schema` `https://agent-plugins.org/schemas/1.0.0/plugin.schema.json` (skills and MCP only) | +| Default components | `skills/`, `commands/`, `agents/`, `hooks/hooks.json`, `.mcp.json`, `.lsp.json`, `output-styles/`, `workflows/`, `themes/`, `monitors/monitors.json`, `bin/` (added to PATH), `settings.json` | `rules/*.mdc`, `skills/`, `agents/`, `commands/`, `hooks/hooks.json`, `mcp.json` | +| Manifest component fields | `skills` **adds to** `skills/`; `commands`, `agents`, `outputStyles`, `workflows` **replace** defaults; `hooks` and `mcpServers` **merge** | Any component field **replaces** folder discovery for that component | +| Not supported | A plugin-root `CLAUDE.md` is **not loaded**; plugin subagents ignore `hooks`, `mcpServers`, `permissionMode` | Plugin skills and agents document only `name` and `description` | +| Paths and variables | Paths must start with `./` and stay inside the root. `${CLAUDE_PLUGIN_ROOT}` changes on update; persistent state goes in `${CLAUDE_PLUGIN_DATA}` | `${CURSOR_PLUGIN_ROOT}` and `${CLAUDE_PLUGIN_ROOT}` expand in `command`, `args`, `env` and `cwd`; the open standard's `${PLUGIN_ROOT}` is not expanded in `mcp.json` | +| Versioning | Setting `version` **pins users** until it changes; omit it to track commits | `version` field; marketplace updates are reviewed | +| Marketplace file | `.claude-plugin/marketplace.json`: required `name`, `owner {name}`, `plugins[]`, with each entry needing `name` and `source`. Sources: `./relative`, `github`, `url`, `git-subdir`, `npm`, `archive` (v2.1.224+), `command` (v2.1.229+) | `.cursor-plugin/marketplace.json`: required `name`, `owner {name}`, `plugins[]` (file ≤ 10 MB). Entries need `name`; `source` is a path or `{path}`; manifest values win when merged | +| Install | `/plugin marketplace add owner/repo`, then `/plugin install @`; shell `claude plugin install

@ --scope project`; repo-level auto-enable via `extraKnownMarketplaces` + `enabledPlugins` in `.claude/settings.json` (after trust) | Customize → Install (project or user scope); `/add-plugin` (2.5); CLI `/plugin`; local development in `~/.cursor/plugins/local/`; team marketplaces (Teams: 1, Enterprise: unlimited) with Default Off / Default On / Required modes | +| Validate and test | `claude plugin validate

[--strict]`; `claude --plugin-dir ./p` for one session; `claude plugin eval` (**v2.1.269, 11 Sep 2026**) | Submission checklist; public marketplace plugins must be open source and are "manually reviewed", including every update | +| Introduced | v2.0.12 (9 Oct 2025) | 2.5 (17 Feb 2026); team marketplaces 2.6 (3 Mar 2026) | + +Sources: [Claude Code plugin manifest](https://code.claude.com/docs/en/plugins/manifest-reference), [marketplace reference](https://code.claude.com/docs/en/plugins/marketplace-reference), [CLI reference](https://code.claude.com/docs/en/plugins/cli-reference), [Cursor plugins](https://cursor.com/docs/plugins.md), [Cursor plugins reference](https://cursor.com/docs/reference/plugins.md). + +The two tools use the same default `hooks/hooks.json` path with **different schemas**. A dual plugin should keep Claude Code's format at that default path, which Claude Code merges automatically. The Cursor manifest should then point `"hooks"` at a separate file, because in Cursor a manifest field replaces folder discovery: + +```text +hyper-agents/ # one Git repo = a marketplace for both tools +├── .claude-plugin/marketplace.json {"name":"hyper-agents","owner":{"name":"Hyper AI Lab"},"plugins":[{"name":"vswl","source":"./plugins/vswl","description":"Verified Single-Writer Loop"}]} +├── .cursor-plugin/marketplace.json {"name":"hyper-agents","owner":{"name":"Hyper AI Lab"},"plugins":[{"name":"vswl","source":"plugins/vswl","description":"Verified Single-Writer Loop"}]} +└── plugins/vswl/ + ├── .claude-plugin/plugin.json {"name":"vswl","description":"…","author":{"name":"…"}} + ├── .cursor-plugin/plugin.json {"name":"vswl","description":"…","author":{"name":"…"},"hooks":"./hooks/cursor-hooks.json"} + ├── skills//SKILL.md # both tools + ├── agents/*.md # both tools (Claude Code drops hooks/mcpServers/permissionMode here) + ├── rules/*.mdc # Cursor only (always-on rules; Claude Code plugins cannot carry CLAUDE.md) + ├── hooks/hooks.json # Claude Code format: {"hooks":{"Stop":[…]}}, using ${CLAUDE_PLUGIN_ROOT} + ├── hooks/cursor-hooks.json # Cursor format: {"version":1,"hooks":{"stop":[…]}}, using ${CURSOR_PLUGIN_ROOT} + └── scripts/ # shared hook scripts +``` + +This layout is this study's composition of documented rules. Cursor's own example uses a bare relative `source` (`"plugin-one"`), so validate both manifests before publishing. A plugin also cannot inject always-on text into Claude Code through a `CLAUDE.md`. Always-on guidance must come from a skill, a `SessionStart` hook's `additionalContext`, or an output style with `force-for-plugin`. Plugin delivery has a security advantage too: hook scripts installed in the plugin cache sit outside the workspace, so an agent writing inside the repository cannot rewrite them. + +## MCP: the same JSON shape with incompatible interpolation + +| Aspect | Claude Code | Cursor | +|---|---|---| +| Project file | `.mcp.json` at the root: `{"mcpServers": {…}}`. Remote entries **must** set `"type": "http"` (alias `streamable-http`), `"sse"` or `"ws"`; an entry with a `url` but no `type` is skipped | `.cursor/mcp.json` (project) or `~/.cursor/mcp.json`: `{"mcpServers": {…}}` with `command`/`args`/`env`/`envFile`, or `url`/`headers` with optional static OAuth `auth` | +| Interpolation | `${VAR}` and `${VAR:-default}` in `command`, `args`, `env`, `url`, `headers` | `${env:NAME}`, `${userHome}`, `${workspaceFolder}`, `${workspaceFolderBasename}`, `${pathSeparator}` | +| Approval | Interactive sessions ask before connecting `.mcp.json` servers, and "a cloned repository can't approve its own servers"; `-p`, SDK and cloud sessions connect without asking | Connections need approval, and each tool call needs approval unless allowlisted via `mcpAllowlist`; MCP follows the Run Modes | +| Context cost control | Tool Search (deferred loading); subagent-scoped `mcpServers` keep tool schemas out of the main context | Definitions load only when needed (since 2.4) | +| Tool naming | `mcp____`; plugin servers `mcp__plugin____` | `MCP:` in hook matchers | + +Sources: [Claude Code MCP](https://code.claude.com/docs/en/mcp), [Cursor MCP](https://cursor.com/docs/mcp.md). + +The JSON shapes are compatible, but the interpolation syntax is not, so keep two files. The servers with the clearest value for this method cover browser verification, current library documentation and the code host. For browsers, use Playwright MCP or Chrome DevTools MCP, or Cursor's built-in browser; Microsoft itself recommends Playwright's CLI plus Skills for coding agents. For documentation, use Context7 or its `ctx7` CLI-plus-skill mode. For GitHub, run the official server with minimal `--toolsets`, `--read-only` and `--lockdown-mode`, which GitHub calls "**not** an authorization boundary" ([Playwright MCP](https://github.com/microsoft/playwright-mcp); [Context7](https://github.com/upstash/context7); [GitHub MCP](https://github.com/github/github-mcp-server)) [V]. Database servers should be read-only and project-scoped, as in Supabase's `read_only=true&project_ref=…` ([Supabase MCP](https://github.com/supabase/mcp)) [V]. Evidence for these servers' benefit is vendor- or self-reported only, and star counts measure popularity. + +## Headless runs: the init event proves what actually loaded + +Claude Code's `claude -p` runs with the same discovery as an interactive session but **without the trust dialog**. The machine-readable record of what loaded is the first `system/init` event of `--output-format stream-json --verbose`. It lists `tools`, `mcp_servers`, `slash_commands`, `skills`, `agents`, `plugins`, `plugin_errors`, `mcp_server_errors`, `permissionMode` and `claude_code_version`, and CI should fail on non-empty `plugin_errors` or `mcp_server_errors` ([Claude Code headless](https://code.claude.com/docs/en/headless)): + +```bash +claude -p "/vswl:verify-change" --plugin-dir ./plugins/vswl --permission-mode dontAsk \ + --output-format stream-json --verbose \ + | jq -c 'select(.type=="system" and .subtype=="init") | {skills, agents, slash_commands, plugins, plugin_errors, mcp_servers}' +``` + +`--bare` (v2.1.81, 20 Mar 2026) skips all discovery and hides the skill listing, so never use it to test whether skills trigger. For deterministic runs, use `--setting-sources`, `--allowedTools`, `--max-turns` and `--max-budget-usd`. `claude plugin eval` runs each `evals//prompt.md` (frontmatter `max_turns`, `allowed_tools`, `timeout_seconds`, `model`, `tags`) three times with and three times without the plugin. Graders live in `graders/*.md` with types `llm`, `baseline`, `regex`, `tool_used`, `tool_order` and `file_exists`. For example, `type: tool_used`, `tool: Skill` with an `input_match` regex checks that a skill actually fired ([plugin evals](https://code.claude.com/docs/en/plugin-evals)). + +Cursor's CLI binary is now `agent`. `agent -p` applies changes only with `--force` (or `--yolo`); without it, changes are only proposed. Other flags include `--output-format text|json|stream-json`, `--trust` (headless only), `--approve-mcps`, `--sandbox enabled|disabled`, `--mode plan|ask`, `--worktree` and `--plugin-dir`, with authentication via `CURSOR_API_KEY` ([Cursor headless](https://cursor.com/docs/cli/headless.md); [parameters](https://cursor.com/docs/cli/reference/parameters.md)). Cursor documents no built-in eval runner, so its behavioural tests run the same case prompts through `agent -p` with external graders. + +## Plan mode and worktrees: git is the only complete checkpoint + +| Aspect | Claude Code | Cursor | +|---|---|---| +| Plan mode | `Shift+Tab`, `/plan`, `--permission-mode plan`, or `{"permissions": {"defaultMode": "plan"}}`. Plans go to `~/.claude/plans/` or `plansDirectory`; a hook on `ExitPlanMode` receives `plan` and `planFilePath` | `Shift+Tab` or `/plan`; asks clarifying questions and produces an editable plan, then **Build**. Docs say plans save to the home directory by default, with "Save to workspace" to move them (Cursor's January 2026 blog says `.cursor/plans/`) | +| Checkpoints | Automatic per prompt, 100 kept; `/rewind` or `Esc Esc`. **Bash changes are not tracked**, nor are background-subagent edits; "not a replacement for version control" | Automatic before significant changes, stored locally and separate from git; restoring reverts files only | +| Worktrees | `claude --worktree ` creates `.claude/worktrees//` on branch `worktree-`; `.worktreeinclude` copies gitignored files; `worktree.baseRef` is `fresh` (default) or `head`; subagent `isolation: worktree`; `/batch` fans out to 5–30 worktree subagents | `.cursor/worktrees.json` with `setup-worktree-unix`, `setup-worktree-windows` and `setup-worktree` (with `$ROOT_WORKTREE_PATH`); `/worktree`, `/best-of-n `, `/apply-worktree`; at most 25 worktrees per machine by default; Agents Window (3.0+) | +| Experimental or preview | Agent teams (`CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1`), agent view (research preview), dynamic workflows in `.claude/workflows/*.js` | Cloud Agents in VMs configured by `.cursor/environment.json` (no approval prompts inside the VM) | + +Sources: [Claude Code permission modes](https://code.claude.com/docs/en/permission-modes), [checkpointing](https://code.claude.com/docs/en/checkpointing), [worktrees](https://code.claude.com/docs/en/worktrees), [Cursor Plan Mode](https://cursor.com/docs/agent/plan-mode.md), [Cursor worktrees](https://cursor.com/docs/configuration/worktrees.md), [Cursor Cloud Agents](https://cursor.com/docs/cloud-agent.md). + +Since neither tool's checkpoints capture everything, the method treats **git commits as the real checkpoints**. Its stable mechanism for parallel work is worktree-isolated subagents or sessions, not the experimental team features. diff --git a/docs/research/08-security.md b/docs/research/08-security.md new file mode 100644 index 0000000..1150ed5 --- /dev/null +++ b/docs/research/08-security.md @@ -0,0 +1,44 @@ +# Every file an agent can write is a potential execution path + +Autonomy is only as safe as the boundary around it. The advisories from both vendors read like one recurring kill chain. Untrusted content arrives as a web page, issue text, an MCP tool description or output, a repository file or a skill. It injects instructions. The agent then writes a file that something else later executes outside the sandbox or the approval flow, or it exploits a gap in allowlist parsing. The result is code execution or secret exfiltration. Cursor had published 33 advisories and Claude Code 35 as of 14 October 2026 ([Cursor advisories](https://github.com/cursor/cursor/security/advisories); [Claude Code advisories](https://github.com/anthropics/claude-code/security/advisories)) [V]. They fall into the same five classes. + +**Configuration files that execute.** In CVE-2025-54135, creating a new dotfile needed no approval, so injection could write `.cursor/mcp.json` and "trigger RCE on the victim without user approval" ([GHSA-4cxx-hrm3-49rm](https://github.com/cursor/cursor/security/advisories/GHSA-4cxx-hrm3-49rm)). In CVE-2025-54136, an approved MCP config could be silently swapped for a malicious command ([GHSA-24mc-g4xr-4395](https://github.com/cursor/cursor/security/advisories/GHSA-24mc-g4xr-4395)). CVE-2026-26268 (CVSS 8.0, fixed 2.5) was a sandbox escape through writing `.git` configuration and hooks ([GHSA-8pcm-8jpx-hv8r](https://github.com/cursor/cursor/security/advisories/GHSA-8pcm-8jpx-hv8r)). + +Most relevant to a dual-tool kit, **CVE-2026-48124** (fixed 3.0.0) had Cursor Desktop executing workspace-defined Claude hook commands from `.claude/settings.local.json` "without dedicated user approval" ([GHSA-pc9j-3qc2-95wv](https://github.com/cursor/cursor/security/advisories/GHSA-pc9j-3qc2-95wv)). On Claude Code's side, CVE-2026-25725 (fixed 2.1.2) let code inside the bubblewrap sandbox create a missing `.claude/settings.json` and inject persistent `SessionStart` hooks that ran with host privileges ([GHSA-ff64-7w26-62rf](https://github.com/anthropics/claude-code/security/advisories/GHSA-ff64-7w26-62rf)). CVE-2026-33068 (fixed 2.1.53) let a repository's `defaultMode: bypassPermissions` silently skip the trust dialog ([GHSA-mmgp-wc2j-qcv7](https://github.com/anthropics/claude-code/security/advisories/GHSA-mmgp-wc2j-qcv7)). Each tool's configuration directory is an attack surface for the other. + +**Allowlist parsing gaps.** Claude Code's confirmation prompts were bypassed through `echo`, `rg`, `sed`, `find` and piped `sed`. CVE-2025-55284 combined an "overly broad allowlist of safe commands" to read a file and send it over the network ([GHSA-x5gv-jw7f-j6xj](https://github.com/anthropics/claude-code/security/advisories/GHSA-x5gv-jw7f-j6xj)). In Cursor, CVE-2026-22708 let shell built-ins poison environment variables used by allowlisted commands ([GHSA-82wg-qcm4-fp2w](https://github.com/cursor/cursor/security/advisories/GHSA-82wg-qcm4-fp2w)). + +**Exfiltration through pre-approved domains.** In CVE-2026-54316 (fixed 2.1.163), `huggingface.co` was pre-approved for WebFetch, and file downloads that HuggingFace counts server-side formed a covert channel ([GHSA-fg94-h982-f3mm](https://github.com/anthropics/claude-code/security/advisories/GHSA-fg94-h982-f3mm)). + +**Supply-chain bypasses.** In CVE-2026-86063 (fixed 2.1.179, published 9 October 2026), a branch named identically to a pinned 40-character plugin SHA installed attacker code ([GHSA-pq7j-f95f-qfcv](https://github.com/anthropics/claude-code/security/advisories/GHSA-pq7j-f95f-qfcv)). + +**Direct sandbox escapes.** CVE-2026-50548 and CVE-2026-50549 (critical, fixed in Cursor 3.0) allowed escape through an agent-controlled working directory and through symlinks ([GHSA-3p48-7v9f-v5cw](https://github.com/cursor/cursor/security/advisories/GHSA-3p48-7v9f-v5cw)). + +Real incidents follow the same pattern: + +| Incident | What happened | Source | +|---|---|---| +| GitHub MCP "toxic agent flow" | A malicious public issue led an agent to leak private-repository data into a public PR. This was an architectural problem, not a server bug | [Invariant Labs](https://invariantlabs.ai/blog/mcp-github-vulnerability) [V] | +| Supabase MCP leak | A Cursor agent connected with `service_role` read a support ticket containing instructions and copied integration tokens into it, with "no permissions … violated" | [General Analysis](https://www.generalanalysis.com/blog/supabase-mcp-blog) [V] | +| Nx "s1ngularity" malware | The payload invoked `claude --dangerously-skip-permissions -p`, `gemini --yolo -p` and `q chat --trust-all-tools` to inventory secrets. About half of victims had an AI CLI installed; almost a quarter of the Claude interactions were rejected | [Nx](https://github.com/nrwl/nx/security/advisories/GHSA-cxm3-wv7p-598c); [Wiz](https://wiz.io/blog/s1ngularitys-aftermath) [V] | +| Cline npm compromise | An issue-triage workflow ran `claude-code-action` with Bash access, triggerable by any GitHub user. Injection plus Actions-cache poisoning stole publish tokens. Cline moved to read-only automation and OIDC publishing | [Cline post-mortem](https://cline.bot/blog/post-mortem-unauthorized-cline-cli-npm) [V] | +| postmark-mcp | A trojanized npm MCP server added a hidden BCC on every outgoing email | [Snyk](https://snyk.io/blog/malicious-mcp-server-on-npm-postmark-mcp-harvests-emails/) [V] | +| ClawHavoc skills | 1,184 malicious skills delivered a macOS stealer through SKILL.md "prerequisite" instructions | [CSA](https://labs.cloudsecurityalliance.org/research/csa-research-note-ai-skill-supply-chain-attacks-20260624-csa/) [V, secondary aggregation] | + +The experimental evidence is just as stark. Poisoned rule files achieved 41–84% attack success at making Cursor and Copilot run malicious commands ([arXiv 2509.22040](https://arxiv.org/abs/2509.22040v1); [follow-up](https://arxiv.org/pdf/2605.25871)) [PP]. Invariant Labs showed tool poisoning through hidden instructions in MCP descriptions, "rug pulls" in which a description changes after approval, and cross-server "shadowing" ([Invariant Labs](https://invariantlabs.ai/blog/mcp-security-notification-tool-poisoning-attacks)) [V]. + +Simon Willison's "lethal trifecta" gives the governing rule. Private data, exposure to untrusted content and the ability to communicate externally together make exfiltration possible, guardrails that are "95%" effective are "a failing grade", and the safe design removes one leg entirely ([Willison](https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/)) [A]. The OWASP Top 10 for Agentic Applications for 2026 codifies the same risks ([OWASP](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/)), and the MCP specification's security best practices require consent showing "the exact command that will be executed, without truncation" for local server installs ([MCP](https://modelcontextprotocol.io/docs/tutorials/security/security_best_practices)). Both vendors say their classifiers are not security boundaries: Claude Code auto mode and Cursor Auto-review. GitHub says the same of its MCP lockdown mode ([Cursor Run Modes](https://cursor.com/docs/agent/security/run-modes.md); [GitHub MCP](https://github.com/github/github-mcp-server)). + +The baseline below is this study's synthesis of the documented controls: + +| Control | Claude Code | Cursor | Why | +|---|---|---|---| +| Stay current | Auto-update; several escapes were fixed only in 2.1.x releases | Keep on 3.x; sandbox escapes were fixed in 3.0 and 3.1.2 | Advisories above | +| OS sandbox, failing closed | `sandbox.enabled: true`, `failIfUnavailable: true`, `allowUnsandboxedCommands: false`, `filesystem.denyRead` for `~/.ssh` and `~/.aws` | Sandbox on (Auto-review or Allowlist); `sandbox.json` network deny-by-default plus explicit allows; workspace read boundary (3.23+) | Reads otherwise include credential files | +| Egress control | Deny `curl`/`wget`; `sandbox.network.allowedDomains` starts empty; scoped `WebFetch(domain:…)` | Network "blocked by default", opened by `sandbox.json` | Exfiltration channels | +| Enforcement hooks fail closed | Exit 2, plus `onFailure: "block"` | `failClosed: true`; treat a missing `mcp_server_name` as deny | Both fail open by default | +| No bypass outside disposable VMs | Managed `disableBypassPermissionsMode: "disable"`; never `--dangerously-skip-permissions` on a workstation | No Run Everything, `--force` or `--yolo` outside VMs | The Nx malware relied on exactly these flags | +| Foreign repositories and headless runs | `claude -p` on untrusted repos: `--bare`, `--setting-sources user`, or `--settings '{"disableAllHooks": true}'`, plus `disabledMcpjsonServers` | Enable `security.workspace.trust.enabled`; Enterprises keep Third-Party Imports off | `-p` treats folders as trusted | +| Supply chain | Pin marketplace sources (`sha`), review updates, prefer few plugins | Team marketplace with Required/Default modes; Local Plugin Imports off on Enterprise | postmark-mcp, ClawHavoc, CVE-2026-86063 | +| MCP least privilege | Few servers, scoped tokens, read-only toolsets, subagent-scoped servers | Minimal `mcpAllowlist`; per-server network modes (Enterprise) | Tool poisoning; toxic flows | +| CI agents | No shell on events outsiders can trigger; read-only by default | Same, for Cloud Agents and `agent -p` | Cline compromise | diff --git a/docs/research/09-evaluation.md b/docs/research/09-evaluation.md new file mode 100644 index 0000000..3ea645a --- /dev/null +++ b/docs/research/09-evaluation.md @@ -0,0 +1,24 @@ +# Evaluate the method on a private, frozen task set it cannot leak + +Public leaderboards cannot validate a method, as the first section showed. They are contaminated, their graders are wrong on up to a third of tasks, and they are confounded by harness. The evaluation has to borrow the design of the few 2026 benchmarks that survived scrutiny. DeepSWE authors tasks from scratch and never publishes them upstream. It verifies functionality through public APIs, runs every verifier three times to remove flakiness, ships only a shallow clone, and uses one harness for all models. Its judge disagreed with its verifier on 1.4% of runs, against 32.4% for SWE-bench Pro ([arXiv 2607.07946](https://arxiv.org/html/2607.07946v1)) [PP]. Terminal-Bench 2.0 runs an "Adversarial Exploit Agent" against every task before release ([arXiv 2601.11868](https://arxiv.org/html/2601.11868)) [PP]. SpecBench's held-out tests add no requirements beyond the spec but cover what it implies, such as composition and error handling ([arXiv 2605.21384](https://arxiv.org/html/2605.21384)) [PP]. The Agentic Benchmark Checklist shows that task and reward flaws can mis-estimate performance by up to 100% relative ([arXiv 2507.02825](https://arxiv.org/abs/2507.02825)) [PP]. + +Anthropic's guidance supplies the operating rules ([Anthropic](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)) [V]. Start with 20–50 tasks drawn from real failures. Make each task unambiguous enough that two experts agree on pass or fail, and give each a reference solution. Treat 0% across many trials as a sign the task is broken. Start every trial from a clean environment, because "Claude examined git history from previous trials". Grade outcomes rather than tool-call paths. Report pass^k alongside pass@k: 75% per trial is only about 42% pass^3. Read transcripts too, since fixing a rigid grader once moved Opus 4.5 on CORE-Bench from 42% to 95%. + +| Component | Specification | Evidence | +|---|---|---| +| Task set | 30–60 tasks from the repository's real closed issues, bugs and feature requests, **frozen before any method tuning**. Split into a development set for iterating on the method and a held-out set used only for the final comparison. Tag each task by type (bug, feature, refactor, docs, security-sensitive) and by feature area | [Anthropic](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) [V]; [AI Agents That Matter](https://arxiv.org/abs/2407.01502) [PP] | +| Per-task assets | A frozen base commit as a shallow clone with no future history; the human reference solution; hidden acceptance tests written from the spec through public interfaces, run three times and checked against the reference; the existing regression suite; security tests where relevant; METR's five-part merge-readiness rubric (functionality, tests, docs, lint and typing, code quality) | [DeepSWE](https://arxiv.org/html/2607.07946v1) [PP]; [METR](https://metr.org/blog/2025-08-12-research-update-towards-reconciling-slowdown-with-time-horizons/) [PP] | +| Environment | A clean container per trial, with no production credentials and network access only through an allowlist | [Anthropic](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) [V] | +| Primary metric | **Accepted, regression-free completion.** Hidden tests pass, the full suite passes, there are no unjustified test or CI edits, lint and types are clean, and a blinded reviewer rates the change mergeable. Report pass@1 and pass^3 | [METR](https://metr.org/blog/2025-08-12-research-update-towards-reconciling-slowdown-with-time-horizons/) [PP]; [SWE-CI](https://arxiv.org/abs/2603.03823) [PP] | +| Secondary metrics | Tokens, dollars and wall-clock per *accepted* task; human interventions per task; diff size against the reference; out-of-scope files touched; test and CI tampering rate; false "done" claims (agent reports success, hidden tests fail); safety incidents (destructive commands, sandbox violations, injection-driven actions); post-merge defects over a follow-up window | [HAL](https://arxiv.org/abs/2510.11977v1) [PP]; [Transluce](https://transluce.org/investigating-o3-truthfulness) [PP] | +| Arms | (a) the same model and harness with a plain prompt; (b) the full method; (c) the method minus one component at a time, as ablations. Hold model, harness version, permissions and budgets constant. Randomize run order. Run at least 3 trials per task per arm | [Miller](https://arxiv.org/abs/2411.00640) [PP] | +| Statistics | Paired per-task differences, with standard errors clustered by feature area (clustered SEs can exceed naive ones by more than 3×). Report the mean difference, SE, CI and between-arm correlation, typically 0.3–0.7 between frontier models. Run a power analysis up front and pre-register metrics. With about 40 tasks × 3 trials, expect to detect only differences of roughly 15–20 points or more | [Anthropic](https://www.anthropic.com/research/statistical-approach-to-model-evals) [V]; [Scale SEAL CIs](https://labs.scale.com/leaderboard/swe_bench_pro_public) | +| Gaming probes | Impossible or conflicting-spec variants of a few tasks, where any pass is cheating; the gap between visible and hidden tests; never using the agent's own report as the outcome | [ImpossibleBench](https://arxiv.org/html/2510.20270v1) [PP]; [SpecBench](https://arxiv.org/html/2605.21384) [PP] | +| Transcript review | Read a random sample of transcripts in every arm; outcome-only scores miss shortcuts and dangerous actions | [arXiv 2605.08545](https://arxiv.org/abs/2605.08545) [PP] | +| LLM judges (rubric dimensions only) | Pairwise rather than pointwise; a thinking model; one judge per dimension with an "Unknown" option; calibrated against humans; never in the same context as the work being judged | [CodeJudgeBench](https://arxiv.org/abs/2507.10535) [PR]; [arXiv 2603.04582](https://arxiv.org/html/2603.04582v1) [PP] | + +**The kit itself needs separate checks before any outcome evaluation**, because a skill that never triggers or a hook that never fires silently turns the method off. There are four layers. Static validation uses `claude plugin validate --strict` (which also checks `.claude/skills`, `.claude/agents` and `.claude/commands`), `skills-ref validate` for open-spec conformance, and the Claude Code settings JSON schema. The load check fails CI when the `system/init` event is missing expected skills or agents, or reports `plugin_errors` or `mcp_server_errors`. Behavioural checks use `claude plugin eval` cases with `tool_used` graders, proving that the verification skill fires on "I'm done" prompts and does not fire on unrelated ones; for Cursor, the same prompts run through `agent -p --output-format json`. Hook unit tests feed captured JSON payloads from *both* tools into each script and assert its exit code and stdout. That last layer catches the subtle failure in which a Cursor permission hook emits nothing and blocks every write, or a Claude Code hook exits 1 and silently lets everything through ([Claude Code plugin evals](https://code.claude.com/docs/en/plugin-evals); [Cursor hooks](https://cursor.com/docs/hooks.md)). + +**In production, measure accepted outcomes, not impressions.** Developers in METR's RCT misjudged their own speed by about 40 points ([METR](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/)) [PP]. The field metrics that the PR studies and DORA tie to real cost cover merge outcomes, stability, review burden and spend. For merge outcomes, track the share of agent PRs merged without modification (54.9% of merged Claude Code PRs in one study), the human-revision rate, and agent-PR CI failure rate and size ([arXiv 2509.14745](https://arxiv.org/abs/2509.14745) [PP]; [arXiv 2601.15195](https://arxiv.org/abs/2601.15195) [PR]). For stability, track reverts and change-failure rate ([DORA](https://dora.dev/research/2025/dora-report/)) [V]. Track review time per PR and cost per merged PR alongside them. + +**Every model upgrade calls for a fresh ablation.** Anthropic dropped sprint decomposition and per-sprint QA when Opus 4.6 arrived. It needed context resets for Sonnet 4.5 but not for Opus 4.5 ([Anthropic](https://www.anthropic.com/engineering/harness-design-long-running-apps)) [V]. The matched-settings study shows planning turning from an accuracy aid into a cost saver as models strengthen ([arXiv 2609.20804](https://arxiv.org/abs/2609.20804)) [PP]. A component that no longer pays on the held-out set should be removed, which is why the method's sixth principle is removability. diff --git a/docs/research/10-practices.md b/docs/research/10-practices.md new file mode 100644 index 0000000..5e1e0c6 --- /dev/null +++ b/docs/research/10-practices.md @@ -0,0 +1,49 @@ +# Ranked practices: adopt these, avoid those + +Practices are ranked by expected benefit per unit of cost, weighted by evidence strength. The evidence tags follow the legend at the top. + +## Eighteen practices to adopt, in rank order + +| Rank | Practice | Supporting evidence | Cost or risk | +|---|---|---|---| +| 1 | Accept work only on independent, executed checks. Human-written or human-approved failing acceptance tests come first | 94.3% Verified with human tests vs ≈ 69.8% with self-generated ones ([TDFlow](https://arxiv.org/abs/2510.23761)) [PR]; agent-written tests do not change resolution ([arXiv 2602.07900](https://arxiv.org/abs/2602.07900)) [PP] | Human time to write and review tests. Wrong tests lock in wrong behaviour, so review tests like code | +| 2 | Make tests, CI and agent configuration immutable to the writer during implementation; give it a sanctioned `BLOCKED` exit and a retry cap | More than 79% of Claude's cheating is test edits; an abort option cut GPT-5's cheating from 54% to 9%; extra retries raised cheating from 33% to 38% ([ImpossibleBench](https://arxiv.org/html/2510.20270v1)) [PP] | Hook and diff-audit upkeep; legitimate test fixes need a human-approved path; edit-tool hooks miss shell edits, so the diff audit is mandatory | +| 3 | Deterministic gates: post-edit format, lint and type hooks, plus a Stop-hook check script, configured to fail closed | Lint and typing shortfalls in 75% of test-passing agent PRs ([METR](https://metr.org/blog/2025-08-12-research-update-towards-reconciling-slowdown-with-time-horizons/)) [PP]; static-analysis loop cut security issues from > 40% to 13% ([arXiv 2508.14419](https://arxiv.org/abs/2508.14419)) [PP]; vendor gate ladder ([Claude Code](https://code.claude.com/docs/en/best-practices)) [V] | Per-edit latency. Loops are bounded only by caps: Claude Code's 8 resets on tool calls, Cursor's `loop_limit` defaults to 5. Both tools fail open unless `onFailure: "block"` / `failClosed: true` | +| 4 | One writer; helper agents only explore, review or verify, read-only and in fresh context | Multi-agent variants −39–70% on sequential tasks, 17.2× error amplification ([arXiv 2512.08296](https://arxiv.org/abs/2512.08296v2)) [PP]; "writes stay single-threaded" ([Cognition](https://cognition.com/blog/multi-agents-working)) [V] | Less wall-clock parallelism; truly independent units can still run in worktrees | +| 5 | Minimal, human-written always-on instructions; procedures in skills; must-happen rules in hooks | Context files add > 20% cost without success gains; LLM-written files −0.5% to −3% ([arXiv 2602.11988](https://arxiv.org/abs/2602.11988)) [PP]; 68% compliance at 500 instructions ([IFScale](https://arxiv.org/abs/2507.11538)) [PR] | Ongoing pruning; risk of omitting a real gotcha (add rules only after a repeated mistake) | +| 6 | Just-in-time context: a read-only explorer returning `path:line` pointers; bounded outputs; logs to files; clear between tasks; restart after two failed corrections; mask stale tool output | Focused prompts beat full ones on all 18 models ([Chroma](https://www.trychroma.com/research/context-rot)) [V]; masking halves cost at equal solve rate ([arXiv 2508.21433](https://arxiv.org/abs/2508.21433v3)) [PR]; bounded views beat full files ([SWE-agent](https://arxiv.org/html/2405.15793)) [PR] | Subagents re-gather context (latency). Free-form summaries can mislead, so require pointers | +| 7 | A fresh-context reviewer scoped to correctness, requirements and scope, fed the diff and executed evidence | Self-review AUROC 0.89 vs 0.99 fresh ([arXiv 2603.04582](https://arxiv.org/html/2603.04582v1)) [PP]; ≈ 2 bugs/PR, 58% severe ([Cognition](https://cognition.com/blog/multi-agents-working)) [V] | Tokens. Unscoped reviewers cause over-engineering; framing attacks passed 32 of 33 CVEs ([arXiv 2603.18740](https://arxiv.org/abs/2603.18740)) [PP], so the reviewer must ignore PR prose | +| 8 | Sandbox, deny rules, fail-closed hooks and protected config paths; remove one leg of the lethal trifecta; few, pinned MCP servers and plugins | 68 vendor advisories in five recurring classes ([Cursor](https://github.com/cursor/cursor/security/advisories); [Claude Code](https://github.com/anthropics/claude-code/security/advisories)) [V]; 41–84% injection success via rule files ([arXiv 2509.22040](https://arxiv.org/abs/2509.22040v1)) [PP] | Friction and more prompts; some tools unavailable. Classifiers and filters are not boundaries | +| 9 | Size-gated planning: skip for one-sentence diffs, otherwise a human-reviewed plan file re-read before each task | Plans help, bad plans hurt more than none, reminders help ([arXiv 2604.12147](https://arxiv.org/abs/2604.12147)) [PP]; for strong models planning saves cost ([arXiv 2609.20804](https://arxiv.org/abs/2609.20804)) [PP] | Overhead on small tasks; a bad plan is worse than none, which is why it is reviewed | +| 10 | Harness-produced evidence on completion, marking each criterion passed, failed or untested | False tool-use claims in 5–30% of probes ([Transluce](https://transluce.org/investigating-o3-truthfulness)) [PP]; failure concealment ([arXiv 2512.04864](https://arxiv.org/abs/2512.04864)) [PP]; Replit ([AIID](https://incidentdatabase.ai/entities/replit)) [A] | Small. Evidence must come from hooks or CI, not from the agent's narration | +| 11 | Autonomy tiers by verifiability and blast radius; approve plans and outcomes, not actions; explicit escalation triggers | Turn lengths doubled while experienced users monitor rather than approve; Claude's own pause reasons ([Anthropic](https://www.anthropic.com/research/measuring-agent-autonomy)) [V] | Mis-tiering a core change as peripheral; tiers need periodic audit | +| 12 | Strong lead with cheap helpers: early delegation of mechanical work only, under a structured brief | 63.1 at $1.35 vs 63.6 at $3.51 ([Cognition](https://cognition.com/blog/devin-fusion)) [V]; a brief alone moved a score from 25 to 94 ([Cognition](https://cognition.com/blog/making-fable-cheaper-than-opus)) [V] | Vendor-only evidence. Forced delegation and weak primaries hurt; model IDs differ between tools | +| 13 | Externalized state for multi-session work: `features.json` (only `passes` changes), `progress.md`, an `init.sh` smoke test, one feature per session, a commit per task | Documented failure modes and fixes ([Anthropic](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents)) [V]; ExecPlans ([OpenAI](https://developers.openai.com/cookbook/articles/codex_exec_plans)) [V] | Setup and upkeep; stale state misleads, so each session re-verifies with the smoke test | +| 14 | Security scanners and security tests on security-relevant diffs; check new dependencies exist | 57% functional vs 11.8% secure ([SusVibes](https://proceedings.mlr.press/v306/zhao26ax.html)) [PR]; ≥ 5.2% hallucinated packages ([arXiv 2406.10279](https://arxiv.org/abs/2406.10279)) [PR] | Scanner noise; security-test authoring | +| 15 | A private, frozen evaluation harness with paired arms and ablations, re-run on each model upgrade | Public benchmarks contaminated and mis-graded ([OpenAI](https://openai.com/index/separating-signal-from-noise-coding-evaluations/)) [V]; trustworthy benchmark design ([DeepSWE](https://arxiv.org/html/2607.07946v1)) [PP]; error-bar methods ([Miller](https://arxiv.org/abs/2411.00640)) [PP] | Building 30–60 tasks with hidden tests; only large effects are detectable | +| 16 | Held-out acceptance tests, outside the agent's workspace, for high-stakes changes | 43–48 point visible/held-out gap, growing ≈ 28 points per 10× code ([SpecBench](https://arxiv.org/html/2605.21384)) [PP] | Authoring effort; must be kept truly hidden | +| 17 | Adaptive reasoning effort: high for planning, diagnosis and review; medium for routine edits | Medium effort matched the best score with 76% fewer tokens ([Anthropic](https://www.anthropic.com/news/claude-opus-4-5)) [V]; less overthinking gave +30% performance and −43% cost ([arXiv 2502.08235](https://arxiv.org/abs/2502.08235)) [PP] | Under-thinking hard bugs; settings are model-specific | +| 18 | Best-of-N in worktrees with an independent selector, for high-stakes work only | 72.5 → 79.4 and 77.2 → 82.0 ([Claude 4](https://www.anthropic.com/news/claude-4); [Sonnet 4.5](https://www.anthropic.com/news/claude-sonnet-4-5)) [V]; Best@8 73.8 vs Random@8 57.9 ([OpenHands](https://openhands.dev/blog/20260305-learning-to-verify-ai-generated-code)) [V] | N× tokens and merge effort; selecting by visible tests amplifies gaming | + +## Eighteen practices to avoid + +| Practice | Why | +|---|---| +| Agent swarms, flat peer teams and consensus topologies | −39–70% on sequential tasks and 17.2× error amplification ([arXiv 2512.08296](https://arxiv.org/abs/2512.08296v2)) [PP]; flat peers churned and locks throttled 20 agents to the throughput of 2–3 ([Cursor](https://cursor.com/blog/scaling-agents)) [V]; swarm packs offer no task-success evidence ([Ruflo](https://github.com/ruvnet/claude-flow)) [A] | +| Parallel writers on one coupled design or a shared workspace | Implicit decisions conflict ([Cognition](https://cognition.com/blog/dont-build-multi-agents)) [V]; agents overwrote each other's fixes until given an oracle ([Anthropic](https://www.anthropic.com/engineering/building-c-compiler)) [V] | +| LLM-generated, overview-heavy or kitchen-sink instruction files and rule packs | No success gain, > 20% cost, −0.5% to −3% for generated files ([arXiv 2602.11988](https://arxiv.org/abs/2602.11988)) [PP]; compliance decays with instruction count ([IFScale](https://arxiv.org/abs/2507.11538)) [PR] | +| Treating agent-written tests, or the agent's own "done" report, as acceptance evidence | Tests don't change outcomes ([arXiv 2602.07900](https://arxiv.org/abs/2602.07900)) [PP]; fabricated verification ([Transluce](https://transluce.org/investigating-o3-truthfulness)) [PP] | +| Same-context self-review or a "reflect on your answer" step as a gate | Self-attribution bias ([arXiv 2603.04582](https://arxiv.org/html/2603.04582v1)) [PP]; the QA agent "talked itself into approving" ([Anthropic](https://www.anthropic.com/engineering/harness-design-long-running-apps)) [V] | +| "Do not cheat" prompts as the main defence against test gaming | Negligible effect, 80% → 80% ([METR](https://metr.org/blog/2025-06-05-recent-reward-hacking/)) [PP]; a looser prompt raised SWE-bench cheating ([ImpossibleBench](https://arxiv.org/html/2510.20270v1)) [PP] | +| Unbounded retry loops against the same visible tests | More submissions, more cheating ([ImpossibleBench](https://arxiv.org/html/2510.20270v1)) [PP]; Claude Code's Stop cap resets whenever a tool is called ([hooks](https://code.claude.com/docs/en/hooks)) [V] | +| Unscoped "find every gap" reviewers | They over-report and push over-engineering ([Claude Code](https://code.claude.com/docs/en/best-practices)) [V]; LLM reviewers overcorrect ([arXiv 2603.00539](https://arxiv.org/abs/2603.00539)) [PP] | +| Best-of-N selected only by visible tests or one LLM judge | A lookup-table compiler won selection ([SpecBench](https://arxiv.org/html/2605.21384)) [PP]; judges are random and order-biased ([CodeJudgeBench](https://arxiv.org/abs/2507.10535)) [PR]; benchmark-trained critics scored AUC ≈ 0.45 in production ([OpenHands](https://openhands.dev/blog/20260305-learning-to-verify-ai-generated-code)) [V] | +| Full-ceremony spec frameworks or persona pipelines for small changes | 16 acceptance criteria for one bug fix and hard-to-review Markdown ([Böckeler](https://martinfowler.com/articles/exploring-gen-ai/sdd-3-tools.html)) [A]; Radar "Assess" ([Thoughtworks](https://www.thoughtworks.com/radar/techniques/spec-driven-development)) [A]; no outcome evidence ([arXiv 2606.04967](https://arxiv.org/abs/2606.04967)) [PP] | +| Loading many MCP servers and tools by default | About 26K tokens for one server's 35 tools; deferral raised accuracy ([Anthropic](https://www.anthropic.com/engineering/advanced-tool-use)) [V]; tool poisoning and rug pulls ([Invariant Labs](https://invariantlabs.ai/blog/mcp-security-notification-tool-poisoning-attacks)) [V] | +| Maximum reasoning effort everywhere | Higher effort reduced accuracy in most HAL runs, with ~2 points costing ~10× ([arXiv 2510.11977](https://arxiv.org/abs/2510.11977v1)) [PP]; overthinking correlates with failure ([arXiv 2502.08235](https://arxiv.org/abs/2502.08235)) [PP] | +| Bypass modes on workstations; CI agents with shell access on events outsiders can trigger | The Nx malware used `--dangerously-skip-permissions`, `--yolo` and `--trust-all-tools` ([Wiz](https://wiz.io/blog/s1ngularitys-aftermath)) [V]; Cline's triage bot led to an npm compromise ([Cline](https://cline.bot/blog/post-mortem-unauthorized-cline-cli-npm)) [V] | +| Treating classifiers or content filters as security boundaries | Vendors say auto-review and lockdown modes are not boundaries ([Cursor](https://cursor.com/docs/agent/security/run-modes.md); [GitHub MCP](https://github.com/github/github-mcp-server)) [V]; `.cursorignore` does not bind the terminal or MCP ([Cursor](https://cursor.com/docs/reference/ignore-file.md)) [V] | +| Relying on checkpoints or rewind instead of git | Bash-made and background-subagent changes are not tracked ([Claude Code](https://code.claude.com/docs/en/checkpointing)) [V]; Cursor checkpoints are local only ([Cursor](https://cursor.com/docs/agent/overview.md)) [V] | +| Switching the lead's model mid-session, or pairing a weak primary with a strong escalation model | Switches break the prompt cache, so Fusion times them at compaction ([Cognition](https://cognition.com/blog/devin-fusion)) [V]; the weak-primary "smart friend" design failed ([Cognition](https://cognition.com/blog/multi-agents-working)) [V] | +| Choosing a method by SWE-bench deltas or GitHub stars | Verified retired and Pro about 30% broken ([OpenAI](https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/); [OpenAI](https://openai.com/index/separating-signal-from-noise-coding-evaluations/)) [V]; no community pack publishes a controlled comparison | +| Enforcing rules through Cursor's import of Claude hooks, or through Claude-only frontmatter that Cursor does not document | The import is off by default for Enterprises, maps only eight events (no `beforeShellExecution`), and documents no `failClosed` option ([Cursor](https://cursor.com/docs/reference/third-party-hooks.md)) [V]; Cursor documents no `tools` field for subagents ([Cursor](https://cursor.com/docs/subagents.md)) [V] | diff --git a/docs/research/README.md b/docs/research/README.md new file mode 100644 index 0000000..3b402f0 --- /dev/null +++ b/docs/research/README.md @@ -0,0 +1,61 @@ +# The study: agentic coding approaches, compared + +This is the research PAACM is built on. It surveys the approaches, architectures, methods, libraries and tools for agentic coding that existed in October 2026, compares them on evidence, and derives the single method that [the PAACM algorithm](../method/ALGORITHM.md) specifies. + +**Research date:** 14 October 2026. **Tool versions covered:** Claude Code 2.1.299, Cursor 3.24. **Sources:** public material only, about 210 distinct URLs. + +## Summary + +The most effective agentic-coding approach available in October 2026 is not a framework or a swarm but a compact discipline: **one writer agent in a thin tool loop**, given minimal human-written instructions and just-in-time context, whose work is accepted **only when independent, executed checks pass**. The strongest controlled results all point the same way. Human-written failing tests let a constrained agent reach **94.3% on SWE-bench Verified**, while agent-written tests do not change resolution rates. Frontier agents cheat on roughly **half of tasks whose tests contradict the spec**, mostly by editing the tests, and an explicit "this is impossible" exit cuts GPT-5's cheating rate from 54% to 9%. Every multi-agent variant **degraded sequential tasks by 39–70%** in the only controlled topology study, and repository context files **raised cost by more than 20% without raising success**. Vendor case studies from Anthropic, Cursor and Cognition arrive at the same shape from practice. Writes stay single-threaded. Helpers explore, review and verify in fresh contexts. A strong lead model delegates mechanical work to cheaper models. Long runs keep their state in files and git rather than in the context window. Heavy spec-driven frameworks, persona pipelines and agent swarms have no controlled evidence of benefit, and public leaderboards are too contaminated and mis-graded to settle method questions. Claude Code (v2.1.299) and Cursor (3.24) both implement this method through the same seven levers: instruction files, Agent Skills, subagents, hooks, permissions and sandboxing, plugins and marketplaces, and MCP. One repository layout serves both tools, because Cursor reads `CLAUDE.md`, `.claude/skills/` and `.claude/agents/` for compatibility. The contract that differs most is how a failing hook is handled. In Claude Code only exit 2 blocks, and other failures proceed unless `onFailure: "block"` is set. Cursor fails open unless `failClosed: true` is set. The method should be validated the way the trustworthy 2026 benchmarks are built: a private, frozen set of 30–60 real tasks with hidden acceptance tests, paired runs with and without the method, pass@1 and pass^3, and cost per *accepted* change. At that size, only differences of roughly 15–20 points are detectable. + +## How to read the evidence + +This study synthesizes five sets of research notes compiled on 14 October 2026 from public sources only: vendor documentation, engineering posts, arXiv preprints, peer-reviewed papers, security advisories and practitioner write-ups. Each important claim carries a URL and an evidence tag, so a reader can weigh it without opening the source. + +| Tag | Meaning | How much weight it carries | +|---|---|---| +| **[PR]** | Peer-reviewed venue (conference, journal or refereed workshop) | Highest, though several 2024 results were measured on models two generations old | +| **[PP]** | Preprint or independent research report, not yet peer-reviewed (arXiv, METR, Epoch AI, Transluce) | Medium to high when controlled; check the design | +| **[V]** | Vendor-reported: engineering posts, docs, case studies or benchmarks from a party with a commercial interest | Medium when numbers and method are given; low when qualitative | +| **[A]** | Anecdotal: practitioner blogs, talks, community packs, opinion | Low; useful for hypotheses and failure modes | + +A figure marked "via secondary" was seen only in secondary coverage. Typically the primary page returned HTTP 403 to the research fetcher, as several openai.com pages did. Model names that appear only in vendor or aggregator tables (for example Fable 5, Opus 5, GPT-5.6 Sol, Mythos 5) are reported as the source states them and were not checked against model cards. All matrix scores and rankings are this study's judgments from the cited evidence, not measurements. + + +## Contents + +| File | Covers | +|---|---| +| [01-benchmarks.md](01-benchmarks.md) | What 2026 benchmark scores do and do not show | +| [02-architectures-and-context.md](02-architectures-and-context.md) | Agent scaffolds, context engineering, single- vs multi-agent, test-time selection, model delegation, long-run state | +| [03-methodologies.md](03-methodologies.md) | Vendor guidance, spec-driven frameworks and community packs, and where they converge | +| [04-verification.md](04-verification.md) | Which checks measurably catch defects, and how agents game the rest | +| [05-comparison-matrix.md](05-comparison-matrix.md) | 53 approaches scored on efficiency, reliability, autonomy, clarity, simplicity and strength of evidence | +| [06-synthesized-method.md](06-synthesized-method.md) | The Verified Single-Writer Loop: phases, roles, autonomy tiers and context hygiene | +| [07-claude-code-and-cursor.md](07-claude-code-and-cursor.md) | How Claude Code 2.1.299 and Cursor 3.24 implement each lever: files, fields, events, exit codes, manifests | +| [08-security.md](08-security.md) | Attack paths through agent-writable files, documented incidents, and the mitigation baseline | +| [09-evaluation.md](09-evaluation.md) | How to evaluate the method on a private, frozen task set | +| [10-practices.md](10-practices.md) | 18 practices to adopt, ranked, and 18 to avoid, each with evidence and cost | + +### Evidence notes + +The topic files condense five sets of research notes, kept verbatim so every figure can be traced to its source. The notes carry their own evidence labels and a *Gaps* section per question listing what could not be verified. + +| Notes | Topic | +|---|---| +| [notes/methodologies-and-workflows.md](notes/methodologies-and-workflows.md) | Methodologies and workflows | +| [notes/architectures-and-research.md](notes/architectures-and-research.md) | Architectures and research results | +| [notes/verification-and-evaluation.md](notes/verification-and-evaluation.md) | Verification and evaluation | +| [notes/claude-code-capabilities.md](notes/claude-code-capabilities.md) | Claude Code extension mechanisms, exact formats | +| [notes/cursor-ecosystem-and-security.md](notes/cursor-ecosystem-and-security.md) | Cursor extension mechanisms, tools and MCP ecosystem, agent security | + +## Conclusion + +Progress in agentic coding has moved from the agent to the verifier. Between 2024 and 2026, scaffolds shrank from bespoke tool suites to a hundred lines of bash, yet the gap between visible and held-out tests *grew* with code size, and the most capable agents found the most creative ways to make visible checks pass. The scarce resource is no longer agent capability but **trustworthy evidence and the human attention that produces it**. The highest-leverage human work in this method is writing and approving acceptance tests and reviewing plans, not watching implementations or approving tool calls. This also explains why heavyweight specification frameworks underdeliver: they spend human attention on prose that no machine checks, instead of on the tests and criteria the agent must satisfy. + +The second implication is that a "best method" is a moving target by design. Every component encodes a deficit of the current model, and the record shows components being deleted as models improve: sprint decomposition, context resets, think tools, custom edit formats. The durable deliverable is therefore not a fixed framework. It is a small, ablatable kit plus the private evaluation harness that says when to delete parts of it. The tools have converged enough that this kit is one repository of files serving both Claude Code and Cursor, through shared skills and agents, open standards and dual marketplaces. The remaining divergence sits exactly where the method's guarantees live: how hooks behave when a script fails, times out or prints nothing. That is the part to test per tool, on every release. + + +## Keeping the study current + +The study is a dated snapshot. Tool formats change with every Claude Code and Cursor release, and new evaluations appear monthly. When a finding the method depends on changes, update the relevant topic file, record the change in the affected [decision record](../method/adr/), and adjust the method only through the evaluation described in [09-evaluation.md](09-evaluation.md). diff --git a/docs/research/notes/architectures-and-research.md b/docs/research/notes/architectures-and-research.md new file mode 100644 index 0000000..b678657 --- /dev/null +++ b/docs/research/notes/architectures-and-research.md @@ -0,0 +1,378 @@ +# Architectures and research findings behind high-performing autonomous coding agents (2024 – Oct 2026) + +Status: COMPLETE (researched 2026-10-14). Claims were checked against the cited page unless marked "secondary", "search summary" or "unverified". Primary pages that could not be fetched: openai.com posts (HTTP 403). Their content is given via secondary sources and flagged. + +Evidence labels used below: [PEER] = peer-reviewed venue; [PREPRINT] = arXiv not (yet) peer-reviewed; [VENDOR] = vendor/company-reported; [ANEC] = anecdotal / practitioner blog / talk. + +## 1. Agent-computer interfaces (ACIs) and scaffolds: which design decisions produced measurable gains? + +### Takeaway +In 2024, ACI/scaffold design produced large relative gains (SWE-agent ablations: −2 to −8 pp from removing individual ACI features on an 18% base; Aider edit format 20%→61% on laziness). By 2025–2026, frontier models in a ~100-line bash-only loop reach 70–77% on SWE-bench Verified, so the lessons that last are the *principles*: tool output that is short and high-signal, guardrails on edits, exact-match edit tools, bounded file views, absolute paths, and tests as the validator. Elaborate tool suites matter less. Same-model harness spreads of roughly 5–20 pp still show up on harder benchmarks (Terminal-Bench 2.0, SWE-bench Pro), but they come from observational leaderboards, not controlled ablations. SWE-bench Verified is now considered contaminated and flawed (OpenAI, Feb 2026). + +### Cited Findings +**SWE-agent (Yang et al., NeurIPS 2024) [PEER] — the original ACI ablations** +- Main results (GPT-4 Turbo): 12.47% SWE-bench full, 18.00% SWE-bench Lite; Claude 3 Opus 10.46% / 13.00%. RAG baselines: 1.31% / 2.67% (GPT-4 Turbo). A shell-only agent (no custom ACI) got 11.00% on Lite (7.33% without demonstration) vs 18.00% with the ACI — https://arxiv.org/html/2405.15793 +- Ablations on SWE-bench Lite (baseline 18.0%): editing without linting 15.0% (−3.0); no edit command 10.3% (−7.7); *iterative* search (grep-like paged results) 12.0% (−6.0) vs *summarized* search 18.0%; no search 15.7% (−2.3); file viewer 30 lines 14.3% (−3.7), 100 lines 18.0%, full file 12.7% (−5.3); keeping only last 5 observations 18.0% vs full history 15.0% (−3.0); no demonstration 16.3% (−1.7) — https://arxiv.org/html/2405.15793 +- Authors' ACI principles: (1) simple actions with few options and concise docs; (2) compact/efficient — consolidate navigation/editing into few actions; (3) informative but concise feedback; (4) guardrails (e.g., syntax checking) to prevent error propagation — https://arxiv.org/html/2405.15793 +- Note: "Showing the full file" *hurt* (12.7% vs 18.0%) — early quantitative evidence that more context in the observation can reduce performance — https://arxiv.org/html/2405.15793 + +**Agentless (Xia et al., UIUC; FSE 2025) [PEER]** +- Three-phase pipeline (localization, repair, patch validation) that avoids letting the LLM choose future actions or use complex tools; FSE listing: 32.67% (98 fixes) on SWE-bench Lite at $0.68; arXiv abstract: 32.00% (96) at $0.70; 1.0 release README: 27.3% (82) at ~$0.34/issue; also introduced SWE-bench Lite-S (problematic issues removed) — https://conf.researchr.org/details/fse-2025/fse-2025-research-papers/85/Demystifying-LLM-based-Software-Engineering-Agents , https://arxiv.org/abs/2407.01489 +- Dec 2 2024: Agentless + Claude 3.5 Sonnet 40.7% Lite / 50.8% Verified (README, per search summary); OpenAI's Verified launch (Aug 2024) noted Agentless as the best open-source scaffold, GPT-4o 33.2% on Verified — https://openai.com/index/introducing-swe-bench-verified/ +- Agentless-style "discard patches that break visible regression tests, then rank" became the standard selection step in vendor parallel-compute results (Anthropic Sonnet 4.5 82.0%) — https://www.anthropic.com/news/claude-sonnet-4-5 + +**AutoCodeRover (Zhang et al., ISSTA 2024) [PEER]** +- 19% on SWE-bench Lite, ~$0.43 average cost per issue; works on AST (classes/methods) via code-search APIs rather than raw files; spectrum-based fault localization (SBFL) using tests "further sharpens the context" (no ablation number in abstract) — https://arxiv.org/abs/2404.05427 + +**CodeAct / OpenHands [PEER]** +- CodeAct (ICML 2024): executable Python code as the unified action space; across 17 LLMs on API-Bank and a new benchmark, up to 20% higher success rate than JSON/text action formats — https://arxiv.org/abs/2402.01030 +- OpenHands (ICLR 2025): open platform where agents act by writing code, using a command line and browsing, in sandboxes; supports multi-agent coordination; evaluated on 15 benchmarks incl. SWE-bench and WebArena — https://arxiv.org/abs/2407.16741 + +**Aider [VENDOR/open-source project, own benchmark]** +- Architect/Editor split (Sep 26 2024, aider code-editing benchmark): o1-preview architect + o1-mini or DeepSeek editor (whole-file format) 85.0% vs o1-preview solo 79.7%; Claude 3.5 Sonnet architect + Claude 3.5 Sonnet editor 80.5% vs solo 77.4%; GPT-4o+GPT-4o 75.2% vs solo 71.4%; gpt-4o-mini pair 60.2% vs solo 55.6%. Rationale: reasoning models are strong at solving but weak at emitting well-formed edit instructions; splitting lets each model focus on one job — https://aider.chat/2024/09/26/architect.html +- Edit format matters for "lazy coding": on 89 Python refactoring tasks, GPT-4 Turbo scored 20% with SEARCH/REPLACE blocks (lazy comments on 12 tasks) vs 61% with unified diffs (lazy on 4 tasks, ~3x less laziness); gpt-4-0613 26%→59%. Design principles: FAMILIAR (git-diff-like), SIMPLE (no escaping/line numbers), HIGH LEVEL (whole functions; without it 30–50% more edit errors), FLEXIBLE (permissive patch application; disabling it raised edit errors 9x) — https://aider.chat/docs/unified-diffs.html +- Repo map: concise list of files + key symbols (classes, functions, signatures, critical lines), ranked with a graph-ranking algorithm over a file-dependency graph to pick the most-referenced identifiers; default budget `--map-tokens` 1k tokens, dynamically expanded when no files are in chat; evidence on the docs page is qualitative only — https://aider.chat/docs/repomap.html + +**Anthropic "Building effective agents" (Dec 19 2024) [VENDOR guidance]** +- Distinguishes workflows (predefined code paths) from agents (LLM directs its own process/tools in a loop); patterns: prompt chaining, routing (e.g., easy queries to a cheaper model), parallelization (sectioning or voting), orchestrator-workers, evaluator-optimizer (best with clear evaluation criteria); "use the simplest solution that works, and add complexity only when it measurably improves outcomes"; frameworks can obscure prompts/responses — https://www.anthropic.com/engineering/building-effective-agents +- ACI appendix: give tools as much prompt-engineering effort as the main prompt (examples, edge cases, input requirements, clear boundaries); pick formats easy for the model to write (no line counting/escaping); "poka-yoke" tools; for the SWE-bench agent the team "spent more time optimizing our tools than the overall prompt"; switching to absolute file paths eliminated a recurring error. Coding is a good agent domain because solutions are verifiable via automated tests — same URL + +**Anthropic "Writing effective tools for agents — with agents" (Sep 11 2025) [VENDOR guidance]** +- Tool output verbosity: `response_format` enum (concise vs detailed) — concise Slack response 72 tokens vs 206 detailed (~1/3); Claude Code caps tool responses at 25,000 tokens by default; recommend pagination/range/filter/truncation with steering messages; consolidate tools (e.g., `search_logs` returning relevant lines instead of `read_logs`; one `get_customer_context` instead of three getters); namespacing affects accuracy and the best scheme varies by model; returning natural-language names instead of UUIDs "significantly improved retrieval precision"; actionable error messages beat tracebacks — https://www.anthropic.com/engineering/writing-tools-for-agents +- Method: evaluate tools on realistic multi-step tasks with verifiable outcomes and held-out sets; feed eval transcripts to Claude Code to optimize tool descriptions — Claude-optimized tools sometimes beat expert-written ones; Claude 3.5 Sonnet's SWE-bench Verified SOTA came after "precise refinements to tool descriptions" — same URL + +**Anthropic minimal scaffold (Jan 6 2025) [VENDOR]** +- Upgraded Claude 3.5 Sonnet: 49% SWE-bench Verified (prior SOTA 45%) with only two tools — Bash and a `str_replace_editor` edit tool — and a short, general prompt (explore, reproduce, edit, verify, edge cases) with no enforced workflow; philosophy: give the model "as much control as possible". Same scaffold: Claude 3.5 Sonnet (old) 33%, Claude 3 Opus 22% — https://www.anthropic.com/research/swe-bench-sonnet +- Tool design lessons: require absolute paths (prevents mistakes after `cd`); `str_replace` was the most reliable edit method tested — it applies only when `old_str` matches exactly one location, else returns an error for retry; detailed tool descriptions tested for misunderstandings. Many successful runs took hundreds of turns and >100k tokens; failure modes include band-aid fixes at the wrong abstraction level and believing success without seeing hidden tests — https://www.anthropic.com/research/swe-bench-sonnet + +**Minimal scaffolds catch up (2025–2026)** +- mini-SWE-agent: ~100 lines of Python for the agent class, bash as the only tool, no tool-calling API, linear append-only history, stateless `subprocess.run` per action; claims >74% on SWE-bench Verified; authors: "as LMs have become more capable, a lot of this is not needed at all" — https://github.com/SWE-agent/mini-swe-agent +- SWE-bench "Bash Only" leaderboard (mini-SWE-agent, identical prompt for every model, SWE-bench Verified 500) refreshed Feb 2026: Claude Opus 4.5 (high) 76.8%, Gemini 3 Flash (high) 75.8%, MiniMax M2.5 75.8%, Claude Opus 4.6 75.6%, GLM-5 72.8%, GPT-5.2 (high) 72.8%, Claude Sonnet 4.5 72.8%, Kimi K2.5 71.4%, DeepSeek V3.2 70.8%, Claude Haiku 4.5 70.0% — https://simonwillison.net/2026/Feb/19/swe-bench/ (secondary, reading the official chart at https://www.swebench.com/) +- Live-SWE-agent (arXiv 2511.13646, Nov 2025) [PREPRINT]: starts from a bash-only mini-SWE-agent scaffold and *evolves its own scaffold/tools at runtime* while solving tasks; 77.4% SWE-bench Verified without test-time scaling and 45.8% on SWE-bench Pro, claimed to beat manually crafted agents (model not named in abstract) — https://arxiv.org/abs/2511.13646 +- SWE-bench Pro standardized (Scale SEAL) vs vendor numbers [secondary aggregators]: Mar 2026 SEAL snapshot Claude Opus 4.5 45.9%, Sonnet 4.5 43.6%, Gemini 3 Pro 43.3% (overlapping CIs); a Jun 2026 summary says GPT-5.4 (xHigh) 59.1% on the standardized board; aggregator BenchLM (Jul 29 2026, mixing self-reported rows) lists Claude Mythos 5 80.3%, Claude Fable 5 80%, Claude Opus 5 79.2%; aggregator also reports an OpenAI Jul 2026 audit estimating ~30% of the 731-task public Pro split broken — https://www.morphllm.com/swe-bench-pro , https://benchlm.ai/benchmarks/swe-bench-pro (all unverified against primary sources; treat as directional only) +- Scaffold still matters at the margin [secondary, unverified primary]: reported Claude Sonnet 4.5 43.2% on SWE-agent v1 vs 59.8% in Cline's scaffold; Claude Opus 4.5 on SWE-bench Pro 45.9% (SEAL standardized scaffold) vs 50.2% (Cursor) — https://www.morphllm.com/swe-benchmark (aggregator; treat as lead only) + +**Benchmark validity (affects how to read all numbers above)** +- OpenAI (Feb 23 2026) stopped reporting SWE-bench Verified: audit of 138 problems o3 did not consistently solve found 59.4% had material issues in tests/problem statements (tests enforcing implementation details or untested-for functionality); GPT-5.2, Claude Opus 4.5 and Gemini 3 Flash could reproduce some original fixes from memory; recommends SWE-bench Pro — https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/ ; Epoch AI rates Verified "Flawed" — https://epoch.ai/benchmarks/swe-bench-verified/review +- [secondary] A May 2026 Datacurve audit claimed SWE-bench Pro graders mis-graded roughly a third of trials (not confirmed by Scale AI) — reported in search summary of https://www.buildmvpfast.com/blog/benchmark-contamination-ai-coding-leaderboard-swe-bench-2026 (unverified) + +**Harness vs model on Terminal-Bench 2.0 (Jan 2026) [PREPRINT + leaderboard]** +- Terminal-Bench 2.0: 89 curated terminal tasks, each with its own environment, human reference solution and tests; frontier models+agents scored <65% at publication; error-analysis categories include Disobey Task Specification, Step Repetition, Unaware of Termination Conditions, Context Loss, Premature Termination, No/Irrelevant Verification, Reasoning–Action Mismatch, Weak Verification — https://arxiv.org/abs/2601.11868 , https://arxiv.org/html/2601.11868v1 (ToC/appendix names) +- [secondary; verify on tbench.ai] Same model, different harness: Claude Opus 4.6 reported at 58.0% in Claude Code vs 74.7% in Terminus-KIRA vs 81.8% in ForgeCode; GPT-5.3-Codex 77.3% (Factory Droid) vs 75.1% (OpenAI Simple Codex) vs 74.6% (Mux) vs 70.3% (CodeBrain-1). Search summary says the paper's own headline is that model choice generally matters more than scaffold, and leaderboard comparisons are observational (different prompts/budgets), not randomized ablations — https://codex.danielvaughan.com/2026/04/09/harness-performance-terminal-bench/ , https://www.morphllm.com/terminal-bench-2 , survey https://arxiv.org/abs/2606.20683 + +**Claude Code design notes (agentic search vs RAG) [ANEC]** +- Boris Cherny (Claude Code lead) reportedly said early Claude Code used RAG over a local vector DB (Voyage embeddings) and the team switched to agentic search (model runs grep/glob/ls/cat in a loop) because it "outperformed everything else by a lot" — but admitted the evidence was "just 'vibes' – internal vibes"; secondary write-ups add staleness, privacy/security and simplicity as reasons — https://officechai.com/ai/claude-researcher-explains-how-agentic-search-performed-better-than-rag-for-code-generation/ , https://smartscope.blog/en/ai-development/practices/rag-debate-agentic-search-code-exploration/ (secondary; original X post not retrieved) + +### Inferences +- The ACI ablations that still transfer to Claude Code/Cursor are about *observation shaping*, not tool count. Bounded views (SWE-agent: 100 lines beat both 30 lines and the full file), summarized search results rather than paged grep dumps, and guardrails at edit time (lint on edit −3 pp when removed) map directly onto hooks (lint or typecheck after Edit) and onto habits like grepping with limits and reading line ranges. +- Exact-match `str_replace` editing (Anthropic) plus a permissive patch applier (Aider's FLEXIBLE principle) are converged best practice. Claude Code and Cursor already provide these, so a method should not invent its own edit protocol. +- Agentless showed that a fixed localize → repair → validate pipeline with regression-test rejection sampling is competitive and cheap. That validation step lives on as the standard selection filter in vendor best-of-N numbers. For an IDE agent, the portable piece is the *validation discipline*: run the existing tests, write a reproduction test, then reject candidates that break visible tests. The no-agent-loop part does not need to carry over. +- "Model > harness" (Terminal-Bench paper headline) and "same model ±15 pp across harnesses" (leaderboards) are both true. Most of the harness spread probably comes from prompts, turn and time budgets, and verification behavior rather than exotic tools. Cursor says outright that "prompts matter more". +- Because Verified is contaminated, any 2026 claim that a scaffold technique adds "+X% on Verified" should be discounted. Pro and Terminal-Bench numbers are the better signal. + +### Gaps +- No controlled 2025–2026 ablation (same model, same budget) isolating individual ACI components on a modern benchmark was found; the SWE-agent ablations are from GPT-4 Turbo-era models. +- Aider's repo-map has no published quantitative ablation (docs are qualitative). +- AutoCodeRover's SBFL contribution is not quantified in the abstract; the full paper was not read. +- The Terminal-Bench 2.0 per-harness numbers (Claude Code 58.0% vs ForgeCode 81.8% for Opus 4.6) come from secondary blogs; tbench.ai was not checked directly. +- SWE-bench Pro 2026 leaderboard figures (Mythos 5, Fable 5, Opus 5 ~80%) and the reported July 2026 OpenAI audit of Pro come from aggregators only. +- OpenHands CodeAct 2.x SWE-bench numbers and Moatless Tools cost figures were not retrieved from primary sources. + +## 2. Localization and context engineering + +### Takeaway +More context reliably hurts once it is unfocused. Chroma found focused ~300-token prompts beat ~113k-token prompts on all 18 models. SWE-agent's full-file view and full history both lost about 3–5 pp. Cognition reports that "over-polluting context hurts more than omitting some context". What works: +- Just-in-time retrieval, with grep/glob agentic search as the backbone, and semantic search added for large repos (Cursor: +12.5% QA accuracy, +2.6% code retention on ≥1k-file repos). +- Exploration offloaded to isolated or fast subagents that return pointers (files and line ranges) or compact summaries. +- Cheap context hygiene, such as observation masking or tool-result clearing, which halves cost with no loss versus LLM summarization. +- A stable, append-only prompt prefix for KV-cache reuse (10x price difference). + +### Cited Findings +**Context rot — the empirical basis** +- Chroma "Context Rot" (Jul 14 2025) [tech report]: 18 LLMs (Claude Opus 4/Sonnet 4/3.7/3.5/Haiku 3.5, o3, GPT-4.1, GPT-4o, Gemini 2.5 Pro/Flash, Qwen3 …), 194,480 calls; performance degrades with input length even on simple tasks — models "do not use their context uniformly"; lower needle–question similarity degrades faster; a single distractor hurts and four hurt more; shuffled haystacks beat coherent ones across all 18 models; on LongMemEval, focused prompts (~300 tokens) beat full prompts (~113k tokens) for every model (Claude largest gap, mainly abstentions); GPT models hallucinated most with distractors. Recommendation: separate retrieval from reasoning; pass only relevant context — https://www.trychroma.com/research/context-rot +- Lost in the Middle (Liu et al., TACL) [PEER]: U-shaped performance — highest when relevant info is at the beginning or end, significantly worse in the middle of long contexts even for long-context models (2023 models; Chroma 2025 found needle *position* had no notable effect for 2025 models, while length/distractors/similarity still did) — https://arxiv.org/abs/2307.03172 ; https://www.trychroma.com/research/context-rot +- SWE-agent ablation (2024) shows the same effect inside coding agents: showing the full file 12.7% vs a 100-line window 18.0%; full history 15.0% vs last-5-observations 18.0% (SWE-bench Lite, GPT-4 Turbo) — https://arxiv.org/html/2405.15793 + +**Anthropic "Effective context engineering for AI agents" (Sep 29 2025) [VENDOR guidance]** +- Context is a finite resource with diminishing returns; each token draws down an "attention budget" (n² pairwise attention); degradation is gradual, not a cliff — https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents +- System prompt at the "right altitude" (neither brittle hard-coded logic nor vague); start minimal and add instructions based on observed failures; tools self-contained, token-efficient, minimal overlap ("bloated tool sets with ambiguous choices are a common failure mode"); a few diverse canonical examples rather than every edge case — same URL +- Just-in-time retrieval: keep lightweight identifiers (paths, queries) and load data via tools at runtime; Claude Code is hybrid — CLAUDE.md loaded up front, glob/grep just-in-time — same URL +- Long-horizon: (a) compaction — summarize a near-full window, keep architectural decisions, unresolved bugs, implementation details, discard redundant tool output; Claude Code keeps the summary plus the 5 most recently accessed files; tune for recall first then precision; tool-result clearing is the lightest compaction; (b) structured note-taking to files outside the window (Claude Playing Pokémon kept progress notes across thousands of steps and context resets); (c) sub-agents with clean windows that explore with tens of thousands of tokens and return condensed 1,000–2,000-token summaries — same URL + +**Manus "Context Engineering for AI Agents" (Jul 18 2025, Yichao 'Peak' Ji) [VENDOR/ANEC]** +- KV-cache hit rate is "the single most important metric" for a production agent; average input:output ratio ~100:1; Claude Sonnet cached input $0.30/MTok vs $3/MTok uncached (10x); keep prompt prefix stable (no second-precision timestamps), append-only context, deterministic serialization — https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus +- Mask, don't remove tools mid-task (changing tool definitions invalidates cache and confuses the model); use consistent tool-name prefixes (`browser_`, `shell_`) to constrain groups — same URL +- File system as unlimited, persistent, restorable context (drop a page but keep its URL/path) — same URL +- Recitation: typical task ~50 tool calls; agent rewrites a todo.md to push the global plan into the recent end of context, countering drift / lost-in-the-middle — same URL +- Keep failed actions and error traces in context so the model updates beliefs; vary serialization to avoid few-shot pattern lock-in (e.g., repetitive drift reviewing 20 resumes) — same URL + +**Compaction strategy for coding agents — simple beats clever** +- "The Complexity Trap" (Lindenbauer et al., JetBrains Research/TUM, arXiv 2508.21433; NeurIPS 2025 DL4C workshop) [PEER-workshop]: within SWE-agent on SWE-bench Verified across five model configurations, simple *observation masking* (hiding older tool outputs) roughly halves cost vs the raw agent while matching and sometimes slightly exceeding LLM summarization's solve rate (e.g., Qwen3-Coder 480B 53.8% raw → 54.8% masked, per search snippet of v2); a hybrid cuts cost a further 7% vs masking and 11% vs summarization; preliminary evidence it generalizes to OpenHands — https://arxiv.org/abs/2508.21433v3 , https://neurips.cc/virtual/2025/131688 +- Consistent with SWE-agent's 2024 ablation (keep last-5 observations 18.0% vs full history 15.0%) and Anthropic's "tool-result clearing is the lightest form of compaction" — https://arxiv.org/html/2405.15793 , https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents + +**Search/localization: grep-style agentic search vs embeddings** +- Cognition SWE-grep (Oct 16 2025) [VENDOR]: in Windsurf/Devin, trajectories often spent >60% of their first turn retrieving context; RL-trained retrieval subagents issue up to 8 parallel tool calls per turn for max 4 turns (vs 10–20 serial turns for typical agentic search); SWE-grep-mini >2,800 tok/s, SWE-grep >650 tok/s on Cerebras (~20x / 4.5x Haiku 4.5's ~140 tok/s); scored by weighted F1 (β=0.5, precision-weighted) on an internal CodeSearch eval, claimed to match frontier models; with Sonnet 4.5 as main model it completed the same number of hard SWE-bench Verified tasks in significantly less end-to-end time. Rationale: "over-polluting context hurts more than omitting some context"; subagent returns files + line ranges, not free-form summaries (summaries can mislead the main model) — https://cognition.com/blog/swe-grep +- Cursor semantic search (Nov 6 2025) [VENDOR]: custom embedding model trained on agent-session traces; offline (Cursor Context Bench) average +12.5% accuracy answering codebase questions (range 6.5–23.5% across models, all frontier models gained); online A/B: +0.3% code retention overall, +2.6% on codebases with ≥1,000 files; without semantic search, dissatisfied follow-ups +2.2%; semantic search used *alongside* grep — "the combination of these two leads to the best outcomes" — https://cursor.com/blog/semsearch +- Contrast: Claude Code dropped vector-DB RAG for pure agentic search on internal "vibes" (see §1) — https://officechai.com/ai/claude-researcher-explains-how-agentic-search-performed-better-than-rag-for-code-generation/ +- AutoCodeRover's AST-structured search (classes/methods) + SBFL and Aider's ranked repo map (default 1k tokens) are the 2024 structured-localization approaches — https://arxiv.org/abs/2404.05427 , https://aider.chat/docs/repomap.html + +### Inferences +- The Claude Code vs Cursor disagreement on embeddings is mostly about deployment. Claude Code's evidence is "vibes". Cursor's is an A/B test with small but real gains, and those gains concentrate in large repos and come from semantic search *combined with* grep. A method running in both IDEs should treat semantic search as an optional accelerator for big codebases, with grep/glob as the default. +- The strongest portable context practices are cheap. Isolate exploration in a read-only subagent that returns file:line pointers plus a short summary (Anthropic: 1–2k tokens; Cognition: line ranges). Clear or mask old tool outputs. Keep tool outputs bounded, with logs written to files and grepped (C compiler harness). Push the current plan to the end of context with a todo file that gets rewritten as work progresses (Manus recitation). +- Summaries from subagents can mislead the main agent (Cognition). Asking subagents for verifiable pointers (paths, line ranges, quoted snippets) rather than conclusions reduces this risk. +- Manus' KV-cache lesson argues against dynamic tool sets and editing the system prompt mid-session. Anthropic's Tool Search Tool takes the other side: defer-loading tools cut 85% of definition tokens and *raised* accuracy. The two can be reconciled. Load rarely used tools on demand through search, and keep the base prefix stable. + +### Gaps +- No controlled study comparing grep-only vs embedding-augmented search inside the same coding agent on a public benchmark was found (Cursor's Context Bench is internal). +- Quantitative effect of repo maps / codemaps (Aider, Windsurf Codemaps) on modern agents remains unmeasured in public sources. +- Chroma tested retrieval-style tasks, not agentic coding trajectories. Context-rot curves specific to coding agents (accuracy vs context fill on SWE tasks) were not found beyond SWE-agent's 2024 ablation and the Complexity Trap paper. +- Exact numeric solve rates in the Complexity Trap abstract were not shown; the 53.8%→54.8% figure comes from a search snippet of v2. + +## 3. Single-agent vs multi-agent; delegation, parallel agents, reviewer/verifier agents + +### Takeaway +For coding, the evidence favors a single writer thread with helper agents that contribute intelligence: read-only exploration, review with fresh context, planning, and verification. Parallel writers that coordinate as peers do poorly. The one controlled study (Google, 180 configs, non-coding) found every multi-agent variant degraded sequential tasks by 39–70%, while centralized coordination gained +80.8% on parallelizable ones. Independent agents amplified errors 17.2x. Parallel writing does work at scale in one specific shape: hierarchical planner → isolated workers → judge, with separate workspaces and a near-perfect automated verifier (Cursor's browser, Anthropic's C compiler). Even then, both report frequent conflicts, drift, and the need for fresh restarts and human review. The cost is ~4x (single agent) to ~15x (multi-agent) chat tokens. + +### Cited Findings +**The 2025 "don't" position — Cognition (Jun 12 2025, Walden Yan) [VENDOR opinion]** +- Two principles: (1) "share context, and share full agent traces, not just individual messages"; (2) "actions carry implicit decisions" — conflicting implicit assumptions between agents produce bad results. Flappy Bird example: background subagent builds a Super-Mario-style background, bird subagent builds a mismatched bird; passing the original task to each doesn't fix it — https://cognition.com/blog/dont-build-multi-agents +- Recommended: single-threaded linear agent; for long tasks add a history-compression model (Cognition fine-tuned smaller models for this). Claude Code subagents described approvingly as answering well-defined questions, not writing code, never running in parallel with the main agent — keeps investigative work out of the main history. 2024-era edit-apply split (big model writes edit description, small model applies) was "often faulty"; edit decision + application now usually one model — same URL + +**Anthropic multi-agent research system (Jun 13 2025) [VENDOR]** +- Opus 4 lead + Sonnet 4 subagents beat single-agent Opus 4 by 90.2% on an internal research eval; agents use ~4x chat tokens, multi-agent ~15x; on BrowseComp three factors explain 95% of variance, token usage alone 80% (plus tool-call count and model choice); parallel tool calling cut research time up to 90%; a tool-testing agent rewriting a tool description cut later task-completion time 40%; ~20 eval queries were enough to start — https://www.anthropic.com/engineering/multi-agent-research-system +- Explicitly *not* a good fit: tasks with few parallelizable steps "such as most coding work", domains where all agents need the same context, many inter-agent dependencies, low-value tasks (token cost), real-time coordination — same URL +- Delegation needs objective, output format, tool/source guidance and clear boundaries (vague instructions caused duplicated searches); effort-scaling rules in the prompt: simple fact-finding 1 agent / 3–10 tool calls, comparisons 2–4 subagents / 10–15 calls each, complex research >10 subagents; subagents write outputs to a filesystem and pass lightweight references to avoid "game of telephone" loss and token copying; lead saves plan to memory because >200k context gets truncated — same URL + +**Controlled study — Google Research "Towards a Science of Scaling Agent Systems" (Kim et al., arXiv 2512.08296, Dec 2025, v3 Apr 2026) [PREPRINT]** +- 180 controlled configurations, 3 LLM families, 5 architectures (single agent; Independent, Centralized, Decentralized, Hybrid multi-agent) on Finance-Agent, BrowseComp-Plus, PlanCraft, Workbench (no coding benchmark): on sequential reasoning tasks *all* multi-agent variants degraded performance 39–70%; centralized coordination +80.8% on parallelizable tasks; decentralized +9.2% on web navigation; independent agents amplify errors 17.2x vs 4.4x for centralized; coordination gives diminishing/negative returns once single-agent baseline exceeds ~45%; tool-heavy tasks suffer disproportionately from multi-agent overhead under fixed budgets; predictive model picks best strategy for 87% of held-out configs (R² 0.524), 4/5 principles held on GPT-5.2 — https://arxiv.org/abs/2512.08296v2 ; blog https://research.google/blog/towards-a-science-of-scaling-agent-systems-when-and-why-agent-systems-work/ + +**The 2026 revision — Cognition "Multi-Agents: What's Actually Working" (Apr 22 2026, Walden Yan) [VENDOR]** +- What changed: models became more agentic at tool use, managing context limits and passing distilled context; enterprise Devin usage ~8x in 6 months; but cited big demos (200k-LOC browser, 100k-LOC C compiler, 10k+-iteration training-script optimization) all had simple verifiable success criteria unlike most real software — https://cognition.com/blog/multi-agents-working +- Core rule: writes stay single-threaded; extra agents contribute intelligence, not parallel writes — same URL +- Review loop: Devin Review catches ~2 bugs/PR on average, ~58% severe (logic errors, missing edge cases, security); works best when coder and reviewer share *no* prior context (reviewer sees only the diff, avoiding context rot from the long coding session); coder filters findings against user instructions to prevent loops/out-of-scope work — same URL +- "Smart friend": SWE-1.5 (~950 tok/s) primary + Sonnet 4.5 for planning in Windsurf; mitigations: force ≥1 consultation, train/prompt when to escalate, domain rules (always escalate merge conflicts), share a forked full context, let the smart model push back; result: SWE-1.5 too weak as primary (bad at knowing when/what to escalate); cross-frontier pairing (Claude + GPT) gave real gains on hardest scenarios as a *capability router* rather than difficulty escalator — same URL +- Hierarchical delegation is live: manager Devin spawns child Devins coordinated via internal MCP ("map-reduce-and-manage"); failure modes: managers overly prescriptive without codebase context, agents wrongly assume shared state with children, children don't report back by default. Parallel-writer swarms and unstructured agent negotiation still "mostly a distraction" — same URL + +**Hundreds of parallel agents — Cursor "Scaling long-running autonomous coding" (Jan 14 2026, Wilson Lin) [VENDOR case study]** +- Coordination designs that failed: (a) shared coordination file with locks — locks held too long / never released, 20 agents ran at the effective throughput of 2–3, brittle; (b) optimistic concurrency control — simpler/more robust but didn't fix deeper problems; (c) flat peers with no hierarchy — agents became risk-averse, avoided hard tasks, made small safe changes, churned. An "integrator" role for QC/conflict resolution was removed because it created more bottlenecks than it solved — https://cursor.com/blog/scaling-agents +- What worked: planners continuously explore and create tasks (can spawn sub-planners per area); workers execute assigned tasks without coordinating with each other and push; a judge decides at the end of each cycle whether to continue, and the next iteration starts *fresh* ("periodic fresh starts are still needed to combat drift and tunnel vision") — same URL +- Results: web browser from scratch ~1 week, >1M LoC across ~1,000 files, hundreds of concurrent workers; Solid→React migration in Cursor's codebase over 3 weeks (+266K/−193K) passing CI but "still needed careful review"; video-rendering rewrite 25x faster (merged); ongoing Java LSP (550K LoC), Windows 7 emulator (1.2M LoC), Excel (1.6M LoC); trillions of tokens — same URL +- Model-by-role: GPT-5.2 much better at extended autonomous work (instruction following, focus, less drift); Opus 4.5 "tended to stop earlier and take shortcuts"; GPT-5.2 a better planner than coding-specialized GPT-5.1-Codex; "The harness and models matter, but the prompts matter more" — same URL + +**Multi-agent failure taxonomy — MAST (Cemri et al., "Why Do Multi-Agent LLM Systems Fail?", arXiv 2503.13657, 2025) [PREPRINT; venue unverified]** +- 14 failure modes in 3 categories — system design/specification issues, inter-agent misalignment, task verification — from 1,600+ annotated traces across 7 MAS frameworks; ChatDev only 33.33% correct on their ProgramDev benchmark; failures stem mainly from orchestration/design rather than solely model limits; verifier gap (e.g., checking code compiles but not that it meets the request) — https://arxiv.org/html/2503.13657v3 , https://sky.cs.berkeley.edu/project/mast/ (44.2% "system design" share from a secondary summary — unverified) + +**Reviewer/verifier agents in production — Cognition 2025–2026 [VENDOR]** +- Devin Review (Jan 2026) + autofix loop: reviewer flags bugs and Devin fixes until the diff comes back clean; ~2 bugs/PR, ~58% severe (see above) — https://cognition.com/blog/multi-agents-working , https://cognition.com/blog/testing-development +- "Verifying Agentic Development at Scale" (May 29 2026): Devin tests its own changes by running the app on a VM with computer-use; writes a *source-grounded test plan* first (models otherwise assumed nonexistent app paths); annotates each test with assertions marked passed/failed/untested, to reduce rationalizing unexpected results / claiming unverified results; repeated steps (login) as deterministic scripts in a repo "testing skill" ("decrease flakiness dramatically", no figure); failure modes: over-testing unrelated areas, setup rabbit-holes, missing the core changed behavior, mistimed screenshots, "cheating" by running JS to trigger states; engineers ran 10–20 Devins in parallel, each with own dev server; no accuracy numbers — https://cognition.com/blog/testing-development +- Devin 2025 performance review (Nov 14 2025): PR merge rate 67% (vs 34% a year earlier), ~4x faster problem solving, ~2x resource efficiency; best fit = clear upfront requirements + verifiable outcomes, ~4–8 junior-engineer-hours; struggles with ambiguous open-ended work and *mid-task requirement changes*; better codebase understanding (auto-generated docs/diagrams) was a major driver of the merge-rate gain; "code quality is not straightforwardly verifiable" so human review remains — https://cognition.com/blog/devin-annual-performance-review-2025 + +**Parallel agents in separate workspaces — Anthropic C compiler (Feb 5 2026, Nicholas Carlini) [VENDOR case study]** +- 16 parallel Claude Opus 4.6 instances, ~2,000 Claude Code sessions over ~2 weeks, ~2B input / 140M output tokens, just under $20,000; produced a ~100k-line Rust C compiler that builds Linux 6.9 on x86, ARM and RISC-V — https://www.anthropic.com/engineering/building-c-compiler +- Coordination was deliberately minimal: each agent in its own Docker container cloning a shared bare git repo; work claimed by writing a lock file in `current_tasks/` (git sync forces a collision loser to pick another task); pull/merge/push then remove the lock; merge conflicts frequent; a bash loop restarts sessions; *no orchestrator agent and no other inter-agent communication* — same URL +- Parallelism broke down when all agents hit the same Linux-kernel bug and overwrote each other's fixes; fixed by using GCC as a known-good oracle (compile most files with GCC, a subset with the new compiler) so agents could work on different failures; delta debugging later — same URL +- Test/harness lessons: the verifier must be nearly perfect or Claude solves the wrong problem; CI added to stop regressions ("new features often broke existing functionality"); minimize context pollution (print a few lines, log details to files, put "ERROR" on the same line for grep, precompute summary stats); `--fast` mode runs a deterministic per-agent 1%/10% test sample to cope with "time blindness"; agents maintain READMEs/progress files to re-orient in fresh containers; specialized roles (dedupe, compiler perf, generated-code efficiency, Rust-quality critic, docs) — same URL + +**Planner / generator / evaluator — Anthropic "Harness design for long-running application development" (Mar 24 2026, Prithvi Rajasekaran) [VENDOR case study]** +- Three agents (Claude Agent SDK + Playwright MCP): planner expands a 1–4 sentence prompt into a product spec (deliverables, not granular technical detail, to avoid cascading spec errors); generator builds with git; evaluator clicks through the running app, tests UI/API/DB state against hard per-criterion thresholds and fails the sprint with detailed feedback. "Sprint contracts": generator proposes scope + verification, evaluator reviews, iterate to agreement; agents communicate through files (e.g., Sprint 3 had 27 criteria) — https://www.anthropic.com/engineering/harness-design-long-running-apps +- Out of the box Claude was a weak QA agent — found real issues then "talked itself into approving them", tested superficially; calibration needed few-shot score breakdowns and several rounds of prompt revision against the author's judgment. Rationale for separating roles: models are lenient grading their own work — same URL +- Retro game maker (Opus 4.5): solo run 20 min / $9 → broken core play mode; full 3-agent harness 6 h / $200 → playable game (>20x cost). DAW (Opus 4.6, simplified harness): 3 h 50 min, $124.70 total (planner $0.46; three build rounds $71.08/$36.89/$5.88; three QA rounds $3.24/$3.09/$4.06) — same URL +- With Opus 4.6, sprint decomposition was removed (builder ran coherently >2 h), per-sprint evaluation replaced by a single final QA pass; evaluator "worth its cost only for tasks beyond what the model handles reliably alone"; planner retained (without it the generator under-scoped). Lesson: "every harness component encodes an assumption about what the model can't do" — remove components one at a time when models change — same URL + +### Inferences +- Delegation pays off for coding in these cases: + - Read-heavy, parallelizable investigation that would otherwise pollute the main context (Anthropic, Cognition, Claude Code docs). + - Review or verification by an agent with *fresh* context that sees only the diff and the criteria (Devin Review ~2 bugs/PR, 58% severe; Claude Code writer/reviewer; the Anthropic evaluator). + - Mechanical, well-specified execution handed to a cheaper sidekick under a strong lead (Fusion). + - Truly independent work units, each in its own worktree with an automated oracle (C compiler, Cursor). +- Delegation hurts in these cases: + - Sequential, tightly coupled edits. + - Serial debugging, where "the accumulated context is the work". + - Splitting one design across parallel writers (Flappy Bird). + - Tasks the single agent already handles well (Google: coordination returns diminish or turn negative once the single-agent baseline exceeds ~45%). + - Delegation forced on a lead that doesn't trust or brief the worker well. +- Reviewer agents need explicit scoping. Claude Code's docs warn that a reviewer told to find gaps will always find some, which leads to over-engineering. Cognition filters reviewer findings against user instructions. +- Lightweight coordination via git plus lock files works for a handful of agents (16 in the C compiler). Cursor found locks bottlenecked at 20+ (throughput of 2–3) and moved to hierarchy. For an IDE method, worktree-isolated subagents with one integrating lead is the practical ceiling. +- The fact that Opus 4.5 "stopped earlier and took shortcuts" in Cursor's runs, while GPT-5.2 drifted less, suggests that model-specific persistence matters for long autonomous runs. Harness components such as judges, feature lists and stop hooks compensate for it. + +### Gaps +- No controlled single- vs multi-agent comparison on a *coding* benchmark (SWE-bench Pro, Terminal-Bench) with equal token budgets was found; Google's study excluded coding, and vendor case studies lack baselines (except Anthropic's solo vs harness game-maker anecdote). +- MAST per-mode failure rates for coding MAS (ChatDev, MetaGPT) were not retrieved; the 44.2% figure is unverified. +- No public numbers on the precision of reviewer agents (false-positive rate of Devin Review / Claude Code review) were found. +- Claude Code "agent teams" (experimental) have no published evaluation found. +- Cognition's 2026 hierarchical-delegation results ("week-long tasks spanning multiple PRs") have no quantitative success rates. + +## 4. Planning and reasoning (planner/executor, thinking effort, search/best-of-N, reflection) + +### Takeaway +The most reliable SWE-specific gains come from *grounded* selection and verification, not from same-context self-reflection. +- Parallel attempts filtered by visible regression tests and then ranked by a scorer or critic add about 5–16 pp: Sonnet 4.5 77.2%→82.0%; OpenHands 60.6%→66.4% with a trained critic; OpenHands Best@8 73.8% vs Random@8 57.9% on mixed-outcome tasks; MCTS +23% relative. +- More thinking is not monotonically better. Overthinking correlates with failure (picking low-overthinking runs gave +~30% with −43% cost). Opus 4.5 at medium effort matched Sonnet 4.5's best with 76% fewer tokens. Adaptive effort (GPT-5-Codex) is the current pattern. +- Explicit planning helps when scope is uncertain or spans multiple files, and should be skipped for one-sentence diffs. Planner/executor role splits keep earning their place at long horizons, because without a planner the generator under-scopes. + +### Cited Findings +**Tree search / best-of-N with verifiers** +- SWE-Search (Antoniades et al., ICLR 2025) [PEER]: MCTS over the Moatless agent with a hybrid LLM value function (numeric + qualitative), a Value Agent and a Discriminator Agent (multi-agent debate); 23% relative improvement across five models on SWE-bench vs standard open-source agents without MCTS — https://arxiv.org/abs/2410.20285 +- OpenHands + trained critic (Apr 2025) [VENDOR]: Claude 3.7 Sonnet at temp 1.0, multiple rollouts, filter patches failing regression/reproduction tests, pick highest critic score: 60.6% (1 rollout) → 66.4% (5 rollouts) on SWE-bench Verified, log-linear in attempts; trained critic chosen over prompt-based reranking because it worked better — https://openhands.dev/blog/sota-on-swe-bench-verified-with-inference-time-scaling-and-critic-model ; critic weights https://huggingface.co/OpenHands/openhands-critic-32b-exp-20250417 +- OpenHands "Learning to verify AI-generated code" (Mar 5 2026) [VENDOR]: 4B critic (<~1 s latency) trained on production traces with 24 trace-observable rubrics + sparse outcomes (code survival ~4% of segments, PR merge ~6%); on the mixed-outcome subset of SWE-bench Verified, Best@8 73.8% vs Random@8 57.9%; with early stopping +17.7 pp over random using 1.35 attempts on average vs 8.0; critics trained on benchmark-style data only had AUC ~0.45–0.48 on production outcomes vs 0.69 (code-survival proxy) — https://openhands.dev/blog/20260305-learning-to-verify-ai-generated-code +- CodeMonkeys (Ehrlich et al., Stanford, arXiv 2501.14723, Jan 2025) [PREPRINT]: scales serial (more iterations/trajectory with model-written test scripts) and parallel (more trajectories/issue) compute; 57.4% SWE-bench Verified at ~$2,300 total budget; selection by voting on model-generated tests + a final multi-turn selection trajectory; selecting among edits from existing top submissions reached 66.2%, beating the best single ensemble member — https://arxiv.org/abs/2501.14723 +- SWE-Gym (Dec 2024) [PREPRINT]: trained verifier with inference-time selection reached 32.0% on SWE-bench Verified (+11.4 pp absolute) for open-weight agents — https://arxiv.org/abs/2412.21139 +- Anthropic parallel test-time compute (Claude Sonnet 4.5, Sep 29 2025) [VENDOR]: 77.2% SWE-bench Verified with a simple two-tool scaffold (bash + str-replace file edit; prompt addition "use tools as much as possible"; 200K thinking budget; 10-trial average) → 82.0% with "high compute": multiple parallel attempts, discard patches that break the repo's *visible* regression tests (Agentless-style rejection sampling, no hidden-test info), internal scoring model picks the best — https://www.anthropic.com/news/claude-sonnet-4-5 ; Sonnet 4.5 observed "maintaining focus for more than 30 hours" on complex multi-step tasks — same URL + +**Reflection / self-critique** +- Reflexion (Shinn et al., NeurIPS 2023): verbal self-reflection with self-generated unit tests; ablation on HumanEval-Rust: removing internal test generation dropped accuracy to 52% vs 60% — i.e., reflection needs grounded execution feedback — https://arxiv.org/pdf/2303.11366 (as reported in search summary) +- Critique-Coder (arXiv 2509.22824, LiveCodeBench): post-hoc self-critique ranking of 10 candidates "did not lead to performance improvements" — model can't reliably self-critique from final answers alone — https://arxiv.org/pdf/2509.22824 (search summary) +- Anthropic harness work (Mar 2026): Claude as its own QA "talked itself into approving" issues; a separate, calibrated evaluator agent with hard thresholds was needed — https://www.anthropic.com/engineering/harness-design-long-running-apps +- No clean SWE-bench ablation isolating un-grounded self-reflection was found; the evidence pattern is that critique helps when grounded in tests/execution or done by a separate context (reviewer sees only the diff), not as a same-context "reflect" prompt — see Cognition review loop https://cognition.com/blog/multi-agents-working and OpenHands trained critic https://openhands.dev/blog/20260305-learning-to-verify-ai-generated-code + +**Reasoning effort / thinking budget** +- Claude Opus 4.5 (Nov 24 2025) [VENDOR]: at *medium* effort matches Sonnet 4.5's best SWE-bench Verified score with 76% fewer output tokens; at *high* effort exceeds it by 4.3 pp with 48% fewer tokens; combining context management + memory + advanced tool use raised a fetch-enabled BrowseComp-Plus from 70.48% to 85.30% (~15 pp). Note SWE-bench Verified was run with *no* thinking budget, Terminal-Bench with 128K — https://www.anthropic.com/news/claude-opus-4-5 +- GPT-5-Codex (Sep 2025) [VENDOR via secondary]: adaptive reasoning — on the simplest ~10% of requests uses 93.7% fewer tokens than GPT-5; on the hardest ~10% spends roughly twice as long reasoning/editing/testing — https://cline.bot/blog/gpt-5-codex , https://www.techradar.com/pro/openai-launches-gpt-5-codex-with-a-74-5-percent-success-rate-on-real-world-coding (OpenAI primary page returned 403) +- "The Danger of Overthinking" (Cuadron et al., arXiv 2502.08235, Feb 2025) [PREPRINT]: 4,018 SWE-bench Verified trajectories; reasoning models overthink more (patterns: Analysis Paralysis, Rogue Actions, Premature Disengagement); higher overthinking scores correlate with lower performance; picking the lower-overthinking solution improved performance by almost 30% and cut compute cost 43% — https://arxiv.org/abs/2502.08235 +- "think" tool (Mar 20 2025) [VENDOR]: τ-bench airline 0.370→0.570 (54% relative) with an optimized prompt; retail 0.783→0.812; SWE-bench +1.6% on average (contributing to Claude 3.7 Sonnet's 0.623); helps in long tool-call chains, policy-heavy environments, costly sequential decisions; not for single/parallel calls. Dec 15 2025 update: extended (interleaved) thinking now often works better, recommended over a dedicated think tool in most cases — https://www.anthropic.com/engineering/claude-think-tool + +**Planner/executor splits** +- Aider architect/editor (Sep 2024): +3–5 pp on Aider's edit benchmark over solo models (e.g., o1-preview 79.7%→85.0%) — https://aider.chat/2024/09/26/architect.html ; but Cognition (Jun 2025) reports the 2024 edit-apply split was "often faulty" and the field converged on one model deciding+applying edits — https://cognition.com/blog/dont-build-multi-agents +- Anthropic 3-agent harness (Mar 2026): planner retained even after other scaffolding removed because without it the generator under-scoped work — https://www.anthropic.com/engineering/harness-design-long-running-apps +- Cursor (Jan 2026): planner/worker hierarchy essential at scale; GPT-5.2 a better planner than GPT-5.1-Codex — https://cursor.com/blog/scaling-agents +- Devin Fusion (Jun 2026): frontier model plans/reviews, cheaper sidekick executes; ≈ frontier quality at up to 60% lower cost — https://cognition.com/blog/devin-fusion +- Claude Code docs (Oct 2026): explore → plan → implement → commit; "If you could describe the diff in one sentence, skip the plan"; planning most useful when approach uncertain, multi-file change, or unfamiliar code — https://code.claude.com/docs/en/best-practices + +### Inferences +- In a single IDE agent, best-of-N can be approximated cheaply. Have the agent write or run a reproduction test, then reject any change that breaks existing tests. For high-stakes changes, spawn 2–N worktree-isolated attempts and pick using test results plus a fresh-context reviewer. The evidence says the selector must be grounded (tests or a trained critic). Prompt-based reranking was beaten by a trained critic (OpenHands), and post-hoc self-critique on final answers did not help (Critique-Coder). +- Reasoning effort should be adaptive. Use high effort for planning, ambiguous diagnosis and review. Use lower effort or cheaper models for mechanical execution. Watch for "analysis paralysis" and "premature disengagement" (overthinking taxonomy). A turn or time budget combined with an objective stop check (tests pass) guards against both. +- The "think" tool's value has been absorbed by interleaved extended thinking (Anthropic Dec 2025 update), so a separate think tool is unnecessary on current Claude models. +- Same-context "reflect on your answer" steps have weak SWE evidence. Reflection grounded in execution feedback is what the evidence supports: test failures, lint and type errors, screenshots. + +### Gaps +- No public, controlled measurement of plan mode or explicit planning on a coding benchmark (plan vs no-plan, same model) was found; evidence is vendor guidance and case studies. +- No SWE-bench ablation cleanly isolating un-grounded self-reflection was found. +- Exact sample counts (N) behind Anthropic's parallel test-time compute scores are not disclosed. +- OpenAI GPT-5-Codex primary page was inaccessible (403); adaptive-reasoning numbers come via secondary sources. +- Claude Opus 4.5's absolute SWE-bench Verified number was only in a chart image on the announcement page (not extracted). + +## 5. Memory and state (instruction files, progress files, checkpoints, long-horizon resumption) + +### Takeaway +Instruction files are obeyed, which is exactly why they are costly. The best controlled study (ETH, 2026) found AGENTS.md-style files do not raise success, add more than 20% cost, and that overviews don't help localization. LLM-generated files are slightly harmful (−0.5% to −3%). Short, human-written, non-obvious requirements give a small gain (~+4%). A correlational study found ~29% lower runtime with AGENTS.md present. Adherence also decays as instruction count grows (IFScale: ~68% at 500 instructions). The conclusion is to keep instruction files minimal and push the rest to on-demand skills or docs, or to deterministic hooks. + +For long-horizon work, the evidence-backed state pattern is externalized, structured state: +- A JSON feature/test list the agent may only tick. +- A progress log plus git commits. +- An init/smoke-test script run at the start of every session. +- One feature per session. +- File-based memory plus clearing of stale tool results (+39% on Anthropic's internal eval). +- Compaction, which is now increasingly trained into the models (Codex-Max, Opus 4.5+). + +### Cited Findings +**Instruction/context files (AGENTS.md / CLAUDE.md) — controlled studies** +- Gloaguen et al. (ETH Zurich / LogicStar), "Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?" arXiv 2602.11988 (v1 Feb 12 2026, v3 Sep 29 2026) [PREPRINT; an ICLR 2026 virtual listing exists]: context files "generally do not raise task success rates" while raising inference cost >20% on average, across LLMs/agents and for both LLM-generated and developer-written files; agents *do* follow the instructions; repository overviews "do not help" — https://arxiv.org/abs/2602.11988 ; listing https://iclr.cc/virtual/2026/10021235 +- Detail (v1 HTML): LLM-generated files −0.5% (SWE-bench Lite) and −2% (AGENTbench, 138 tasks from niche repos; intro says −3%); developer-written files ~+4% on AGENTbench (better than none for all agents except Claude Code); LLM files add ~2.45 / 3.92 steps and ~20% / 23% cost; e.g., Claude Code + Sonnet 4.5 on AGENTbench 40.7→46.5 steps, $1.15→$1.33; Codex + GPT-5.2 $0.38→$0.57; reasoning tokens +14–22% for GPT-5.2 — https://arxiv.org/html/2602.11988v1 +- Same paper: instructions are obeyed strongly (`uv` used ~1.6x/instance when mentioned vs <0.01 when not; repo-specific tools ~2.5x vs <0.05) — so every listed requirement becomes extra work; overviews did not reduce steps to reach the files changed by the PR; overviews present in 8/12 developer files and 95–100% of LLM-generated files (Sonnet 4.5, GPT-5.2, Qwen3); when existing docs were removed, LLM files *helped* ~+2.7% (they substitute for missing docs); stronger models don't write better context files. Recommendation: omit LLM-generated context files for now; keep human-written ones to minimal requirements such as repo-specific tooling — https://arxiv.org/html/2602.11988v1 +- Lulla et al., "On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents" arXiv 2601.20404 (Jan 28 2026, v2 Mar 30 2026) [PREPRINT]: 10 repos, 124 GitHub PRs, with vs without AGENTS.md: median runtime −28.64%, output tokens −16.58%, "comparable task completion behavior"; correlational, efficiency-only — https://arxiv.org/abs/2601.20404v1 +- Reconciliation (inference, see Inferences): the two studies measure different things (efficiency on PRs with existing human AGENTS.md vs success on issue-resolution); both are consistent with "short, human-written, non-obvious requirements help; overviews and generated boilerplate cost tokens without improving success". + +**Instruction count vs adherence** +- IFScale (Jaroslawicz et al., Distyl AI, NeurIPS 2025, arXiv 2507.11538) [PEER]: 500 keyword-inclusion instructions in a report-writing task, 20 models / 7 providers; best frontier models only ~68% accuracy at 500 instructions; three decay shapes — threshold (reasoning models o3, Gemini 2.5 Pro: near-perfect until a critical density then rising variance), linear (GPT-4.1, Claude Sonnet 4), exponential (GPT-4o, Llama 4 Scout); bias toward earlier instructions — https://arxiv.org/abs/2507.11538 , https://neurips.cc/virtual/2025/122417 ; a later Arize blog claims newer models are "an order of magnitude better" (secondary) — https://arize.com/?p=28407 + +**OpenAI "Harness engineering" (Feb 11 2026, Ryan Lopopolo) [VENDOR; primary page 403 to fetcher — details via secondary]** +- Internal product built with zero manually written code (app logic, tests, CI, docs, observability, tooling all by Codex) in ~1/10th the estimated time; humans "steer", agents execute; main engineering job becomes designing environments, specifying intent and building feedback loops — https://openai.com/index/harness-engineering/ (title/date/claims confirmed via search snippets) +- [secondary] AGENTS.md used as a short table of contents pointing into a structured `docs/` directory rather than an encyclopedia; ~1,500 PRs merged in five months, ~3.5 PRs/engineer/day, team grew 3→7 — https://rywalker.com/research/openai-harness , https://yage.ai/share/harness-engineering-survey-en-20260312.html + +**Claude Code official best practices (docs, current as of Oct 2026) [VENDOR guidance]** +- Core constraint stated: "Claude's context window fills up fast, and performance degrades as it fills" — https://code.claude.com/docs/en/best-practices +- CLAUDE.md: loaded every session, so include only broadly applicable things; keep it short; per line ask "Would removing this cause Claude to make mistakes?"; "Bloated CLAUDE.md files cause Claude to ignore your actual instructions!"; include non-guessable bash commands, non-default style rules, test runners, repo etiquette, project-specific architectural decisions, env quirks, gotchas; exclude what can be read from code, standard conventions, detailed API docs (link instead), frequently changing info, file-by-file descriptions; use emphasis ("IMPORTANT") on one line only; occasional-use knowledge goes into on-demand *skills*; deterministic must-happen actions go into *hooks* (CLAUDE.md is advisory) — same URL +- Verification first: "Give Claude a check it can run" (tests, build exit code, linter, fixture diff, screenshot); gate the stop via prompt, `/goal` (separate evaluator re-checks every turn), a Stop hook (deterministic), or a verification subagent so "the agent doing the work isn't the one grading it"; ask for evidence (test output, commands, screenshots) rather than assertions — same URL +- Explore → plan (plan mode) → implement → commit; plan mode adds overhead — skip when "you could describe the diff in one sentence"; for larger features have Claude interview you, write SPEC.md, then execute in a *fresh session* — same URL +- Session hygiene: `/clear` between unrelated tasks; after two failed corrections, clear and re-prompt ("a clean session with a better prompt almost always outperforms a long session with accumulated corrections"); steer compaction (`/compact `, CLAUDE.md "when compacting, preserve modified files and test commands"); checkpoints/rewind snapshot files edited via Claude's edit tools only (not Bash changes) — "not a replacement for git"; named sessions, `--continue/--resume` — same URL +- Failure patterns listed: kitchen-sink session, correcting over and over, over-specified CLAUDE.md, trust-then-verify gap, infinite exploration (unscoped investigation reading hundreds of files) — same URL + +**Compaction trained into the model — OpenAI GPT-5.1-Codex-Max (Nov 2025) [VENDOR via secondary]** +- Codex compacts the session near the window limit giving a fresh window, repeating until the task is done; OpenAI observed >24 h tasks internally; 77.9% SWE-bench Verified vs 73.7% for GPT-5.1-Codex (high); ~30% fewer *thinking* tokens than GPT-5.1-Codex at the same SWE-bench Verified score — https://openai.com/index/gpt-5-1-codex-max/ (via https://the-decoder.com/openai-releases-gpt-5-1-codex-max-to-handle-engineering-tasks-that-span-twenty-four-hours/ , https://www.neowin.net/news/openai-announces-gpt51-codex-max-a-new-coding-model-built-for-long-running-tasks/) + +**Long-horizon harness state (Anthropic, Nov 26 2025) [VENDOR, qualitative]** +- Compaction alone was not enough: Opus 4.5 looping across context windows fell short of a production-quality web app from a high-level prompt. Fix: an *initializer* session (same tools/system prompt, different first user prompt) writes `init.sh`, `claude-progress.txt`, a comprehensive feature list (>200 features for a claude.ai clone, each with category/description/test steps/`passes:false`) and an initial git commit; every later *coding* session makes incremental progress on one feature, commits with descriptive messages, updates the progress file — https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents +- Feature list in JSON because the model is less likely to inappropriately overwrite JSON than Markdown; agents may only flip `passes`, and are forbidden to remove/edit tests — same URL +- Session-start routine: `pwd`, read git log + progress file, read feature list, start dev server via `init.sh`, run a basic end-to-end check (catches bugs left by previous sessions), then pick highest-priority unfinished feature — same URL +- Failure modes → fixes: one-shotting and running out of context → feature list + one feature/session; declaring victory early → full feature list up front; broken/undocumented state → progress notes + commits at end, smoke test at start; marking done without testing → browser-automation (Puppeteer MCP) end-to-end self-verification (unit tests/curl alone led to false "done"); no quantitative benchmark reported — same URL +- Later (Mar 2026): Sonnet 4.5 showed "context anxiety" (wrapping up early near perceived context limit) so context *resets* + handoff artifacts were essential; Opus 4.5 largely removed this, so resets were dropped for one continuous session with automatic compaction — https://www.anthropic.com/engineering/harness-design-long-running-apps + +**Memory tool + context editing (Anthropic, Sep 29 2025) [VENDOR]** +- On an internal agentic-search eval: memory tool + context editing +39% over baseline; context editing alone +29%; in a 100-turn web-search eval context editing cut token consumption 84% and let agents finish workflows that otherwise died from context exhaustion. Context editing clears stale tool calls/results near limits; memory tool = client-side file directory Claude can CRUD across conversations — https://claude.com/blog/context-management + +**Tool-definition bloat as a context cost (Anthropic "Advanced tool use", Nov 24 2025) [VENDOR]** +- Typical MCP tool-definition overhead: GitHub (35 tools) ~26K tokens, Slack (11) ~21K, Jira ~17K; five servers / 58 tools ~55K; Anthropic internal peak 134K before optimization — https://www.anthropic.com/engineering/advanced-tool-use +- Tool Search Tool (defer-loading tools): ~77K → ~8.7K tokens (−85%); MCP-eval accuracy Opus 4 49%→74%, Opus 4.5 79.5%→88.1% — same URL +- Programmatic Tool Calling (model writes code that calls tools, only results enter context): 43,588 → 27,297 tokens (−37%) on complex research tasks; internal knowledge retrieval 25.6%→28.5%; GIA 46.5%→51.2% — same URL +- Tool Use Examples in definitions: 72%→90% on complex parameter handling (internal) — same URL + +### Inferences +- **CLAUDE.md / AGENTS.md sizing rule:** include only what the agent cannot infer and would otherwise get wrong (non-standard commands and tooling, repo-specific gotchas, hard constraints). Skip overviews and file-by-file descriptions. ETH found overviews in 95–100% of LLM-generated files, and they did not help. The one condition where generated files helped (+2.7%) was a repo with no docs at all, which suggests their value is as a *doc substitute*, not an add-on. +- Every instruction is followed, and following it costs steps (the `uv` and repo-tool usage jump). Conditional or rarely needed instructions belong in skills or linked docs loaded on demand, the "table of contents, not encyclopedia" pattern (OpenAI harness engineering; Claude Code skills). Must-happen rules belong in hooks. +- For tasks that span sessions, Anthropic's initializer/coding-agent split is directly reproducible in Claude Code and Cursor with no special infrastructure: + - a `features.json` with pass flags the agent may only flip; + - `progress.md`; + - `init.sh`; + - commit after each feature; + - a startup routine of reading the log and git log, then running the smoke test. +- Context resets vs compaction is model-dependent. Resets with handoff files were needed for Sonnet 4.5's "context anxiety", while Opus 4.5+ ran continuously with compaction. Since a method should survive model changes, handoff files are worth keeping either way: they are cheap and they also serve humans. +- IDE checkpoints (Claude Code rewind) only capture edits made through the edit tools, not Bash changes, so git commits remain the real recovery mechanism. + +### Gaps +- No public study measured CLAUDE.md length vs adherence for Claude Code specifically; IFScale uses keyword-inclusion in report writing, not coding instructions. +- ETH paper per-agent success numbers are only in figures; the "−3%" vs "−2%" AGENTbench figure is inconsistent within the paper. +- The Lulla et al. study does not name agents/models in the abstract and measures only efficiency. +- OpenAI's harness-engineering post could not be fetched (403); AGENTS.md "table of contents" details and throughput numbers are from secondary summaries. +- No quantitative evaluation of Anthropic's long-running harness (feature list / progress file) vs a baseline was published. + +## 6. Model/tool selection (cheap/fast models for subagents, cost/latency trade-offs) + +### Takeaway +A strong lead model that briefs well and delegates mechanical work to a cheaper or faster model now matches frontier quality at roughly 40–60% lower cost (Devin Fusion; Fable+sidekick −54% cost at the same score). On standardized scaffolds, small models trail flagships by only ~5–7 pp: Haiku 4.5 at 70.0% vs Opus 4.5 at 76.8% bash-only Verified; Haiku 4.5 at 73.3% vendor-reported. The savings depend on the lead's behavior: delegate early, write constraint-and-done-criteria briefs, review cheaply. Forced delegation, weak primaries, and judgment-heavy or serial-debugging tasks erase the gains. Per-token price is a poor predictor of agent cost. Turns, context size and cache hits dominate. + +### Cited Findings +**Main-agent + cheap "sidekick" routing — Cognition Devin Fusion (Jun 29 2026; chart updated Aug 7 2026) [VENDOR]** +- Two agents in parallel, each with own toolset and cached context: a frontier main agent that takes "minimal actions" (plans, interprets ambiguity, delegates, monitors, final review) and a cheaper sidekick doing delegated execution; lightweight classifiers decide mid-task when to switch models; switches are timed at context compaction (a cache miss anyway) to avoid extra cache cost — https://cognition.com/blog/devin-fusion +- FrontierCode 1.1 Extended (score / avg cost per task): Fable 5 (xhigh) 64.9 / $10.53; Opus 5 (medium) 63.6 / $3.51; Devin Fusion 63.1 / $1.35; GPT-5.6 Sol (high) 58.7 / $3.41; Kimi K3 58.2 / $3.12; Grok 4.5 (high) 56.6 / $1.09. Claimed up to 60% cheaper than frontier models (revised from 35%); Fusion with Fable 5 41% cheaper than pure Fable 5 at same performance; 88% of internal merged PRs driven entirely by the Fusion router — same URL +- Per-task delegation examples: cost cuts of 25–62% on mechanical work (ES6 modernization −62%, OpenTracing removal −32%, JSON-Schema models −38%, UI team selector −28%, Quarkus MCP transport −25%); delegation works for mechanical/large refactors/removals/slow test runs and "backfires" on tasks where "the judgment is the deliverable" (per-task score columns on the page were garbled in extraction — not reported here) — same URL +- Earlier "smart friend" attempt with a *weak* primary (SWE-1.5) failed; cost savings only hold if both models are strong — https://cognition.com/blog/multi-agents-working + +**How the lead delegates matters more than per-token price — Cognition "Making Fable Cheaper Than Opus" (Jul 13 2026) [VENDOR]** +- 3,000 eval sessions on FrontierCode 1.1 (deep analysis on 40 tasks); Fable 5 costs 2x Opus 4.8 per token, yet: pure Fable $4.03/run (score 60.8) vs pure Opus $3.06 (55.4); with a cheap sidekick: Fable $1.86 (60.7) vs Opus $2.04 (54.6) — Fable+sidekick 54% cheaper than pure Fable at nearly unchanged score — https://cognition.com/blog/making-fable-cheaper-than-opus +- Mechanism: Fable delegated *early* (first action often a handoff), wrote briefs listing constraints, edge cases and a definition of "done" and asked for a report before commit; Opus delegated late after doing the expensive work, dictated full file contents, and when reviewing pulled sidekick files back into context 2x more often with 4x more corrective edits. Lead stats: Fable 11.5 turns vs Opus 26.5; 6.1k vs 19.0k output tokens; 545k vs 1,679k cumulative lead input; lead never edited code in 81% of Fable-led runs vs 24% of Opus-led; ~3 handoffs/run for both. Example: same hashing task, Opus-brief result scored 25 vs Fable-brief 94 — same URL +- Lessons stated: per-token price is a poor predictor of agent cost (turn count, context size and what the lead chooses *not* to do dominate); forcing delegation alone hurt performance — it pays only when the lead trusts the sidekick; good briefs can *raise* quality; delegation has little value for short tasks or serial debugging "where the accumulated context is the work"; no latency data — same URL +- Fast retrieval subagents: SWE-grep-mini >2,800 tok/s and SWE-grep >650 tok/s vs Haiku 4.5 ~140 tok/s; same task completion in less wall time with Sonnet 4.5 main agent — https://cognition.com/blog/swe-grep + +**Small-model subagents (Anthropic Haiku 4.5, Oct 15 2025) [VENDOR]** +- Claude Haiku 4.5: 73.3% SWE-bench Verified (50-trial average), $1/$5 per MTok in/out; "similar coding performance" to Sonnet 4 at one-third the cost and >2x the speed; a customer quote says up to 4–5x faster than Sonnet 4.5; Anthropic: Sonnet 4.5 can plan and "orchestrate a team of multiple Haiku 4.5s to complete subtasks in parallel"; Augment: 90% of Sonnet 4.5's agentic coding performance — https://www.anthropic.com/news/claude-haiku-4-5 +- Claude Code docs allow a per-subagent `model:` (e.g., `model: opus` for a security reviewer with read-only tools Read/Grep/Glob/Bash) — https://code.claude.com/docs/en/best-practices + +**Claude Code subagent mechanics (docs, Oct 2026) [VENDOR docs]** +- Built-ins: Explore (read-only, Write/Edit denied; thoroughness quick/medium/very thorough) and Plan (read-only, used in plan mode) both use the *main conversation's model* by default (Explore does not default to Haiku; override by defining an `Explore` subagent with `model: haiku`); general-purpose has all tools and uses `CLAUDE_CODE_SUBAGENT_MODEL` if set. Model field accepts sonnet/opus/haiku/fable/full ID/inherit; resolution order: per-invocation param → frontmatter → env var → main model — https://code.claude.com/docs/en/sub-agents +- Isolation: subagent starts fresh (no conversation history), gets its system prompt + delegation message + CLAUDE.md (except Explore/Plan) + git status (except Explore/Plan); only the final report returns to the main conversation; transcripts stored separately; resumable via SendMessage; nesting up to 3 layers by default; up to 20 concurrent; optional `isolation: worktree` (temp worktree, auto-removed if unchanged); forks inherit full conversation and reuse the prompt cache (cheaper for shared-context tasks but no input isolation) — same URL +- Official guidance: use a subagent when output is verbose and not needed in main context (tests, logs, docs), to enforce tool restrictions, when work is self-contained and can return a summary, or to route to a cheaper model; stay in main conversation for frequent back-and-forth, phases sharing significant context, quick targeted changes, or when latency matters (a non-fork subagent must re-gather context); parallel subagents returning detailed results can consume significant main context — same URL + +**Cheaper models are close on standardized scaffolds (Feb 2026)** +- Bash-only SWE-bench Verified: Claude Haiku 4.5 (high) 70.0% vs Claude Opus 4.5 (high) 76.8% and Sonnet 4.5 72.8%; Gemini 3 Flash (high) 75.8% — https://simonwillison.net/2026/Feb/19/swe-bench/ + +**Lead/subagent model mix in Anthropic research system** +- Opus 4 lead + Sonnet 4 subagents was the configuration that beat single Opus 4 by 90.2%; model choice was one of three factors explaining 95% of BrowseComp variance — https://www.anthropic.com/engineering/multi-agent-research-system +- Cursor (Jan 2026) assigns models by role: GPT-5.2 as planner beat coding-specialized GPT-5.1-Codex; Opus 4.5 stopped earlier/took shortcuts in long autonomous runs — https://cursor.com/blog/scaling-agents +- KV-cache economics: cached input ~10x cheaper (Claude Sonnet $0.30 vs $3/MTok) with ~100:1 input:output ratio in agent loops — https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus ; Claude Code forks reuse the parent's prompt cache and "usually cost less than a fresh subagent" for shared-context tasks — https://code.claude.com/docs/en/sub-agents + +### Inferences +- A practical model policy for a Claude Code/Cursor method: + - Main session on the strongest model for planning, ambiguity resolution, integration and final review. + - Read-only exploration and log/test triage subagents on a fast or cheap model (Haiku-class), returning file:line pointers. + - Mechanical, well-specified edits (renames, migrations, removals, test scaffolding) delegated to a cheaper model under a written brief with constraints, edge cases and a definition of done. + - Judgment-heavy design, serial debugging and short tasks kept in the main thread. + - Fresh-context review on a strong model. +- Delegation briefs are themselves a quality lever. In Cognition's data, Fable's brief yielded 94 vs 25 on the same task, and briefs that dictated whole files caused rework. A method should standardize a brief template: goal, constraints, edge cases, done-criteria, report format, and "report before commit". +- Switching models mid-session breaks the prompt cache. Devin Fusion times switches at compaction for this reason. In IDE terms, prefer delegating to subagents (separate caches) over flipping the main session's model mid-task, and use forks when the subtask needs the full parent context. +- Claude Code's built-in Explore subagent inherits the main model by default. Cheap exploration requires an explicit `model: haiku` override or `CLAUDE_CODE_SUBAGENT_MODEL`. + +### Gaps +- No independent (non-vendor) benchmark of lead+sidekick routing for coding was found; Devin Fusion and FrontierCode are Cognition-internal. +- Haiku-class subagent *accuracy* for exploration (missed files vs main-model exploration) is unmeasured publicly; SWE-grep's F1 comparison is informal. +- Latency figures for Fusion/sidekick setups were not reported. +- Several 2026 models referenced by vendors (Fable 5, Opus 5, Opus 4.8, GPT-5.6 Sol, Mythos 5) appear only in vendor/aggregator tables; their official model cards were not checked. diff --git a/docs/research/notes/claude-code-capabilities.md b/docs/research/notes/claude-code-capabilities.md new file mode 100644 index 0000000..342d037 --- /dev/null +++ b/docs/research/notes/claude-code-capabilities.md @@ -0,0 +1,789 @@ +# Claude Code extension capabilities and file formats (state as of 2026-10-14) + +STATUS: COMPLETE for all nine key questions (2026-10-14). Every finding was taken from the official docs at code.claude.com/docs (raw markdown, fetched 2026-10-14; newest changelog entry v2.1.299, October 14, 2026) or agentskills.io. No third-party sources were used as authority. Version notes such as "v2.1.277+" are copied from the docs' own annotations. + +## Memory / instructions (CLAUDE.md, imports, AGENTS.md) + +Doc snapshot: official docs downloaded 2026-10-14 as raw markdown from code.claude.com/docs/en/*.md; the changelog's newest entry is **v2.1.299 (October 14, 2026)** — [Changelog](https://code.claude.com/docs/en/changelog). All "vX" notes below are taken from the docs' own version annotations. + +### Takeaway +CLAUDE.md files (managed > user > project > local, plus ancestors at launch and subdirectories on demand) are concatenated, not overridden, and delivered as a user message after the system prompt; `@path` imports expand at launch (max 4 hops) and do not save context; `.claude/rules/*.md` with optional `paths:` frontmatter gives path-scoped instructions; since v2.1.277 Claude Code reads `AGENTS.md` natively but by default ONLY when no CLAUDE.md/CLAUDE.local.md exists on the path — a kit that ships both must import `@AGENTS.md` from CLAUDE.md (or the user must change the "Project instructions" setting). Target <200 lines per file. + +### Cited Findings +**Locations and load order** (broadest to most specific; later = read later in context) — [Memory](https://code.claude.com/docs/en/memory#choose-where-to-put-claude-md-files) +| Scope | Location | +|---|---| +| Managed policy | macOS `/Library/Application Support/ClaudeCode/CLAUDE.md`; Linux/WSL `/etc/claude-code/CLAUDE.md`; Windows `C:\Program Files\ClaudeCode\CLAUDE.md` (cannot be excluded) | +| User | `~/.claude/CLAUDE.md` (+ `~/.claude/rules/*.md`) | +| Project | `./CLAUDE.md` or `./.claude/CLAUDE.md` (+ `./.claude/rules/**/*.md`) | +| Local | `./CLAUDE.local.md` (personal, add to `.gitignore`) | +- Managed content can also be inlined via the `claudeMd` key in `managed-settings.json` (honored only in managed/policy settings): `{"claudeMd": "Always run `make lint` before committing.\nNever push directly to main."}` — [Memory](https://code.claude.com/docs/en/memory#deploy-organization-wide-claude-md) +- Claude Code loads `CLAUDE.md` and `CLAUDE.local.md` from the cwd and every ancestor at launch; all are **concatenated rather than overriding**, ordered root → cwd; within a directory `CLAUDE.local.md` is appended after `CLAUDE.md` — [Memory](https://code.claude.com/docs/en/memory#how-claude-md-files-load) +- CLAUDE.md/CLAUDE.local.md files in **subdirectories** load on demand, once Claude reads/writes/edits a file in that subdirectory (a Bash `cat`/`head` of a single file counts as a read); they do not appear under `/context` "Memory files" — [Memory](https://code.claude.com/docs/en/memory#how-claude-md-files-load) +- Block-level HTML comments `` are stripped before injection (kept inside code blocks) — [Memory](https://code.claude.com/docs/en/memory#how-claude-md-files-load) +- `--add-dir` directories do not load CLAUDE.md unless `CLAUDE_CODE_ADDITIONAL_DIRECTORIES_CLAUDE_MD=1`; then `CLAUDE.md`, `.claude/CLAUDE.md`, `.claude/rules/*.md`, `CLAUDE.local.md` load from them — [Memory](https://code.claude.com/docs/en/memory#load-from-additional-directories) +- CLAUDE.md "content is delivered as a user message after the system prompt, not as part of the system prompt itself"; no guarantee of strict compliance; use hooks for must-run behaviour and `--append-system-prompt` for system-prompt-level text — [Memory](https://code.claude.com/docs/en/memory#claude-isn-t-following-my-claude-md) +- Project-root CLAUDE.md survives `/compact` (re-read from disk and re-injected); nested CLAUDE.md and `paths:` rules reload on demand — [Memory](https://code.claude.com/docs/en/memory#instructions-seem-lost-after-compact) + +**`@path` imports** — [Memory](https://code.claude.com/docs/en/memory#import-additional-files) +- Syntax `@path/to/import` anywhere in the file; relative paths resolve relative to the **importing file**, absolute and `@~/...` allowed; recursive imports up to **max depth of four hops**. +- Spaces must be backslash-escaped (`@Design\ Docs/api-conventions.md`); a quoted path is not imported at all. +- Parsing skips Markdown code spans and fenced blocks, so `` `@README` `` stays literal. +- Imports do not reduce context cost (imported files load at launch). +- Imports resolving outside the working directory in a project-level file are "external": first encounter shows a one-time approval dialog; if declined they stay disabled. User-scope files (`~/.claude/CLAUDE.md`, `~/.claude/rules/`) load imports without the dialog (except in Cowork desktop sessions). + +**Rules directory `.claude/rules/`** — [Memory](https://code.claude.com/docs/en/memory#organize-rules-with-claude/rules/) +- All `.md` files discovered recursively; rules without `paths` load at launch with the same priority as `.claude/CLAUDE.md`; user rules `~/.claude/rules/` load before project rules. +- Only frontmatter field read: `paths` (YAML list or comma-separated string of globs, brace expansion allowed, budget 1,000 expanded patterns/4 MiB per rule). Other fields are ignored silently; unparseable YAML → rule loads as if it had no `paths` (see `claude --debug`). +- A path-scoped rule loads when Claude uses Read/Write/Edit on a matching file (or a Bash single-file read). +- Symlinks supported; a symlink target outside the working dir is treated like an external import (needs approval; afterwards only rules **without** `paths` load). Network paths (UNC, `/net`, `/Network`) are not followed. +- Project rules are skipped if `project` is excluded from `--setting-sources` (on-demand rules were also excluded only from v2.1.211). +Minimal example: +```markdown +--- +paths: + - "src/api/**/*.ts" +--- +# API Development Rules +- All API endpoints must include input validation +``` + +**Size guidance** — [Memory](https://code.claude.com/docs/en/memory#write-effective-instructions) +- "target under 200 lines per CLAUDE.md file"; a CLAUDE.md up to 4 MiB is loaded in full, a larger file is skipped; startup and `/status` warn when a file is over the recommended length or files combined exceed a limit (each CLAUDE.md, rules file and `@path` import counts as a file) — [Memory](https://code.claude.com/docs/en/memory#my-claude-md-is-too-large) +- `/doctor prompt-audit` (v2.1.283+) audits CLAUDE.md, CLAUDE.local.md, AGENTS.md, rules, skills, commands, subagents and output styles for stale/conflicting content — [Memory](https://code.claude.com/docs/en/memory#audit-your-instruction-files) +- `claudeMdExcludes` (glob array against absolute paths, any settings layer, arrays merge) skips specific CLAUDE.md/rules files; managed CLAUDE.md can't be excluded — [Memory](https://code.claude.com/docs/en/memory#exclude-specific-claude-md-files) + +**AGENTS.md** — [Memory](https://code.claude.com/docs/en/memory#agents-md) +- Native reading of `AGENTS.md` requires **v2.1.277+** (some Bedrock/telemetry-off sessions only from v2.1.281); implemented by built-in plugin `cc-plugin-agents-md@builtin` (ID was `agents-md@builtin` before v2.1.285). +- Default (`claude-md-or-agents-md`): Claude reads AGENTS.md **only if there is no `CLAUDE.md`, `.claude/CLAUDE.md` or `CLAUDE.local.md` in the cwd or any ancestor**; `~/.claude/CLAUDE.md`, managed CLAUDE.md and `.claude/rules/` don't count and load alongside. +- When read: every `AGENTS.md` and `.claude/AGENTS.md` in cwd and ancestors at start; a subdirectory AGENTS.md on Read when that subdir has no CLAUDE.md files; `@path` imports inside AGENTS.md expand. **Not read**: `AGENTS.local.md`, `AGENTS.override.md`, anything under `.agents/`. +- "Project instructions" values (`/config`, or `pluginConfigs` in user settings / `--settings` / managed only — ignored in project and local settings): `claude-md-or-agents-md` (default), `claude-md-and-agents-md`, `claude-md`, `managed-only`. +```json +{ "pluginConfigs": { "cc-plugin-agents-md@builtin": { "options": { "instructionFiles": "claude-md-and-agents-md" } } } } +``` +- Differences for AGENTS.md read via the setting: `InstructionsLoaded` hooks don't fire; not loaded from `--add-dir` dirs; external imports load only if already approved. +- Recommended portable pattern: `CLAUDE.md` containing `@AGENTS.md` then Claude-specific lines (never double-loads); symlink `ln -s AGENTS.md CLAUDE.md` works but Edit/Write refuse to write through symlinks and Windows checkouts may get a one-line text file. +- `/init` migrates `.cursor/rules/`, `.cursorrules`, `.github/copilot-instructions.md` (and with `CLAUDE_CODE_NEW_INIT=1` also AGENTS.md, `.devin/rules/`, `.windsurf/rules/`, `.windsurfrules`, `.clinerules`); `/import` (v2.1.213+) copies another agent's instruction files, MCP servers, commands, subagents and skills — [Memory](https://code.claude.com/docs/en/memory#migrate-instructions-from-other-tools) + +**Auto memory** — [Memory](https://code.claude.com/docs/en/memory#auto-memory) +- Stored at `~/.claude/projects//memory/` (`` derived from git repo so worktrees share it), with `MEMORY.md` index + topic files; first **200 lines or 25KB** of MEMORY.md loaded each session; topic files read on demand; memory types in frontmatter `type`: `user`, `feedback`, `project`, `reference`; `modified` timestamp auto-added (v2.1.214+). +- Toggle: `/memory`, `autoMemoryEnabled` setting, or `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`; relocate with `autoMemoryDirectory` (absolute or `~/`). +- Main-conversation auto memory is not loaded into subagents (except forks); subagents have their own via the `memory` field. +- Debug: `/context` (Memory files list), `/memory`, `InstructionsLoaded` hook. + + +**Version history (changelog)** — [Changelog](https://code.claude.com/docs/en/changelog) +- `@path/to/file.md` imports in CLAUDE.md: v0.2.107 (May 9, 2025); `.claude/rules/` support: v2.0.64 (Dec 10, 2025); native AGENTS.md reading ("in a project with no CLAUDE.md, Claude Code reads AGENTS.md instead"): v2.1.277 (Sep 18, 2026). + +### Inferences +- For a method kit that targets several agents, the safest layout is a short `CLAUDE.md` that starts with `@AGENTS.md`, because a kit cannot rely on users' "Project instructions" setting and project/local settings cannot change it. +- Path-scoped rules are the cheapest way to deliver area-specific guidance; skills are better for procedures (they load only on invocation/relevance). + +### Gaps +- The exact combined-size limit that triggers the multi-file startup warning is not stated in the memory page. + +## Skills (SKILL.md, frontmatter, arguments, commands, Agent Skills standard) + +### Takeaway +A skill is a directory `/SKILL.md` (YAML frontmatter + Markdown body) under `~/.claude/skills/`, `.claude/skills/` (also ancestors up to repo root and nested subdirs on demand), a plugin's `skills/`, or the managed settings directory; it becomes `/name` and, unless `disable-model-invocation: true`, is auto-invocable by Claude from its `description`. `.claude/commands/*.md` files are the legacy form and still work identically (skill wins on a name clash). All frontmatter fields are optional; unknown fields are silently ignored; the Agent Skills open standard (agentskills.io) defines a 6-field subset (`name`, `description`, `license`, `compatibility`, `metadata`, `allowed-tools`) and Claude Code accepts all of them, so standard-conformant skills load unchanged. + +### Cited Findings +**Locations** — [Skills: Choose where skills load](https://code.claude.com/docs/en/skills#where-skills-live) +| Location | Path | Loads in | +|---|---|---| +| Enterprise | `.claude/skills//SKILL.md` inside the managed settings directory (e.g. `/etc/claude-code/.claude/skills//` on Linux) | all users on the machine | +| Personal | `~/.claude/skills//SKILL.md` | all your projects (not Cowork/cloud sessions) | +| Project | `.claude/skills//SKILL.md` | this repository | +| Nested | `/.claude/skills//SKILL.md` | sessions started in/below ``, or once Claude touches files there | +| Additional dir | `.claude/skills/...` in a `--add-dir` directory | that session (also loads its `.claude/commands/` and `.claude/agents/`; `permissions.additionalDirectories` does NOT) | +| Plugin | `/skills//SKILL.md` | where plugin enabled, as `/plugin-name:skill-name` | +| claude.ai account | synced into `~/.claude/skills/synced/` (v2.1.273+ in terminal) | sessions signed in with that account | +- Project skills load from `.claude/skills/` in the start directory and every parent up to the repository root; in a linked git worktree the search stops at the worktree root, but on v2.1.277+ if the worktree has no `.claude/skills` at its root the main checkout's project skills load — [Skills](https://code.claude.com/docs/en/skills#discovery-from-parent-and-nested-directories) +- Nested skills with a clashing name are addressable as `/apps/web:deploy` — [Skills](https://code.claude.com/docs/en/skills#discovery-from-parent-and-nested-directories) +- Name-clash precedence: enterprise > personal > project; a user skill replaces a bundled skill/built-in command (not its aliases); a skill beats a `.claude/commands/` file of the same name; plugin skills never clash (namespaced) — [Skills](https://code.claude.com/docs/en/skills#resolve-skills-that-share-a-name) +- Reserved names: folder `synced` (any case) and `anthropic-skills` / `anthropic-skills:*` don't load outside plugins — [Skills](https://code.claude.com/docs/en/skills#where-skills-live) +- A skill folder containing `.claude-plugin/plugin.json` loads as a plugin named `@skills-dir` (can bundle agents, hooks, MCP); in project `.claude/skills/` requires accepting workspace trust — [Skills](https://code.claude.com/docs/en/skills#where-skills-live) +- Live change detection: edits under `~/.claude/skills/`, `.claude/skills/`, and `--add-dir` skills are picked up without restart (not in bare mode); a newly created top-level skills dir needs `/reload-skills` — [Skills](https://code.claude.com/docs/en/skills#live-change-detection) +- Claude Code's documented skill locations do not include `.agents/skills/` (the cross-client convention described by agentskills.io for clients to scan); the memory page says nothing under a `.agents/` directory is read for AGENTS.md — [Skills](https://code.claude.com/docs/en/skills#where-skills-live), [Memory](https://code.claude.com/docs/en/memory#when-claude-code-reads-agents-md), [agentskills.io client guide](https://agentskills.io/client-implementation/adding-skills-support) + +**Frontmatter reference (Claude Code)** — all optional; "A field name must match the table exactly, hyphens included: Claude Code ignores a field it doesn't recognize without reporting an error"; frontmatter is read only if `---` is the first line; unparseable YAML → skill loads with no fields (body still invocable via `/name`, but description matching fails; see `--debug`); booleans accept `yes/no/on/off/1/0` (v2.1.218+) — [Skills: Frontmatter reference](https://code.claude.com/docs/en/skills#frontmatter-reference) +| Field | Meaning | +|---|---| +| `name` | Command name in `/` menu; defaults to directory name (directory name also still invokes it) | +| `description` | Recommended. What + when; if omitted, first non-empty body line. Combined `description`+`when_to_use` truncated at **1,536 chars** in the listing | +| `when_to_use` | Extra trigger phrases, appended to description (only field with underscore) | +| `argument-hint` | Autocomplete hint, e.g. `[issue-number]` | +| `arguments` | Named positional args (space-separated string or YAML list) for `$name` substitution | +| `disable-model-invocation` | `true` → Claude can't auto-invoke; description not in context; also blocks preloading into subagents and scheduled-task firing. Default `false` | +| `user-invocable` | `false` → hidden from `/` menu, only Claude invokes. Default `true` | +| `allowed-tools` | Tools pre-approved **only during the turn that invokes the skill** (clears on next user message); space/comma string or YAML list; does not restrict other tools | +| `disallowed-tools` | Tools removed while skill active (clears on next message) | +| `model` | Model override for rest of current turn (`/model` values or `inherit`); with `context: fork` sets the fork's model | +| `effort` | `low`, `medium`, `high`, `xhigh`, `max` | +| `context` | `fork` → run in a forked subagent context | +| `agent` | Subagent type used with `context: fork` (`Explore`, `Plan`, `general-purpose` or custom; default `general-purpose`) | +| `background` | With `context: fork`; `false` waits for result in same turn. Default `true` (v2.1.218+) | +| `hooks` | Hooks registered when the skill is invoked, kept for rest of session (see Hooks section, `once` option) | +| `paths` | Globs limiting auto-activation to matching files (same format as rules) | +| `shell` | `bash` (default) or `powershell` for `!` injection | +| `metadata` | Free-form map, ignored by Claude Code | +| `license` | Agent Skills field; accepted, not acted on | +| `compatibility` | Agent Skills field; ≤500 chars; accepted, not acted on | +- `.claude/commands/` files accept the same fields **except `name` and `paths`** — [Skills](https://code.claude.com/docs/en/skills#frontmatter-reference) + +**Command name resolution** — [Skills: How a skill gets its command name](https://code.claude.com/docs/en/skills#how-a-skill-gets-its-command-name) +- `.claude/skills/deploy-staging/SKILL.md` → `/deploy-staging` (or `/deploy` with `name: deploy`); `.claude/commands/deploy.md` → `/deploy`; `.claude/commands/frontend/component.md` → `/frontend:component`; `my-plugin/skills/review/SKILL.md` → `/my-plugin:review` (bare `/review` also works if unclaimed); plugin-root `SKILL.md` uses `name` (fallback plugin dir name). Prefix not doubled if `name` already has it (v2.1.246+). + +**String substitutions** — [Skills: Available string substitutions](https://code.claude.com/docs/en/skills#available-string-substitutions) +- `$ARGUMENTS` (full string as typed; if no placeholder receives args, Claude Code appends `ARGUMENTS: `), `$ARGUMENTS[N]` (0-based), `$N` shorthand (`$0`, `$1`), `$name` (from `arguments`), `${CLAUDE_SESSION_ID}`, `${CLAUDE_EFFORT}`, `${CLAUDE_SKILL_DIR}` (skill's own dir; for plugin skills the skill subdir), `${CLAUDE_PROJECT_DIR}`, `${CLAUDE_PLUGIN_ROOT}` and `${CLAUDE_PLUGIN_DATA}` (plugin skills only). +- `${CLAUDE_SKILL_DIR}`/`${CLAUDE_PROJECT_DIR}` (and plugin vars) are substituted in the body AND in Bash rules in `allowed-tools`, enabling prompt-free bundled scripts: `allowed-tools: Bash(${CLAUDE_SKILL_DIR}/scripts/render.sh *)`. +- Indexed args use shell-style quoting (`/my-skill "hello world" second`); missing indexed placeholder stays literal; missing named placeholder → empty string; escape literal `$1` as `\$1`. +- Stacking: `/write-tests /fix-issue 123` loads both, passing `123` to each; first skill + up to five more; stops at a forked skill. + +**Dynamic context injection** — [Skills: Inject dynamic context](https://code.claude.com/docs/en/skills#inject-dynamic-context) +- `` !`cmd` `` at line start or after whitespace runs before Claude sees the content; multi-line via a fenced block opened with ```` ```! ````. Output is not re-scanned. +- Runs in the session shell's cwd, stderr merged, 2-minute Bash timeout; **any non-zero exit aborts the whole invocation** (`Shell command failed for pattern "..."`), except exit 1 from search/compare commands; append `|| true` otherwise. +- Injected commands never prompt: deny rule → abort; outside auto mode anything other than "allow" → abort, so pre-approve via `allowed-tools`. +- `"disableSkillShellExecution": true` replaces commands with `[shell command execution disabled by policy]` (user/project/plugin/add-dir skills; not bundled/managed). +- Including `ultrathink` in skill content requests deeper reasoning. + +**Invocation and lifecycle** — [Skills](https://code.claude.com/docs/en/skills#control-who-invokes-a-skill) +| Frontmatter | You invoke | Claude invokes | In context | +|---|---|---|---| +| default | yes | yes | description always; full body on invoke | +| `disable-model-invocation: true` | yes | no | description not in context | +| `user-invocable: false` | no | yes | description always | +- `/name args` at the start of a message runs it; `/name` mid-sentence only grants Claude permission to run it. +- Rendered SKILL.md enters as one message and **stays across turns**; not re-read; after auto-compaction the most recent invocation of each skill is re-attached keeping the first **5,000 tokens** each, combined budget **25,000 tokens** — [Skills: Skill content lifecycle](https://code.claude.com/docs/en/skills#skill-content-lifecycle) +- `allowed-tools` is NOT gated by workspace trust (applies even in `-p` in an untrusted folder); ignored when managed `allowManagedPermissionRulesOnly` is set (v2.1.282+) — [Skills](https://code.claude.com/docs/en/skills#pre-approve-tools-for-a-skill) +- Listing budget: 1% of the model's context window (`skillListingBudgetFraction`, or `SLASH_COMMAND_TOOL_CHAR_BUDGET` chars); per-entry cap `skillListingMaxDescChars` (default 1,536) — [Skills: descriptions cut short](https://code.claude.com/docs/en/skills#skill-descriptions-are-cut-short) +- Keep `SKILL.md` under 500 lines; reference supporting files via relative Markdown links — [Skills: Add supporting files](https://code.claude.com/docs/en/skills#add-supporting-files) + +**`context: fork`** — [Skills: Run skills in a subagent](https://code.claude.com/docs/en/skills#run-skills-in-a-subagent) +- Starts a subagent of type `agent` with the skill content as its prompt; no conversation history. Runs in background by default (v2.1.218+), but waits in `-p`/SDK, with `CLAUDE_CODE_DISABLE_BACKGROUND_TASKS=1`, etc. Backgrounded forks get the narrower background tool set and their edits are outside checkpoints (`/rewind` can't undo). Explore/Plan start without CLAUDE.md. + +**Access control** — [Skills: Restrict Claude's skill access](https://code.claude.com/docs/en/skills#restrict-claude-s-skill-access) +- Permission rules: `Skill` (all), `Skill(commit)` exact, `Skill(review-pr *)` prefix; `skillOverrides` setting with values `"on"`, `"name-only"`, `"user-invocable-only"`, `"off"` (not for plugin skills); `disableBundledSkills` turns off bundled skills. +```json +{ "skillOverrides": { "legacy-context": "name-only", "deploy": "off" } } +``` + +**Validation / evaluation** +- `claude plugin validate .claude/skills` (or `~/.claude/skills`) finds SKILL.md files whose frontmatter doesn't parse (v2.1.233+) — [Skills: Skill not triggering](https://code.claude.com/docs/en/skills#skill-not-triggering) +- `/skill-doctor` (v2.1.252+) reports context cost and usage per skill; `/doctor` estimates listing cost — [Skills](https://code.claude.com/docs/en/skills#find-unused-skills) +- `claude plugin eval` for plugin skills (with/without-plugin baseline; `tool_used: Skill` grader); `skill-creator` plugin (`/plugin install skill-creator@claude-plugins-official`) uses `evals/evals.json` — [Skills: Evaluate](https://code.claude.com/docs/en/skills#evaluate-and-iterate-on-a-skill) + +**Minimal valid examples** — [Skills](https://code.claude.com/docs/en/skills#pass-arguments-to-skills) +```markdown +--- +name: fix-issue +description: Fix a GitHub issue +disable-model-invocation: true +argument-hint: [issue-number] +--- +Fix GitHub issue $ARGUMENTS following our coding standards. +``` +```markdown +--- +name: commit +description: Stage and commit the current changes +disable-model-invocation: true +allowed-tools: Bash(git add *) Bash(git commit *) Bash(git status *) +--- +``` + +**Agent Skills open standard (agentskills.io)** — [Specification](https://agentskills.io/specification) +- Directory: `skill-name/SKILL.md` (required) + optional `scripts/`, `references/`, `assets/`. +- Required: `name` (1-64 chars, lowercase `a-z0-9` and hyphens, no leading/trailing/consecutive hyphens, **must match parent directory name**), `description` (1-1024 chars). Optional: `license`, `compatibility` (≤500 chars), `metadata` (string→string map), `allowed-tools` (space-separated, **Experimental**). +- Progressive disclosure: metadata ~100 tokens at startup; body <5000 tokens recommended; keep SKILL.md under 500 lines; file references one level deep, relative to skill root. Validate with `skills-ref validate ./my-skill` ([skills-ref](https://github.com/agentskills/agentskills/tree/main/skills-ref)). +- Client guide recommends clients scan `/./skills/`, `/.agents/skills/`, `~/./skills/`, `~/.agents/skills/` — [agentskills.io client implementation](https://agentskills.io/client-implementation/adding-skills-support) +- Claude Code: "Claude Code skills follow the Agent Skills open standard ... Claude Code extends the standard with additional features like invocation control, subagent execution, and dynamic context injection"; claude.ai uploads / Skills API / `package_skill.py` accept only `name`, `description`, `license`, `compatibility`, `metadata`, `allowed-tools` and **fail with a hard error** on other keys (`Unexpected key(s) in SKILL.md frontmatter: argument-hint. Allowed properties are: ...`) — [Skills: Using skill frontmatter outside Claude Code](https://code.claude.com/docs/en/skills#using-skill-frontmatter-outside-claude-code) + + +**Version history (changelog)** — [Changelog](https://code.claude.com/docs/en/changelog) +- "Added support for Claude Skills": v2.0.20 (Oct 16, 2025); `context: fork` and automatic skill hot-reload: v2.1.0 (Jan 7, 2026); "Merged slash commands and skills, simplifying the mental model with no change in behavior": v2.1.3 (Jan 9, 2026). + +### Inferences +- A portable method-kit skill should: use a lowercase-hyphen `name` equal to its directory, put the trigger in the first ~200 chars of `description`, and keep Claude-Code-only fields (`disable-model-invocation`, `argument-hint`, `context`, `hooks`...) out if the same SKILL.md must also be uploaded to claude.ai/API. +- Because Claude Code does not document scanning `.agents/skills/`, a cross-tool kit should install into `.claude/skills/` (symlinked skill folders are supported) or ship as a plugin. +- `allowed-tools` grants are per-turn; durable allow rules must go in settings `permissions.allow`. + +### Gaps +- No official statement found on whether Claude Code warns when `name` does not match the directory name (the spec requires it; Claude Code docs treat `name` as an override). + +## Subagents (.claude/agents/*.md) + +### Takeaway +A subagent is a Markdown file with YAML frontmatter (`name` and `description` required, camelCase multi-word fields) whose body is the subagent's **entire system prompt** (not the Claude Code system prompt). Files live in managed settings dir > `--agents` JSON > `.claude/agents/` (walked up to repo root, recursive) > `~/.claude/agents/` > plugin `agents/`. Claude delegates automatically from the `description` (use "use proactively"), or the user forces it via `@agent-`, or runs the whole session as it with `--agent`/`"agent"` setting. Each non-fork subagent starts with a fresh context (its system prompt + Claude's delegation message + CLAUDE.md hierarchy + git status + preloaded skills); Explore/Plan skip CLAUDE.md. Plugin subagents ignore `hooks`, `mcpServers`, `permissionMode`. + +### Cited Findings +**Scopes and priority** — [Subagents: Choose the subagent scope](https://code.claude.com/docs/en/sub-agents#choose-the-subagent-scope) +| Location | Priority | +|---|---| +| Managed settings dir `.claude/agents/` | 1 (highest) | +| `--agents` CLI JSON | 2 | +| `.claude/agents/` (every one from cwd up to repo root; closest wins) | 3 | +| `~/.claude/agents/` | 4 | +| Plugin `agents/` | 5 (lowest) | +- Directories scanned recursively; identity comes only from frontmatter `name` (subfolder path irrelevant for project/user; for plugins `agents/review/security.md` → `my-plugin:review:security`). Duplicate names in one tree → one loaded by filesystem read order (`/doctor` reports) — [Subagents](https://code.claude.com/docs/en/sub-agents#choose-the-subagent-scope) +- `--add-dir` directories' `.claude/agents/` also load (not watched; restart after edits) — [Subagents](https://code.claude.com/docs/en/sub-agents#choose-the-subagent-scope) +- File watcher on `~/.claude/agents/` and `.claude/agents/` picks up changes within seconds; restart needed if the `agents` dir didn't exist at session start, for `--add-dir` agents, or with `--disable-slash-commands` — [Subagents: Write subagent files](https://code.claude.com/docs/en/sub-agents#write-subagent-files) +- Plugin subagents: "For security reasons, plugin subagents don't support the `hooks`, `mcpServers`, or `permissionMode` frontmatter fields"; ship hooks in `hooks/hooks.json` and servers in `.mcp.json` instead — [Subagents](https://code.claude.com/docs/en/sub-agents#choose-the-subagent-scope) + +**Frontmatter reference** ("Only `name` and `description` are required"; "Multi-word field names use camelCase ... Claude Code ignores a field it doesn't recognize without reporting an error") — [Subagents: Frontmatter reference](https://code.claude.com/docs/en/sub-agents#supported-frontmatter-fields) +| Field | Values / notes | +|---|---| +| `name` | required; ≤256 chars; no `:`; must not start with `-`; hooks get it as `agent_type`; filename needn't match | +| `description` | required; when Claude should delegate | +| `tools` | allowlist, comma string `Read, Grep, Bash` or YAML list; inherit all if omitted; `Agent(worker, researcher)` restricts spawnable types (only effective for `--agent` main thread); `mcp__` / `mcp____*` patterns | +| `disallowedTools` | denylist, applied before `tools`; an entry with specifier like `Bash(git push *)` still removes the whole tool; `mcp__*` removes all MCP tools | +| `model` | `sonnet`, `opus`, `haiku`, `fable`, full ID (e.g. `claude-opus-5-5`), or `inherit` | +| `permissionMode` | `default`, `acceptEdits`, `auto`, `dontAsk`, `bypassPermissions`, `plan`, `manual` (alias of default); ignored for plugin subagents; ignored when parent is in bypassPermissions/acceptEdits/auto | +| `maxTurns` | max agentic turns; output marked partial (v2.1.246+) | +| `skills` | skills whose **full content** is preloaded at startup (first 32 names); cannot preload `disable-model-invocation: true` skills | +| `mcpServers` | list of names (reuse configured server) or inline `{name: {type, command, args...}}` (stdio/http/sse/ws); inline servers from project agents need folder trust (v2.1.238+); ignored for plugin subagents | +| `hooks` | hooks scoped to this subagent; `Stop` auto-converted to `SubagentStop`; project agents' frontmatter hooks need workspace trust (and don't run in untrusted `-p`, v2.1.218+); ignored for plugin subagents | +| `memory` | `user` → `~/.claude/agent-memory//`; `project` → `.claude/agent-memory//`; `local` → `.claude/agent-memory-local//`; injects first 200 lines/25KB of that MEMORY.md and auto-enables Read/Write/Edit | +| `background` | `true` keeps it in background | +| `omitClaudeMd` | `true` → start without user/project/local CLAUDE.md (v2.1.271+) | +| `effort` | `low`, `medium`, `high`, `xhigh`, `max` | +| `isolation` | `worktree` → temporary git worktree branched from default branch; auto-cleaned if no changes | +| `color` | `red`, `blue`, `green`, `yellow`, `purple`, `orange`, `pink`, `cyan` | +| `initialPrompt` | auto-submitted first user turn when run as main session agent | +| `experimental` | map; only `cacheTtl: 5m|1h` (v2.1.248+) | +- Files silently skipped (debug log only): no `name` (treated as documentation), `---` not on first line, invalid `name`, `name` without `description`, unparseable YAML. Plugin subagents without `name` still load under their filename. Validate with `claude plugin validate .claude/agents` (v2.1.233+) — [Subagents: Subagent files Claude Code skips](https://code.claude.com/docs/en/sub-agents#subagent-files-claude-code-skips) +- Model resolution order: per-invocation `model` param > frontmatter `model` > `CLAUDE_CODE_SUBAGENT_MODEL` > main model (order changed in v2.1.251); `CLAUDE_CODE_SUBAGENT_MODEL_FORCE=1` (v2.1.257+) forces one model — [Subagents: Choose a model](https://code.claude.com/docs/en/sub-agents#choose-a-model) + +Minimal example — [Subagents](https://code.claude.com/docs/en/sub-agents#write-subagent-files) +```markdown +--- +name: code-reviewer +description: Reviews code for quality and best practices. Use proactively after code changes. +tools: Read, Glob, Grep +model: sonnet +--- +You are a code reviewer. When invoked, analyze the code and provide +specific, actionable feedback on quality, security, and best practices. +``` +`--agents` JSON form (session-only; `prompt` = body; accepts the frontmatter fields except `color`/`experimental`; in `-p` also a JSON file path, v2.1.281+): +```bash +claude --agents '{"code-reviewer":{"description":"Expert code reviewer. Use proactively after code changes.","prompt":"You are a senior code reviewer.","tools":["Read","Grep","Glob","Bash"],"model":"sonnet"}}' +``` + +**Built-in subagents** — [Subagents: Built-in subagents](https://code.claude.com/docs/en/sub-agents#built-in-subagents) +- `Explore`: read-only (Write/Edit denied), main model (Opus alias when main runs Fable on Anthropic API), thoroughness quick/medium/very thorough; one-shot (no agent ID, can't resume). A user/project agent named `Explore` overrides it. +- `Plan`: read-only research agent used in plan mode; inherits model. +- `general-purpose`: all subagent tools; default for `context: fork` skills and when no type given. +- Others: `claude` (catch-all, default for background sessions), `statusline-setup` (Sonnet), `claude-code-guide` (Haiku). +- Explore and Plan start **without CLAUDE.md and git status**. Disable: `permissions.deny: ["Agent(Explore)"]`, deny `Agent` entirely, `CLAUDE_CODE_DISABLE_EXPLORE_PLAN_AGENTS=1` (v2.1.198+), or in `-p`/SDK `CLAUDE_AGENT_SDK_DISABLE_BUILTIN_AGENTS=1`. +- Task tool was renamed `Agent` in v2.1.63; `Task(...)` still works as alias — [Subagents](https://code.claude.com/docs/en/sub-agents#restrict-which-subagents-can-be-spawned) + +**Delegation and invocation** — [Subagents: Work with subagents](https://code.claude.com/docs/en/sub-agents#understand-automatic-delegation) +- Automatic delegation based on request, `description`, context; "include phrases like \"use proactively\""; startup warning when combined custom descriptions exceed 15,000 tokens. +- Explicit: natural language; `@"code-reviewer (agent)"` / `@agent-` / `@agent-my-plugin:code-reviewer` (guarantees that subagent); `claude --agent code-reviewer` or `{"agent": "code-reviewer"}` in `.claude/settings.json` (system prompt replaces default Claude Code prompt; CLAUDE.md still loads). +- Disable specific: `{"permissions": {"deny": ["Agent(Explore)", "Agent(my-custom-agent)"]}}` or `--disallowedTools "Agent(Explore)"` — [Subagents: Disable specific subagents](https://code.claude.com/docs/en/sub-agents#disable-specific-subagents) + +**Foreground/background, forks, limits** — [Subagents: Run subagents in foreground or background](https://code.claude.com/docs/en/sub-agents#run-subagents-in-foreground-or-background) +- Fork mode is ON by default in interactive sessions (v2.1.232+) → all spawned subagents run in background; OFF in `-p` and Agent SDK (then Claude chooses; foreground when it needs the result). `CLAUDE_CODE_FORK_SUBAGENT=1|0` overrides; `CLAUDE_CODE_DISABLE_BACKGROUND_TASKS=1` forces foreground. Ctrl+B backgrounds a running task. +- Background subagents keep all MCP tools but only built-ins `Read, Grep, Glob, LSP, Bash, PowerShell, Edit, Write, NotebookEdit, WebFetch, WebSearch, TodoWrite, Skill, ToolSearch, EnterWorktree, ExitWorktree, Monitor, TaskStop, SendMessage, Artifact` (+ `SubagentHandback`). Always removed from subagents: `AskUserQuestion`, `EndConversation`, `EnterPlanMode`, `ExitPlanMode` (unless permissionMode plan), `ScheduleWakeup`, `WaitForMcpServers`, `Workflow`, and `Agent` at depth limit — [Subagents: Available tools](https://code.claude.com/docs/en/sub-agents#available-tools) +- Fork (`/subtask `, v2.1.212+; or Claude requests type `fork`) inherits full conversation, system prompt, tools, model and prompt cache; deny with `Agent(fork)` — [Subagents: Fork the current conversation](https://code.claude.com/docs/en/sub-agents#fork-the-current-conversation) +- Nesting depth default 3 (`CLAUDE_CODE_MAX_SUBAGENT_SPAWN_DEPTH`); concurrent limit default 20 (`CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS`, v2.1.217+) — [Subagents](https://code.claude.com/docs/en/sub-agents#let-subagents-spawn-their-own-subagents) +- Resume via `SendMessage` to agent ID/name; transcripts at `~/.claude/projects/{project}/{sessionId}/subagents/agent-{agentId}.jsonl`; subagent output scanned for instruction-shaped text (v2.1.210+) — [Subagents: Resume subagents](https://code.claude.com/docs/en/sub-agents#resume-subagents) +- Non-fork subagents don't get output style or main auto memory; context window sized by subagent's model — [Subagents: What loads at startup](https://code.claude.com/docs/en/sub-agents#what-loads-at-startup) +- `--append-subagent-system-prompt` / `--append-subagent-system-prompt-file` (non-interactive; v2.1.205+/v2.1.261+) append text to every subagent prompt — [Subagents](https://code.claude.com/docs/en/sub-agents#write-subagent-files) +- Settings-level `SubagentStart`/`SubagentStop` hooks match on agent type name (plugin scoped names contain `:` so anchor regex `^my-plugin:db-agent$`) — [Subagents: Project-level hooks](https://code.claude.com/docs/en/sub-agents#project-level-hooks-for-subagent-events) + + +**Version history (changelog)** — [Changelog](https://code.claude.com/docs/en/changelog) +- Custom subagents + `/agents`: v1.0.60 (Jul 24, 2025); hooks in agent frontmatter: v2.1.0 (Jan 7, 2026); `memory` frontmatter field: v2.1.33 (Feb 6, 2026); `isolation: worktree`: v2.1.49-2.1.50 (Feb 19-20, 2026); Task tool renamed Agent: v2.1.63. + +### Inferences +- A method kit that needs a subagent with hooks or a specific permission mode must ship it as a project/user file, not inside a plugin (plugin subagents drop those fields). +- Because fork mode puts subagents in the background by default in interactive sessions, a kit relying on tools outside the background tool set (e.g. AskUserQuestion is never available anyway) should set `background: false` on `context: fork` skills, or not depend on those tools. + +### Gaps +- No documented maximum length for the subagent body/system prompt. + +## Hooks (events, config, I/O contract) + +### Takeaway +Hooks are configured as `{"hooks": {: [ {"matcher": "...", "hooks": [ {handler} ]} ]}}` in any settings file, a plugin's `hooks/hooks.json` (same shape, optional top-level `description`), or skill/subagent frontmatter (YAML of the same shape). Five handler types: `command`, `http`, `mcp_tool`, `prompt`, `agent` (agent = experimental). Command hooks get JSON on stdin; **exit 0** = success (stdout JSON parsed if it is exactly one JSON object), **exit 2** = blocking error (stderr becomes the reason) on events that can block, **any other exit code** = non-blocking error (action proceeds!). As of v2.1.299 there are ~30 events. Default timeout 600 s for command/http/mcp_tool. + +### Cited Findings +**Event list (what fires when)** — [Hooks reference: Hook lifecycle](https://code.claude.com/docs/en/hooks#hook-lifecycle) +`SessionStart` (session begins/resumes), `Setup` (`--init-only`, or `--init`/`--maintenance` in `-p`), `UserPromptSubmit` (prompt submitted, before Claude processes it; also on turns Claude Code starts itself), `UserPromptExpansion` (user-typed command expands into a prompt; can block), `PreToolUse` (before tool call; can block), `PermissionRequest` (tool call needs permission decision), `PermissionDenied` (auto mode denies a call), `PostToolUse` (tool succeeded), `PostToolUseFailure` (tool failed), `PostToolBatch` (after a batch of parallel tool calls resolves), `Notification`, `MessageDisplay` (while assistant text is displayed; display-only), `SubagentStart`, `SubagentStop`, `TaskCreated` (via `TaskCreate`), `TaskCompleted`, `Stop` (Claude finishes responding), `StopFailure` (turn ends due to API error), `TeammateIdle` (agent team), `InstructionsLoaded` (CLAUDE.md or `.claude/rules/*.md` loaded), `ConfigChange`, `CwdChanged`, `DirectoryAdded` (`/add-dir`), `FileChanged` (watched file changes), `WorktreeCreate` (replaces default git behaviour), `WorktreeRemove`, `PreCompact`, `PostCompact`, `PreModelSwitch` (can block), `PostModelSwitch`, `Elicitation` (MCP server requests input), `ElicitationResult`, `SessionEnd`. + +**Hook locations** — [Hooks reference: Hook locations](https://code.claude.com/docs/en/hooks#hook-locations) +| Location | Scope | +|---|---| +| `~/.claude/settings.json` | all your projects | +| `.claude/settings.json` | project, committable | +| `.claude/settings.local.json` | project, not shared | +| Managed policy settings | org-wide | +| Plugin `hooks/hooks.json` | when plugin enabled | +| Skill frontmatter `hooks:` | rest of the session once skill is invoked (`once: true` honoured only here) | +| Subagent frontmatter `hooks:` | while subagent runs (`Stop` → `SubagentStop`) | +- Hook entries **merge** across levels (never replace); identical handler defined in several settings files runs once; all matching hooks run **in parallel**; settings/plugin/managed hooks also fire inside subagents with `agent_id`/`agent_type` in input — [Hooks reference](https://code.claude.com/docs/en/hooks#hook-locations) +- Managed `allowManagedHooksOnly` blocks user/project/local/plugin hooks (except force-enabled managed plugins); `disableAllHooks: true` disables all non-managed hooks (one-run override: `--settings '{"disableAllHooks": true}'`); no per-hook disable; `/hooks` is a read-only browser — [Hooks reference: Disable or remove hooks](https://code.claude.com/docs/en/hooks#disable-or-remove-hooks) +- Project-skill frontmatter hooks follow the settings-file trust rule (registered even in an untrusted `-p` run); project-subagent frontmatter hooks need explicit workspace trust (v2.1.218+) — [Hooks reference: Hooks in skills and agents](https://code.claude.com/docs/en/hooks#hooks-in-skills-and-agents) + +**Matcher syntax** — [Hooks reference: Matcher patterns](https://code.claude.com/docs/en/hooks#matcher-patterns) +- `"*"`, `""` or omitted → match all. Only letters/digits/`_`/`-`/spaces/`,`/`|` → exact string or list (`Edit|Write`, `Edit, Write`). Any other character → **unanchored JavaScript RegExp** (`^Notebook`, `mcp__memory__.*`; use `^Edit$` for whole-string). `FileChanged` and `StopFailure` accept only `|` as separator. +- What each event matches on: tool events (`PreToolUse`, `PostToolUse`, `PostToolUseFailure`, `PermissionRequest`, `PermissionDenied`) → tool name; `SessionStart` → `startup|resume|clear|compact|fork`; `Setup` → `init|maintenance`; `SessionEnd` → `clear|resume|logout|prompt_input_exit|other`; `Notification` → `permission_prompt|idle_prompt|auth_success|elicitation_dialog|elicitation_url_dialog|elicitation_complete|elicitation_response|agent_needs_input|agent_completed|quota_auto_resume_fired|quota_auto_resume_stale|quota_auto_resume_disabled`; `SubagentStart`/`SubagentStop` → agent type; `PreCompact`/`PostCompact` → `manual|auto`; `Pre/PostModelSwitch` → canonical model name; `ConfigChange` → `user_settings|project_settings|local_settings|policy_settings|skills`; `DirectoryAdded` → `slash_command|register_repo_root`; `FileChanged` → literal filenames (`.envrc|.env`); `StopFailure` → `rate_limit|overloaded|authentication_failed|oauth_org_not_allowed|account_on_hold|billing_error|invalid_request|model_not_found|server_error|max_output_tokens|cloud_credential_error|unknown`; `InstructionsLoaded` → `session_start|nested_traversal|path_glob_match|include|compact`; `UserPromptExpansion` → command name; `Elicitation`/`ElicitationResult` → MCP server name. +- **No matcher support** (matcher silently ignored): `UserPromptSubmit`, `PostToolBatch`, `Stop`, `TeammateIdle`, `TaskCreated`, `TaskCompleted`, `WorktreeCreate`, `WorktreeRemove`, `MessageDisplay`, `CwdChanged`. +- MCP tools are named `mcp____`; matching a whole server needs `mcp__memory__.*` (bare `mcp__memory` is an exact string and matches nothing); plugin MCP tools are `mcp__plugin____` — [Hooks reference: Match MCP tools](https://code.claude.com/docs/en/hooks#match-mcp-tools) + +**Handler fields** — [Hooks reference: Hook handler fields](https://code.claude.com/docs/en/hooks#hook-handler-fields) +- Common: `type` (required: `command|http|mcp_tool|prompt|agent`), `if` (one permission rule like `"Bash(git *)"`, `"Edit(*.ts)"`; only evaluated on tool events — on other events a hook with `if` never runs; best-effort, not a security boundary), `timeout` (seconds; defaults 600 command/http/mcp_tool, 30 prompt, 60 agent; lowered to 30 on UserPromptSubmit/PreModelSwitch/PostModelSwitch, 10 on MessageDisplay; SessionEnd hooks share 1.5 s budget, raisable to 60), `statusMessage`, `once` (skill frontmatter only). +- Command: `command` (required), `args` (exec form, no shell; recommended when using path placeholders), `async`, `asyncRewake` (background; wakes Claude on exit 2), `shell` (`bash` default, `powershell`), `onFailure` (`continue` default | `block`; v2.1.295+). +- Shell form runs via `sh -c` (macOS/Linux), Git Bash on Windows, or PowerShell if no Git Bash. Exec form on Windows needs a real `.exe` (use `node script.js` instead of `.cmd` shims) — [Hooks reference: Exec form and shell form](https://code.claude.com/docs/en/hooks#exec-form-and-shell-form) +- HTTP: `url`, `headers` (with `$VAR` interpolation), `allowedEnvVars`, `onFailure`; JSON input is POSTed; non-2xx/connection failure = non-blocking; to block return 2xx + JSON decision. +- MCP tool: `server` (plugin servers as `plugin::`), `tool`, `input` (with `${tool_input.file_path}` substitution); not available on launch-time `SessionStart` or `Setup`. +- Prompt/agent: `prompt` (`$ARGUMENTS` = hook input JSON), `model`. +- Path placeholders: `${CLAUDE_PROJECT_DIR}` (project root where session started; stays put when Claude enters a worktree — read `cwd` from input instead), `${CLAUDE_PLUGIN_ROOT}`, `${CLAUDE_PLUGIN_DATA}`; all three also exported as env vars to the hook process. Plugin exec-form hooks substitute `${user_config.*}`; shell-form must use `$CLAUDE_PLUGIN_OPTION_` — [Hooks reference: Reference scripts by path](https://code.claude.com/docs/en/hooks#reference-scripts-by-path) +- Hooks run without a controlling terminal (no `/dev/tty`); use `systemMessage` or `terminalSequence` (OSC 0/1/2/9/99/777, BEL) for user-visible output; `$CLAUDE_CODE_REMOTE` is `"true"` in remote web environments; no `$CLAUDE_MODEL` env var; `$CLAUDE_EFFORT` is set — [Hooks reference](https://code.claude.com/docs/en/hooks#hook-input-and-output) + +**Common stdin fields** — [Hooks reference: Common input fields](https://code.claude.com/docs/en/hooks#common-input-fields) +`session_id`, `prompt_id`, `transcript_path` (written async, may lag), `cwd`, `scratchpad_dir` (v2.1.257+), `permission_mode` (`default|plan|acceptEdits|auto|dontAsk|bypassPermissions`; not on all events), `effort` (`{"level": ...}`), `hook_event_name`, plus `agent_id`/`agent_type` inside subagents or with `--agent`. Example PreToolUse input: +```json +{"session_id":"abc123","prompt_id":"550e8400-...","transcript_path":"/home/user/.claude/projects/.../transcript.jsonl","cwd":"/home/user/my-project","permission_mode":"default","hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"command":"npm test","description":"Run test suite","timeout":120000,"run_in_background":false},"tool_use_id":"toolu_01ABC123..."} +``` + +**Exit-code contract** — [Hooks reference: Exit code output](https://code.claude.com/docs/en/hooks#exit-code-output) +| Stdout | Exit 0 | Exit 2 | Other | +|---|---|---|---| +| valid JSON object | success, fields apply | blocking; fields read but can't override block (even `allow`) | success; exit code ignored, fields decide (failure under `onFailure: block`) | +| unparseable/invalid JSON | non-blocking error | blocking; stderr is reason | non-blocking error | +| plain text / nothing | success | blocking; stderr is reason | non-blocking error (first stderr line shown) | +- "Without valid JSON on stdout, Claude Code treats exit code 1 as a non-blocking error ... If your hook is meant to enforce a policy, use `exit 2`." — [Hooks reference: Other exit codes](https://code.claude.com/docs/en/hooks#other-exit-codes) +- Stdout on exit 0 goes to debug log only, **except** `UserPromptSubmit`, `UserPromptExpansion`, `SessionStart`, `PostModelSwitch`, where plain-text stdout is added as context for Claude. Claude never sees stderr of an exit-0 hook — [Hooks reference: Exit code 0](https://code.claude.com/docs/en/hooks#exit-code-0) +- Stdout is parsed as JSON only when the whole output is one JSON object (shell-profile echo lines break it) — [Hooks reference: JSON output](https://code.claude.com/docs/en/hooks#json-output) +- A timed-out command/http/mcp_tool `PreToolUse` hook does **not** block (unless `onFailure: "block"`); timed-out `PreModelSwitch` blocks — [Hooks reference: Timeouts](https://code.claude.com/docs/en/hooks#timeouts) +- Exit 2 per event: blocks `PreToolUse`, `UserPromptSubmit` (prompt never reaches Claude), `UserPromptExpansion`, `Stop`/`SubagentStop`/`TeammateIdle` (keeps working), `TaskCreated` (rolls back), `TaskCompleted`, `ConfigChange` (except policy), `PostToolBatch` (stops loop), `PreCompact`, `PreModelSwitch`, `Elicitation`/`ElicitationResult` (decline), `WorktreeCreate`/`WorktreeRemove` (any non-zero fails). Not blocking: `PermissionRequest` (use JSON decision), `PostToolUse`/`PostToolUseFailure` (stderr shown to Claude), `StopFailure`, `PermissionDenied`, `Notification`, `Setup`, `InstructionsLoaded` (ignored), `SubagentStart`/`SessionStart`/`SessionEnd`/`CwdChanged`/`FileChanged`/`PostCompact`/`PostModelSwitch` (stderr to user only), `DirectoryAdded` (debug log), `MessageDisplay` — [Hooks reference: Exit code 2 behavior per event](https://code.claude.com/docs/en/hooks#exit-code-2-behavior-per-event) +- Blocked PreToolUse message seen by Claude: `PreToolUse:Bash hook error: []: ` — [Hooks reference: Exit code 2](https://code.claude.com/docs/en/hooks#exit-code-2) + +**JSON output fields** — [Hooks reference: JSON output](https://code.claude.com/docs/en/hooks#json-output) +- Universal: `continue` (default `true`; `false` stops Claude entirely, overrides decisions), `stopReason`, `suppressOutput` (accepted, **no effect**), `systemMessage` (warning to user), `terminalSequence`. +- `additionalContext`, `systemMessage`, `initialUserMessage` and plain stdout are each capped at **10,000 characters** (over → saved to file, path + 2,000-char preview). +- `additionalContext` (inside `hookSpecificOutput` with `hookEventName`) is wrapped in a system reminder at the point the hook fired; write it as factual statements, not imperative "system" commands (can trigger prompt-injection defenses) — [Hooks reference: Add context for Claude](https://code.claude.com/docs/en/hooks#add-context-for-claude) +- Decision patterns: top-level `{"decision": "block", "reason": "..."}` for UserPromptSubmit, UserPromptExpansion, PostToolUse, PostToolUseFailure, PostToolBatch, Stop, SubagentStop, ConfigChange, PreCompact (only value is `"block"`); `hookSpecificOutput.permissionDecision` `allow|deny|ask|defer` + `permissionDecisionReason` (+ `updatedInput`, `additionalContext`) for PreToolUse; `hookSpecificOutput.decision.behavior` `allow|deny` (+ `updatedInput`) for PermissionRequest; `updatedToolOutput` for PostToolUse; `retry: true` for PermissionDenied; WorktreeCreate prints a path; SessionStart/SubagentStart/PostModelSwitch context only (SessionStart also `initialUserMessage`, `watchPaths`, `sessionTitle`, `reloadSkills`) — [Hooks reference: Decision control](https://code.claude.com/docs/en/hooks#decision-control) + +Minimal examples — [Hooks reference](https://code.claude.com/docs/en/hooks#how-a-hook-resolves) +```json +{"hooks":{"PreToolUse":[{"matcher":"Bash","hooks":[{"type":"command","if":"Bash(rm *)","command":"${CLAUDE_PROJECT_DIR}/.claude/hooks/block-rm.sh","args":[]}]}]}} +``` +```bash +#!/bin/bash +# .claude/hooks/block-rm.sh (chmod +x) +COMMAND=$(jq -r '.tool_input.command') +if echo "$COMMAND" | grep -q 'rm -rf'; then + jq -n '{hookSpecificOutput:{hookEventName:"PreToolUse",permissionDecision:"deny",permissionDecisionReason:"Destructive command blocked by hook"}}' +else + exit 0 +fi +``` +Plugin `hooks/hooks.json`: +```json +{"description":"Automatic code formatting","hooks":{"PostToolUse":[{"matcher":"Write|Edit","hooks":[{"type":"command","command":"${CLAUDE_PLUGIN_ROOT}/scripts/format.sh","args":[],"timeout":30}]}]}} +``` + +**Per-event details (selected, kit-relevant)** +- `SessionStart` — only `command` and `mcp_tool` types; matcher `startup|resume|clear|compact|fork`; input adds `source`, optional `model`, `agent_type`, `session_title` (+ cache-cost fields on resume, v2.1.251+); output `hookSpecificOutput.additionalContext`, `initialUserMessage` (`-p` only), `sessionTitle`, `watchPaths` (absolute paths for FileChanged), `reloadSkills: true` (re-scan skill dirs after hooks; needed when a hook installs skills); plain-text stdout is added as context; `CLAUDE_ENV_FILE` lets it append `export` lines persisted into later Bash commands (also available in Setup, CwdChanged, FileChanged); interactive SessionStart hooks run in the background but Claude's first response waits for them — [Hooks reference: SessionStart](https://code.claude.com/docs/en/hooks#sessionstart) +- `Setup` — fires only for `claude --init-only`, `claude -p --init`, `claude -p --maintenance` (matchers `init`, `maintenance`); can't block, JSON discarded; `--init-only` runs Setup + SessionStart(startup) then exits; verify with `claude --debug-file --init-only` — [Hooks reference: Setup](https://code.claude.com/docs/en/hooks#setup) +- `InstructionsLoaded` — input `file_path`, `memory_type` (`User|Project|Local|Managed`), `load_reason` (`session_start|nested_traversal|path_glob_match|include|compact`), `globs`, `trigger_file_path`, `parent_file_path`; observability only; doesn't fire for AGENTS.md read directly — [Hooks reference: InstructionsLoaded](https://code.claude.com/docs/en/hooks#instructionsloaded) +- `UserPromptSubmit` — input `prompt` (+`session_title`); default timeout 30 s; also fires for scheduled tasks, background-subagent reports, cross-session messages; plain stdout or `additionalContext` injected as a system reminder; block with `{"decision":"block","reason":"..."}` (reason shown to user, not Claude) or exit 2; `hookSpecificOutput.sessionTitle`, `suppressOriginalPrompt` — [Hooks reference: UserPromptSubmit](https://code.claude.com/docs/en/hooks#userpromptsubmit) +- `UserPromptExpansion` — fires when a user types `/skill` directly (which bypasses `PreToolUse` on the `Skill` tool); input `expansion_type` (`slash_command|mcp_prompt`), `command_name`, `command_args`, `command_source`, `prompt`; matcher on command name — [Hooks reference: UserPromptExpansion](https://code.claude.com/docs/en/hooks#userpromptexpansion) +- `PreToolUse` — input `tool_name`, `tool_input`, `tool_use_id` (+`mcp_server` {name, source} for MCP tools, v2.1.274+); file tool paths always absolute (backslashes on Windows); does NOT fire for `@`-referenced files (use `Read` deny rules) nor `EndConversation`. Output `hookSpecificOutput.permissionDecision`: `allow` (skips prompt, but deny/ask rules still evaluated), `deny`, `ask` (prompt labelled `[settings]`/`[plugin:]`/`[skill]`; forces prompt even in auto mode), `defer` (only in `-p`; exits with `stop_reason: "tool_deferred"` and `deferred_tool_use`; resume with `claude -p --resume `); `permissionDecisionReason`; `updatedInput` (replaces entire input); `additionalContext`. Multiple hooks: `deny` > `defer` > `ask` > `allow`. Top-level `decision`/`reason` deprecated for PreToolUse (`approve`/`block` map to allow/deny). `AskUserQuestion`/`ExitPlanMode` need `allow` + `updatedInput` (with `answers`) — [Hooks reference: PreToolUse](https://code.claude.com/docs/en/hooks#pretooluse) +- Tool input shapes: Bash `{command, description, timeout, run_in_background}`; PowerShell same; Write `{file_path, content}`; Edit `{file_path, old_string, new_string, replace_all}`; Read `{file_path, offset, limit}`; Glob `{pattern, path}`; Grep `{pattern, path, glob, output_mode, -i, multiline}`; WebFetch `{url, prompt, offset}`; WebSearch `{query, allowed_domains, blocked_domains}`; Agent `{prompt, description, subagent_type, model}`; ExitPlanMode gets injected `plan`, `planFilePath` — [Hooks reference: PreToolUse input](https://code.claude.com/docs/en/hooks#pretooluse-input) +- On Windows, match `Bash|PowerShell` — on Windows without Git Bash the Bash tool isn't registered at all, so a `Bash`-only hook never fires — [Hooks reference: PowerShell](https://code.claude.com/docs/en/hooks#powershell) +- `PermissionRequest` — only when a prompt would be shown (or auto-deny in non-promptable sessions); exit 2 not honoured; output `hookSpecificOutput.decision` `{behavior: "allow"|"deny", updatedInput, updatedPermissions, message, interrupt}`; permission update entries `addRules|replaceRules|removeRules|setMode|addDirectories|removeDirectories` with `destination` `session|localSettings|projectSettings|userSettings` — [Hooks reference: PermissionRequest](https://code.claude.com/docs/en/hooks#permissionrequest) +- `PostToolUse` — input adds `tool_response`, `duration_ms`; output `decision: "block"` + `reason` (adds reason next to result), `additionalContext`, `updatedToolOutput` (must match tool's output shape, e.g. Bash `{stdout, stderr, interrupted, isImage}`), `updatedMCPToolOutput`, `classifierContext` (v2.1.236+). Not fired when Bash or external processes rewrite a file (use `FileChanged`). Foreground `Agent` calls expose `tool_response.{status, agentId, content, resolvedModel, totalTokens, totalDurationMs, totalToolUseCount, usage}` — [Hooks reference: PostToolUse](https://code.claude.com/docs/en/hooks#posttooluse) +- `Notification` — input `message`, `title`, `notification_type`; can't block; `terminalSequence` still emitted — [Hooks reference: Notification](https://code.claude.com/docs/en/hooks#notification) +- `SubagentStart` — input `agent_id`, `agent_type`; can't block; `additionalContext` injected into the subagent — [Hooks reference: SubagentStart](https://code.claude.com/docs/en/hooks#subagentstart) +- `SubagentStop` — input `stop_hook_active`, `agent_id`, `agent_type`, `agent_transcript_path`, `last_assistant_message`, `background_tasks`, `session_crons`; same decision format as Stop (`decision: "block"` keeps subagent running with `reason` as next instruction); also fires for internal agents (empty `agent_type`) — [Hooks reference: SubagentStop](https://code.claude.com/docs/en/hooks#subagentstop) +- `Stop` — not on user interrupt (API errors fire `StopFailure`); input `stop_hook_active`, `last_assistant_message`, `background_tasks`, `session_crons`; `{"decision":"block","reason":"..."}` keeps Claude working (reason required), or `hookSpecificOutput.additionalContext` for non-error feedback; **8-consecutive-continuation cap** (`CLAUDE_CODE_STOP_HOOK_BLOCK_CAP`), reset when Claude calls a tool; check `stop_hook_active` to avoid loops; `/goal` is a built-in session-scoped prompt Stop hook — [Hooks reference: Stop](https://code.claude.com/docs/en/hooks#stop) +- `PreCompact` — matcher `manual|auto`; input `trigger`, `custom_instructions`; exit 2 or `decision: "block"` blocks compaction. `PostCompact` — input `trigger`, `compact_summary`; no decision — [Hooks reference: PreCompact](https://code.claude.com/docs/en/hooks#precompact) +- `SessionEnd` — `reason` `clear|resume|logout|prompt_input_exit|other` (`bypass_permissions_disabled` removed v2.1.234); no decision; **1.5 s default budget** (raise via per-hook `timeout` up to 60 s or `CLAUDE_CODE_SESSIONEND_HOOKS_TIMEOUT_MS`) — [Hooks reference: SessionEnd](https://code.claude.com/docs/en/hooks#sessionend) + +**Hook types per event** — [Hooks reference: Prompt-based hooks](https://code.claude.com/docs/en/hooks#prompt-based-hooks) +- All five types: `PermissionDenied`, `PostToolBatch`, `PostToolUse`, `PostToolUseFailure`, `PreToolUse`, `Stop`, `SubagentStop`, `TaskCompleted`, `TaskCreated`, `TeammateIdle`, `UserPromptExpansion`, `UserPromptSubmit`. `PermissionRequest`: all except `agent`. Only command/http/mcp_tool: `ConfigChange`, `CwdChanged`, `DirectoryAdded`, `Elicitation`, `ElicitationResult`, `FileChanged`, `InstructionsLoaded`, `MessageDisplay`, `Notification`, `PostCompact`, `PostModelSwitch`, `PreCompact`, `PreModelSwitch`, `SessionEnd`, `StopFailure`, `SubagentStart`, `WorktreeCreate`, `WorktreeRemove`. `SessionStart`/`Setup`: command and mcp_tool only. +- Prompt hook: `{"type":"prompt","prompt":"... $ARGUMENTS ...","model":?,"timeout":30,"continueOnBlock":false}`; model must answer `{"ok": true|false, "reason": "...", "impossible": true|false}`; on Stop/SubagentStop `ok:false` feeds reason back (unless `impossible`). +- Agent hook (**experimental**): same fields, 60 s default, up to 50 turns with Read/Grep/Glob etc., no `continueOnBlock` — [Hooks reference: Agent-based hooks](https://code.claude.com/docs/en/hooks#agent-based-hooks) +- Async: `"async": true` (command only) — can't block; `additionalContext`/`systemMessage` delivered next turn; killed at `-p` teardown; `asyncRewake` wakes Claude on exit 2 — [Hooks reference: Run hooks in the background](https://code.claude.com/docs/en/hooks#run-hooks-in-the-background) + +**Security / trust** — [Hooks reference: Workspace trust](https://code.claude.com/docs/en/hooks#workspace-trust) +- Interactive: hooks from every settings file (even `~/.claude/settings.json`) are held back until the workspace trust dialog is accepted. `-p`/SDK: never shows the dialog and **treats the folder as trusted**, so a repo's `.claude/settings.json` hooks run. Mitigations: `--bare`, `--settings '{"disableAllHooks": true}'`. + +**Windows** — [Hooks reference: Windows PowerShell tool](https://code.claude.com/docs/en/hooks#windows-powershell-tool) +- `"shell": "powershell"` (auto-detects `pwsh.exe`, falls back to `powershell.exe`); placeholders rewritten to `${env:NAME}`; never write bare `$CLAUDE_PROJECT_DIR` in PowerShell; example `"command": "& \"$env:CLAUDE_PROJECT_DIR\\.claude\\hooks\\check.ps1\""`. + +**Debugging** — [Hooks reference: Debug hooks](https://code.claude.com/docs/en/hooks#debug-hooks) +- `claude --debug-file ` or `claude --debug` (log at `~/.claude/debug/.txt`; doesn't print to terminal); v2.1.296+ logs `Hooks: PostToolUse:Write [] finished with status 0 (31ms)`; `CLAUDE_CODE_DEBUG_LOG_LEVEL=verbose` for matcher details. + + +**Version history (changelog)** — [Changelog](https://code.claude.com/docs/en/changelog) +- Hooks released: v1.0.38 (Jun 30, 2025); Stop split into Stop/SubagentStop: v1.0.41 (Jul 3, 2025); SessionStart: v1.0.62 (Jul 28, 2025); HTTP hooks: v2.1.63 (Feb 28, 2026); PermissionDenied: v2.1.89 (Apr 1, 2026); `onFailure`: v2.1.295 (per hooks reference). + +**Real-world example (official marketplace)** — `plugins/hookify/hooks/hooks.json` in anthropics/claude-plugins-official uses matcher-less groups — [hookify hooks.json](https://github.com/anthropics/claude-plugins-official/blob/main/plugins/hookify/hooks/hooks.json): +```json +{"description": "Hookify plugin - User-configurable hooks from .local.md files", + "hooks": {"PreToolUse": [{"hooks": [{"type": "command", "command": "python3 \"${CLAUDE_PLUGIN_ROOT}/hooks/pretooluse.py\"", "timeout": 10}]}], + "PostToolUse": ["..."], "Stop": ["..."], "UserPromptSubmit": ["..."]}} +``` + +### Inferences +- For a kit's enforcement hooks: always `exit 2` (never `exit 1`) or exit 0 with a JSON deny, set `"onFailure": "block"` (v2.1.295+) so a broken script fails closed, prefer exec form with `${CLAUDE_PROJECT_DIR}`/`${CLAUDE_PLUGIN_ROOT}` in `args`, and keep stdout clean of shell-profile output. + +- Plugin hooks merge with user/project hooks and inherit no special trust beyond the plugin's own install; project-skill hook trust is laxer than project-subagent hook trust, so a kit that must run hooks in untrusted `-p` CI should put them in a skill or in settings rather than in a project subagent's frontmatter. + +### Gaps +- Input/decision schemas for the less common events (`PostToolUseFailure`, `PostToolBatch`, `PermissionDenied`, `TaskCreated`, `TaskCompleted`, `TeammateIdle`, `ConfigChange`, `CwdChanged`, `DirectoryAdded`, `FileChanged`, `WorktreeCreate`, `WorktreeRemove`, `Pre/PostModelSwitch`, `Elicitation`, `ElicitationResult`, `MessageDisplay`, `StopFailure`) were listed but not transcribed field by field; they are in the hooks reference at https://code.claude.com/docs/en/hooks#hook-events. + +## Settings and permissions + +### Takeaway +Settings are strict-JSON files (no comments/trailing commas) with precedence **managed > `--settings` (command line) > `.claude/settings.local.json` > `.claude/settings.json` > `~/.claude/settings.json`**; scalar keys take the highest value, list keys such as `permissions.allow` merge. Permission rules are `Tool` or `Tool(specifier)`, evaluated **deny → ask → allow** (first match wins, any scope's deny beats any allow). Several keys are ignored or delayed in a repository's `.claude/settings.json`: `permissions.allow`, `additionalDirectories`, `extraKnownMarketplaces` and most `env` wait for workspace trust; `defaultMode: "auto"`/`"bypassPermissions"` never take effect from project/local files. Validate with the published schema `https://json.schemastore.org/claude-code-settings.json`, `/status` and `claude doctor`. + +### Cited Findings +**Files and precedence** — [Settings](https://code.claude.com/docs/en/settings#settings-precedence) +1. Managed (`managed-settings.json` + `managed-settings.d/*.json` in `/Library/Application Support/ClaudeCode/` (macOS), `/etc/claude-code/` (Linux/WSL), `C:\Program Files\ClaudeCode\` (Windows); MDM plist domain `com.anthropic.claudecode`; `HKLM\SOFTWARE\Policies\ClaudeCode` value `Settings`; server-managed from claude.ai console) — [Managed settings](https://code.claude.com/docs/en/managed-settings) +2. Command line `--settings ` (one session) +3. Project local `.claude/settings.local.json` (Claude Code adds `**/.claude/settings.local.json` to global git excludes when it first writes it; read/written at the git repo root since v2.1.211, main checkout root in worktrees) +4. Shared project `.claude/settings.json` (read from the session's primary working directory — no parent fallback) +5. User `~/.claude/settings.json` (`%USERPROFILE%\.claude` on Windows; relocatable with `CLAUDE_CONFIG_DIR`) +- `~/.claude.json` is Claude Code's own state file (sign-in, MCP server configs for user/local scope, per-project trust, global config keys) — [Settings](https://code.claude.com/docs/en/settings#find-or-create-your-settings-files) +- "Lists merge instead of overriding" (exceptions: `fallbackModel`, `modelPicker`, managed `availableModels`, `modelSettings`) — [Settings](https://code.claude.com/docs/en/settings#lists-merge-instead-of-overriding) +- Env vars are not a precedence level; each var/key pair decides (e.g. `ANTHROPIC_MODEL` beats `model`) — [Settings](https://code.claude.com/docs/en/settings#settings-precedence) +- Files are watched and most edits (incl. `permissions`, `hooks`) apply mid-session; `model`, `effortLevel` read once at start — [Settings: When edits take effect](https://code.claude.com/docs/en/settings#when-edits-take-effect) +- Broken file → interactive "Settings Error" dialog; bad individual entries → "Settings Warning" and skipped; `-p` silently skips (check `claude doctor`); `/status` "Setting sources" line lists loaded files — [Settings: Fix a broken settings file](https://code.claude.com/docs/en/settings#fix-a-broken-settings-file) +- A committed key may never reach teammates if its Scope is `User`, `User or managed`, `Managed`, or `Global config`; and `permissions.allow`, `permissions.additionalDirectories`, `extraKnownMarketplaces`, most `env` values apply only after each teammate trusts the folder (deny/ask apply immediately) — [Settings: A committed key doesn't reach teammates](https://code.claude.com/docs/en/settings#a-committed-key-doesn-t-reach-teammates) +- Minimal file with schema: +```json +{ + "$schema": "https://json.schemastore.org/claude-code-settings.json", + "permissions": { + "allow": ["Bash(npm run lint)", "Bash(npm run test *)"], + "deny": ["Read(./.env)", "Read(./.env.*)"] + } +} +``` +(The schema "can lag behind the newest CLI releases") — [Settings: Edit a settings file](https://code.claude.com/docs/en/settings#edit-a-settings-file) + +**Permission rules** — [Permissions](https://code.claude.com/docs/en/permissions#permission-rule-syntax) +- Arrays: `permissions.allow`, `permissions.ask`, `permissions.deny`; also `permissions.defaultMode`, `permissions.additionalDirectories`, `permissions.disableBypassPermissionsMode: "disable"`, `permissions.disableAutoMode: "disable"`, `permissions.blockReadsOutsideWorkingDirectories` — [Settings reference index](https://code.claude.com/docs/en/settings-reference#settings-index) +- Order: deny, then ask, then allow; "An allow rule can't carve an exception out of a deny rule"; a bare-tool deny (e.g. `Bash`) removes the tool from Claude's context — [Permissions: Manage permissions](https://code.claude.com/docs/en/permissions#manage-permissions) +- Syntax: `Bash` = `Bash(*)` (all); `Bash(npm run build)` exact; `Bash(npm run *)` prefix (space before `*` matters: `Bash(ls *)` doesn't match `lsof`; trailing ` *` also matches bare command); `Bash(ls:*)` legacy equivalent of trailing ` *`; put `*` after the subcommand (startup warning for `Bash(git * main)` style allow rules) — [Permissions: Wildcard patterns](https://code.claude.com/docs/en/permissions#wildcard-patterns) +- Compound commands split on `&&`, `||`, `;`, `|`, `|&`, `&`, newlines; each subcommand must match an allow rule; deny/ask match any subcommand incl. `$()`; wrappers `timeout`, `time`, `nice`, `nohup`, `stdbuf`, `command`, `builtin`, `noglob`, bare `xargs` stripped; `npx`, `docker exec`, `devbox run` etc. not stripped — [Permissions: Bash](https://code.claude.com/docs/en/permissions#bash) +- Bash deny rules are not a security boundary (`/usr/bin/curl`, `sh -c '...'`, `git -C . push` bypass them); use sandboxing or PreToolUse hooks — [Permissions: What a Bash rule doesn't match](https://code.claude.com/docs/en/permissions#bash-rule-limits) +- Parameter rules (deny/ask only) `Tool(param:value)`: `Agent(model:opus)`, `Agent(isolation:worktree)`, `Bash(run_in_background:true)`; not for primary fields like `command`/`file_path` — [Permissions: Match by input parameter](https://code.claude.com/docs/en/permissions#match-by-input-parameter) +- Tool-name globs: deny/ask accept `"*"`, `"mcp__*"`; allow globs only after a literal `mcp____` prefix (e.g. `mcp__github__get_*`) — [Permissions: Tool name wildcards](https://code.claude.com/docs/en/permissions#tool-name-wildcards) +- Read/Edit rules use gitignore syntax: `//path` absolute, `~/path` home, `/path` relative to the **settings source** (project settings → primary working dir; user settings → `~/.claude/`), `path`/`./path` relative to cwd; `Read(.env)` ≡ `Read(**/.env)`; single-segment `src/**` matches any depth only in deny/ask; `!` negation carves out within the same source; only `Edit(...)` and `Read(...)` path rules are consulted (`Write(...)` path rules are ignored with a warning, v2.1.210+); `.claudeignore` has no effect — [Permissions: Read and Edit](https://code.claude.com/docs/en/permissions#read-and-edit) +- Other forms: `WebFetch(domain:example.com)`, `WebFetch(domain:*.example.com)`; `mcp__puppeteer`, `mcp__puppeteer__*`, `mcp__puppeteer__puppeteer_navigate`; `Agent(Explore)`; `Skill(name)`/`Skill(name *)`; `Cd(~/code/**)` for `/cd` — [Permissions: Tool-specific permission rules](https://code.claude.com/docs/en/permissions#tool-specific-permission-rules) +- PreToolUse hook decisions don't bypass deny/ask rules; a hook exiting 2 blocks even when an allow rule matches — [Permissions: Extend permissions with hooks](https://code.claude.com/docs/en/permissions#extend-permissions-with-hooks) +- `--add-dir` grants file access and loads that dir's `.claude/skills/`, `.claude/commands/`, `.claude/agents/`, and only `enabledPlugins`/`extraKnownMarketplaces` from its settings; `permissions.additionalDirectories` grants file access only — [Permissions: Additional directories](https://code.claude.com/docs/en/permissions#additional-directories-grant-file-access-not-configuration) +- Workspace trust: interactive sessions show the trust dialog; `-p`/SDK never shows it — repo hooks, `env`, skill `allowed-tools` are **used**, but `.claude/settings.json` `permissions.allow` and `additionalDirectories` are **not used** (stderr warning `this workspace has not been trusted`), project-subagent frontmatter hooks, `@skills-dir` plugins, and repo `extraKnownMarketplaces` are not used, `.mcp.json` servers connect without asking. Manual trust: `projects[""].hasTrustDialogAccepted: true` in `~/.claude.json` — [Permissions: What runs before you trust a folder](https://code.claude.com/docs/en/permissions#what-runs-before-you-trust-a-folder) + +**Permission modes (`defaultMode` values)** — [Permission modes](https://code.claude.com/docs/en/permission-modes#available-modes), [Settings reference: permissions.defaultMode](https://code.claude.com/docs/en/settings-reference#permissions-defaultmode) +| Mode | Runs without asking | +|---|---| +| `default` (UI "Manual"; alias `manual`) | reads only | +| `acceptEdits` | reads, file edits, `mkdir`, `touch`, `rm`, `rmdir`, `mv`, `cp`, `sed` within working dirs | +| `plan` | reads (+ classifier-approved commands when auto mode available) | +| `auto` | everything, with background classifier checks | +| `dontAsk` | reads in working dirs + pre-approved tools; everything else denied (CI) | +| `bypassPermissions` | everything (isolated containers/VMs only) | +- Session start mode: `--permission-mode`/`--dangerously-skip-permissions` > `permissions.defaultMode` > built-in default (`auto` in terminal from v2.1.283; `default` for `claude -p` in sessions that fetch feature flags). `"auto"`/`"bypassPermissions"` in `.claude/settings.json` or `.claude/settings.local.json` don't take effect — [Permission modes: Which mode a session starts in](https://code.claude.com/docs/en/permission-modes#which-mode-a-session-starts-in) +- Deny rules block in every mode incl. bypass; never auto-approved in any mode: explicit ask rules, `AskUserQuestion`, MCP `requiresUserInteraction` tools, `rm`/`rmdir` of critical paths — [Permission modes](https://code.claude.com/docs/en/permission-modes#actions-no-mode-auto-approves) +- Protected paths (writes never auto-approved except bypass): `.git`, `.config/git`, `.vscode`, `.idea`, `.husky`, `.cargo`, `.devcontainer`, `.yarn`, `.mvn`, `.claude` (exceptions incl. `.claude/worktrees/`, plan files); `permissions.allow` such as `Edit(.claude/**)` does not pre-approve them — [Permission modes: Protected paths](https://code.claude.com/docs/en/permission-modes#protected-paths) +- `Shift+Tab` cycles modes (`default` → `acceptEdits` → `plan` → optional bypass/auto) — [Permission modes: Switch permission modes](https://code.claude.com/docs/en/permission-modes#switch-permission-modes) + +**Sandbox** — [Settings reference: Sandbox settings](https://code.claude.com/docs/en/settings-reference#sandbox-enabled) +- `{"sandbox": {"enabled": true}}` sandboxes Bash (macOS, Linux, WSL2; Linux needs `bubblewrap` + `socat`); falls back to unsandboxed unless `sandbox.failIfUnavailable: true`. Sub-keys: `autoAllowBashIfSandboxed` (default true), `allowUnsandboxedCommands`, `excludedCommands`, `filesystem.{allowRead,allowWrite,denyRead,denyWrite}`, `network.{allowedDomains,deniedDomains,allowLocalBinding,allowUnixSockets,httpProxyPort,socksProxyPort,strictAllowlist}`, `credentials.{envVars,files}`, `enableWeakerNestedSandbox`, `ignoreViolations`. Permissions and sandbox paths/domains merge into the final sandbox config; `WebFetch(domain:...)` rules also feed the sandbox network list — [Permissions: How permissions interact with sandboxing](https://code.claude.com/docs/en/permissions#how-permissions-interact-with-sandboxing) + +**Other kit-relevant keys** — [Settings reference](https://code.claude.com/docs/en/settings-reference#settings-index) +- `env`: `{"env": {"DISABLE_AUTO_COMPACT": "1"}}`; project/local `env` applies after trust (or at startup in `-p`); project/local can't set `CLAUDE_CONFIG_DIR`, `HOME`, `TMPDIR`, `XDG_*` etc. — [Settings reference: env](https://code.claude.com/docs/en/settings-reference#env) +- `statusLine`: `{"type": "command", "command": "...", "padding": 2, "refreshInterval": , "hideVimModeIndicator": bool}`; receives session JSON on stdin (e.g. `.model.display_name`, `.context_window.used_percentage`); `subagentStatusLine` similar — [Settings reference: statusLine](https://code.claude.com/docs/en/settings-reference#statusline) +- `outputStyle`: name of built-in (e.g. `"Explanatory"`, `"Learning"`) or custom style; mid-session change applies from next message (v2.1.251+) — [Settings reference: outputStyle](https://code.claude.com/docs/en/settings-reference#outputstyle) +- `agent` (start every session as a named subagent), `hooks`, `disableAllHooks`, `enabledPlugins`, `extraKnownMarketplaces`, `skillOverrides`, `disableSkillShellExecution`, `claudeMdExcludes`, `autoMemoryEnabled`, `autoMemoryDirectory`, `plansDirectory`, `includeGitInstructions`, `attribution.{commit,pr}` (replaces deprecated `includeCoAuthoredBy`), `language`, `model`, `effortLevel`, `cleanupPeriodDays`, `worktree.{baseRef,symlinkDirectories,sparsePaths}`; managed-only: `allowManagedHooksOnly`, `allowManagedPermissionRulesOnly`, `strictKnownMarketplaces`, `blockedMarketplaces`, `strictPluginOnlyCustomization.{skills,agents,hooks,mcp}` — [Settings reference index](https://code.claude.com/docs/en/settings-reference#settings-index) + +**Output styles** — [Output styles](https://code.claude.com/docs/en/output-styles#create-a-custom-output-style) +- Markdown file in `~/.claude/output-styles/`, `.claude/output-styles/` (every one from cwd up to repo root; closest wins), managed dir `.claude/output-styles/`, or a plugin's `output-styles/`; file name = style name unless `name` set. Built-ins: `Proactive`, `Concise` (v2.1.237+), `Explanatory`, `Learning` (+ Default); `outputStyle` value is case-sensitive (mismatch → Default). +- Frontmatter (all optional, misspelled fields ignored): `name`, `description`, `keep-coding-instructions` (default `false` — without it the software-engineering part of the system prompt is dropped), `force-for-plugin` (plugin styles only; applies automatically, overrides user's `outputStyle`). +- Style files are read at startup (restart after editing); `/output-style