Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
332 changes: 80 additions & 252 deletions .claude/workflows/docs-accuracy-audit.js

Large diffs are not rendered by default.

22 changes: 12 additions & 10 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1871,17 +1871,19 @@ jobs:
- name: Template version-time rewriter self-test
run: pnpm check:template-version-sync

# #4851: the docs-accuracy-audit workflow carries its default scope inline
# #4851: the docs-accuracy-audit workflow's default scope is a generated list
# (a workflow script runs in a vm with no filesystem, so it cannot enumerate
# content/docs/ itself). Hand-kept, that list rotted in BOTH directions —
# 16 entries pointing at files that no longer existed after the
# protocol/objectos → protocol/kernel rename, and 48 existing docs missing
# from it — while every "FULL audit" run reported green, which is how the
# accuracy defects in #4781 and #4817 survived ~2 months. This regenerates
# the list from the filesystem and fails when the two disagree either way.
# It lives in this job deliberately: the change that breaks the list is a
# docs rename, so a `packages/**` paths filter would blind the gate to
# exactly its own failure mode.
# content/docs/ itself — the caller hands the list in). Hand-kept, that list
# rotted in BOTH directions — 16 entries pointing at files that no longer
# existed after the protocol/objectos → protocol/kernel rename, and 48
# existing docs missing from it — while every "FULL audit" run reported
# green, which is how the accuracy defects in #4781 and #4817 survived
# ~2 months. This derives the list from the filesystem and fails when it and
# scripts/docs-audit/handwritten-docs.json disagree either way; it also runs
# the workflow against stub agents to check the body still consumes what it
# is handed. It lives in this job deliberately: the change that breaks the
# list is a docs rename, so a `packages/**` paths filter would blind the gate
# to exactly its own failure mode.
- name: Docs-audit scope is derived, not hand-kept
run: pnpm check:docs-audit-scope

Expand Down
47 changes: 35 additions & 12 deletions scripts/docs-audit/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -456,17 +456,33 @@ node scripts/docs-audit/check-audit-scope.mjs --write # regenerate the list fr
node scripts/docs-audit/check-audit-scope.mjs --self-test
```

The `docs-accuracy-audit` workflow (part 3) carries its default scope **inline**, as
`ALL_HANDWRITTEN`. It has to: a workflow script runs inside a `node:vm` context whose
only globals are `log`/`phase`/`console`/`budget`/timers plus
`agent`/`parallel`/`pipeline`/`workflow`/`args`, with code generation disabled — no
`require`, no `import`, no filesystem. It can neither walk `content/docs/` nor read a
JSON artifact, so the list cannot be derived *at run time*.
**THE single source is `scripts/docs-audit/handwritten-docs.json`** — one generated file,
with two consumers: this gate, and whoever invokes the `docs-accuracy-audit` workflow
(part 3), who reads it and hands its `docs` array in as `args.handwritten`.

It is therefore derived at *generation* time instead: `--write` rewrites the block from
The workflow body cannot read it itself. A workflow script runs inside a `node:vm`
context whose only globals are `log`/`phase`/`console`/`budget`/timers plus
`agent`/`parallel`/`pipeline`/`workflow`/`args`, with code generation disabled — no
`require`, no `import`, no filesystem. It can neither walk `content/docs/` nor open a
JSON artifact. But it does not have to: `args` **is** an injection channel, delivered
verbatim from the invocation, so the read happens in the **caller**, outside the sandbox,
and the list arrives as data.

The list used to live inline in the workflow body, as `ALL_HANDWRITTEN`. `.claude/**` is
a governed surface (human-merge-only, never armed, never queued), so adding one customer
documentation page forced a governed edit — through a bookkeeping list that merely
happened to live there, and invisibly: nothing in such a card's file list showed a
governed path until this gate ran. Maintainer ruling, 2026-09-01, verbatim 「同意」: move
it off. ⚠️ The governed register itself is **unchanged** — narrowing it was the option
that ruling explicitly rejected, and this file is no precedent for relocating anything
else out of `.claude/**`.

The list is derived at *generation* time: `--write` rewrites the artifact from
`affected-docs.mjs --all` (one definition of "hand-written doc", not two), and the plain
run is a CI gate in `lint.yml` that fails when the block and `content/docs/` disagree
**in either direction**.
run is a CI gate in `lint.yml` that fails when the artifact and `content/docs/` disagree
**in either direction**. It also runs the workflow against stub agents and checks that the
body still *consumes* what it is handed — an artifact in sync with `content/docs/` proves
nothing about a body that has stopped reading it, and that failure would be silent.

Both directions matter, and only one had ever been noticed (#4851):

Expand Down Expand Up @@ -642,15 +658,22 @@ a consumer, add it to that list.**
A Claude Code multi-agent workflow (`.claude/workflows/docs-accuracy-audit.js`). For each
doc: an agent reads it, locates the real implementation, and applies evidence-backed
fixes in place; a second **adversarial verifier** re-checks every fix against the code and
repairs over-corrections. Scope it with `args.docs`; omit for a full audit.
repairs over-corrections. Scope it with `args.docs`; for a full audit hand in the whole
set as `args.handwritten`.

```js
// scoped to the docs a code change touched:
Workflow({ name: 'docs-accuracy-audit', args: { docs: [/* output of affected-docs.mjs */] } })
// full audit of all hand-written docs:
Workflow({ name: 'docs-accuracy-audit' })
// full audit of all hand-written docs — read the artifact first, OUTSIDE the sandbox:
// node -e "console.log(JSON.stringify(require('./scripts/docs-audit/handwritten-docs.json').docs))"
Workflow({ name: 'docs-accuracy-audit', args: { handwritten: [/* that array */] } })
```

⛔ There is no "omit `args` and audit everything" invocation, and there cannot be: the
body has no filesystem, so with nothing handed in it does not know what "everything" is.
It refuses by name rather than inventing a scope — the two shapes it could invent are a
silent audit of nothing and a stale list, and both report success.

It edits files in place (frontmatter preserved, no moves) and returns a per-doc log of
fixes, verifier repairs, and residual items that couldn't be confirmed against code —
**except** for `content/docs/releases/**`, which is reviewed read-only and returns
Expand Down
2 changes: 1 addition & 1 deletion scripts/docs-audit/affected-docs.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -236,7 +236,7 @@ let diffBaseRef = null;
* the audit scoping command still returns them and they keep getting audited. #4920
* considered excluding them and REJECTED it: the most-read pages in the docs would go
* permanently unaudited and silently, and a second definition of "docs this tooling
* covers" would grow next to the generated block. What forks is the DELIVERABLE — the
* covers" would grow next to the generated scope artifact. What forks is the DELIVERABLE — the
* drift comment tells the reader to file an issue instead of editing (#6893: a comment
* listing `content/docs/releases/v17.mdx` next to editable pages steers a dev who
* treats the list as a worklist straight into the one edit the repo forbids).
Expand Down
Loading
Loading