Repository navigation
fail_on_miss fails projects that document the topic where the step doesn't look #702
Description
Follow-up to #667, which applies once it lands. #665 needs a pattern miss to be a FAIL so remediation creates the missing document, but fail_on_miss also concludes FAIL for a project that documents the topic somewhere the step doesn't look, or in words the regex doesn't match. Those are false negatives, which the conservative rule tolerates, but they tell a project it lacks documents it has, and remediation then adds more.
Controls with fail_on_miss in #667: OSPS-DO-03.02, DO-04.01, DO-05.01, DO-06.01, QA-06.02, SA-01.01.
Measured on throwaway repositories with darnit audit, on main (075556e) and on 36b525df, where the documents hold text that matches each control's regex. Main is PENDING for all of these.
| repository | FAIL on 36b525df |
|---|---|
| README.md holding the text (control) | SA-01.01 only, which it doesn't cover |
| README.rst, text in CONTRIBUTING.rst | all six |
| README.md, text in .github/CONTRIBUTING.md | all six |
| README.adoc holding the text | all six |
| README.md, dependency policy in docs/dependency-policy.md | DO-06.01 is the false FAIL (the fixture documents nothing else) |
Three related gaps:
- Regex-only misses. A README.rst that does document the policy, wrapped across two lines ("We manage dependencies with pip-tools; each library is pinned in / requirements.txt and updated weekly"), fails DO-06.01. The regex needs a noun followed by a keyword on one line.
- Remediation writes beside the existing document. With the text in .github/CONTRIBUTING.md, an applied remediation creates docs/DEPENDENCIES.md, docs/TESTING.md, docs/RELEASE-VERIFICATION.md, SUPPORT.md and ARCHITECTURE.md. The existing file is left alone.
- community-spec. The handler change moves CSL-06.01 from WARN to FAIL on a repo with only README.txt, which its list omits (it does list README.md, readme.md and README.rst).
The corpus has no README.rst fixture, so its "every new FAIL conclusion is correct" result doesn't cover any of this.
Possible directions, not yet decided:
- Widen the lists:
.rstand.github/forms of CONTRIBUTING, SECURITY and SUPPORT, plusdocs/dependency-policy.md, and add corpus fixtures for these shapes. - Conclude FAIL only when no listed file exists, and send a miss on an existing file to the model. That gives back the judgment, but remediation then wouldn't plan a document for a project whose existing files don't cover the topic.
Drafted with AI assistance (Claude), reviewed and filed by me. Measurements quoted were executed and verified.
Activity