Skip to content

docs: record mutation-harness and heading-assertion pitfalls - #6

Merged
mxuanvan02 merged 1 commit into
mainfrom
docs/mutation-harness-pitfalls
Sep 4, 2026
Merged

mxuanvan02 merged 1 commit into
mainfrom
docs/mutation-harness-pitfalls

Conversation

@mxuanvan02

Copy link
Copy Markdown
Owner

Three lessons measured while adding the third gate, each of which produced a wrong conclusion before it was caught:

  1. A mutation harness must verify it actually changed bytes on disk. Three mutations were reported as "not caught" when the script had silently failed to write; the assertions were sound. A harness that cannot prove it mutated is evidence of nothing.
  2. assertIn("Gate name", rubric) cannot detect a deleted heading when the same string also appears in a link on the next line. Pin the heading form (## Gate name) so the assertion fails when the section is removed.
  3. A class-level detector test passes while any one of its patterns fires, so a single deleted regex leaves the suite green. Pin the pattern inventory by identity; the inventory is the contract.

Also bumps SKILL.md to 3.5.1.

Three lessons measured while adding the third gate, each of which produced a
wrong conclusion before it was caught:

1. A mutation harness must verify it actually changed bytes on disk. Three
   mutations were reported as "not caught" when the script had silently failed
   to write; the assertions were sound. A harness that cannot prove it mutated
   is evidence of nothing.
2. `assertIn("Gate name", rubric)` cannot detect a deleted heading when the same
   string also appears in a link on the next line. Pin the heading form
   (`## Gate name`) so the assertion fails when the section is removed.
3. A class-level detector test passes while any one of its patterns fires, so a
   single deleted regex leaves the suite green. Pin the pattern inventory by
   identity; the inventory is the contract.

Also bumps SKILL.md to 3.5.1.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@mxuanvan02
mxuanvan02 merged commit b3adfea into main Sep 4, 2026
2 checks passed
@mxuanvan02
mxuanvan02 deleted the docs/mutation-harness-pitfalls branch September 4, 2026 06:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant