[johnml1135]

Start here: conformance/docs/what-it-is.md, then .github/workflows/conformance.yml: what the suite claims and the gate that enforces it.

Status: stacked on #500 — merge in order #556 → #500 → #480 → #578.

Status: CI is red at head 262ab300. Both Build jobs fail ConformanceFixtureGateTests.EveryConformanceFixturePassesSelfCheck on four edge-cases/truncate-morphotactic* fixtures (word analyses differ from expectation), and both Conformance jobs fail, with the workflow reporting conformance/coverage.csv or rules.csv stale against the fixtures. The branch is behind master. Work is held until after the Hermit Crab Optimizations release; nothing stacks on this PR.

A conformance suite of synthetic HermitCrab grammars, run in-process with HermitCrab as the oracle, with CI gates for parse results, coverage claims and generated ledgers. The latest revision adds named parser-settings variants: the runner builds a fresh morpher per run, applies the variant's settings, and compares complete parse-identity multisets and statuses against that run's own expectations. Variants earn no fixture or semantic-coverage credit.

Why is it so big? The change is about 940 files. Harness code and its tests are the reviewable part; the rest is fixtures, generated ledgers and evidence cards, and docs. Generated files are rewritten by hc-conformance --write-* and pinned by tests, so check freshness, do not read line by line. No parser engine changed.

Where to look:

• .github/workflows/conformance.yml: the gate (self-check, coverage report, freshness of generated CSVs).
• conformance/dead-rules-allowed.tsv: a rule no word exercises fails the coverage report unless listed with a reason.
• conformance/PROTOCOL.md and WordsYamlLoader.cs: the adapter contract; settings are closed, so an unsupported setting fails rather than being silently ignored. Pinned by WordsYamlMorpherSettingsTests.
• src/SIL.Machine.Morphology.HermitCrab.Conformance/Runner.cs: self-check.
• global.json: new; pins SDK 10.0.100 with latestFeature roll-forward for the whole repository.

Deliberately not here: MaxAlternatives (PanGloss lacks HC's typed overflow behaviour) and MergeMSAs (needs the FieldWorks project loader) are unsupported variant settings. MergeAnalyses verifies output preservation only, not merge-fold counts. Suite-side work for #534 (open) waits for its merge.

Verification: the earlier numbers (local_check.sh --agent-strict, C# and PanGloss variant matrices) were taken at 184bcd46, not at the current head, and are not repeated here; the red jobs above are the current evidence. Broader Rust workspace and Foma suites were not run.

Next: fix the four truncate-morphotactic* fixtures and regenerate the CSVs, then rebase; or say if the red gate should block holding this branch.

---

Reading this a year from now -- start here

The suite is an executable adequacy argument: synthetic grammars whose expected parse-identity multisets are checked against HermitCrab (self-check) and against a second engine (PanGloss) through an adapter protocol. The decisions below explain why the gates look the way they do.

Decisions and why

• Variants as run-level settings. Settings come from constructor defaults and per-call GuessRoots; supported: DelReapps, MaxRoots, MakePartialsFinal, MergeAnalyses, GuessRoots. The main risk is claiming support for a setting an adapter cannot apply or treating incomplete output as an empty result, so unsupported settings, missing rows and zero-comparison variants fail, and PanGloss also rejects capped or timed-out output.
• Source-generator loading. PCRE.NET ships an IIncrementalGenerator; the compilation-graph loader admits it only by content hash, and a probe runs it and fails closed on any generated output.
• Crash fixtures. An expect_crash fixture passes only if the same engine completes an ordinary fixture in the same run, so a broken adapter command cannot pass as a crash (conformance/PROTOCOL.md).
• Binary .fwdata. Marked binary in .gitattributes because its pinned hash depended on core.autocrlf. Four fixture grammars keep trailing whitespace because their hashes are pinned in the manifest. Hand-maintained fixture totals were removed from docs; the manifest is the count.
• Trace.FailureAllomorph dropped. An engine API change does not belong in a test-suite PR; Trace HermitCrab failure reasons and template slot outcomes #534 adds the same fact through its own trace field.
• ci.yml back to master. Its added fixture run duplicated ConformanceFixtureGateTests under dotnet test.

Surprising findings

• The coverage report could not fail; it now gates dead rules and uncovered constructs.
• The counterfactual sweep read mutant output in the console code page, so a mutation that changed nothing looked like evidence. It now reads UTF-8 (CounterfactualGate, with a regression test). Regenerating the ledger demoted two surfaces that had only that false evidence (Gloss to Unobservable, HeadFeatures to RequiredByDtd).
• local_check.sh ran --semantic-coverage, which exits 1 by design while the catalog is incomplete, so it could never pass; it now runs the same gates as conformance.yml.
• chained-output-feature-override-loss claimed to fail on master on purpose; it has passed since Change Add to PriorityUnion #494 (merged).

Deferred, and what would unblock it

#540, #522 and #519 are merged and #543 is closed; their suite-side work is already in this branch or no longer stacks here. #534 is open: once it merges, FailureRuleAttributor should credit through its new trace field and the FieldWorks-producibility ledger needs rows for its four new failure reasons. #543 would have added fixtures for the #504 and #507 evidence gaps; they are not in this branch. Unsupported variant settings (MaxAlternatives, MergeMSAs) need PanGloss overflow behaviour and a FieldWorks project loader respectively.

Evidence and oracle observations

C# HC observations for the variants were collected on 2026-10-22 with the engine unchanged; the five measured fixture/settings pairs cover deletion reapplications (DelReapps), compounding breadth (MaxRoots), final-template partial discriminators (MakePartialsFinal), the final-template merge key (MergeAnalyses) and guess-roots (GuessRoots). Exact witness outputs and reproduction commands are in conformance/docs/parser-settings-variants.md. The variant protocol forbids capped or timed-out output from matching an expected empty result; the PanGloss tests include that negative control. The companion PanGloss adapter was tested at e2627c9 on feat/conformance-settings-variants. Counts quoted in earlier revisions of this description (tests, runs, fixtures, words) were taken at an older head and are not carried forward.

🤖 Generated with Claude Code

---

This change is

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.78%. Comparing base (b86120f) to head (69c4ee0).
⚠️ Report is 1 commits behind head on fix/signature-drops-id-under-identity-rule.

Additional details and impacted files

@@                              Coverage Diff                               @@
##           fix/signature-drops-id-under-identity-rule     #480      +/-   ##
==============================================================================
+ Coverage                                       74.62%   74.78%   +0.16%     
==============================================================================
  Files                                             453      556     +103     
  Lines                                           38345    56544   +18199     
  Branches                                         5285     7722    +2437     
==============================================================================
+ Hits                                            28614    42289   +13675     
- Misses                                           8568    12360    +3792     
- Partials                                         1163     1895     +732     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[johnml1135]

Added two independently justified conformance fixtures in a20bce1: final-template-partial-discriminators (10 words; #507) and template-category-sharing (6 words; #505). Fresh Release build: zero warnings/errors. Both fixtures pass with memoization on and off, zero skipped; 22 focused manifest/coverage tests pass. Four grammar counterfactuals produce the independently predicted mismatches, documented under conformance/docs. These are grammar-sensitivity controls, not proof that pruning, cache hits, or same-key feature collisions execute. No parser code changed. Generated inventory also restores previously omitted rows for the already-existing Exact-inverse fixture; its valid zudiua loss remains tracked in #504, not relabeled as correct.

[johnml1135]

Added the shared-template unconstrained-suffix conformance fixture in f150e2a. The grammar has a slot-only feature-unconstrained suffix shared by mandatory TV/IV templates, an ordinary N-to-IV verbalizer, four positive rows, and six rejection controls. The companion rationale explains complete forward derivations and loader reachability (not a FieldWorks UI round trip).

Fresh C# harness: all 10 rows pass in both template orders, memoization on/off (40 comparisons; no skips). Focused template/manifest tests: 22 passed, no skips. Regenerated manifest is current at 40 fixtures; coverage/rule reports updated. No Machine parser code changed.

PanGloss main now contains the narrow gate-only alignment (fc357daf), with this fixture mirrored unchanged until its old Machine submodule pin is upgraded. Direct state assertions provide red/green evidence; full-parse fixture checks pass both before and after the change. Issue #505 remains open for collision/correlation and broader merge-equivalence evidence.

[claude]

FYI: F45. This line scans conformance-stress, which exists nowhere in this repository (a tree-wide grep finds only this line and the comment above it), so after merge it is dead code. The comment at line 33 explains it by "the branches that carry it", a pointer outside this tree. Drop both lines, or have DiscoverProducible read only conformance.

[claude]

1. Verdict: approve. The 22 commits since 6b36f45 add two test files, the conformance/AGENTS.md guide, and fixture conversions with regenerated ledgers. None of it reopens an earlier finding.
2. What matters most: F45 (Low), FieldWorksProducibleShapeTests.cs:38. The new producibility test also scans conformance-stress, a directory that exists nowhere in this repository, and its comment points at other branches.
3. Counts: 1 new Low (F45). Of 45 total, 4 are Critical, 8 Important and 33 Low; only F45 is open.
4. What I ran:
   • gh pr checks 480: both Conformance suite runs and both comment-hygiene reports pass. Build on ubuntu-22.04 and windows-latest are pending.
   • dotnet test tests/SIL.Machine.Morphology.HermitCrab.Tests -c Release filtered to FieldWorksProducibleShapeTests, GrammarsOverviewTests, ConstructClaimCorroborationTests, GrammarCoverageLedgerTests, InterfaceWitnessLedgerTests, ConformanceFixtureGate and ConformanceManifestTests: 43 passed, 0 failed.
   • I read the src//tests/ diff since 6b36f45 and compared the new linear-stratum check with XmlLanguageLoader.LoadStratum. Both read only the stratum's own active definitions, so they agree.
   • I checked that every CLI flag conformance/AGENTS.md names exists in Program.cs, and that ConformanceFixtureGateTests.cs:258 enforces fieldworks_producible: true as that guide claims.
   • I did not run /code-review, because the code delta is two test files.
5. What I could not verify:
   • the claim that the C# oracle's rows are unchanged for all 137 runs after the outputPartOfSpeech and unordered conversions, beyond the self-check passing at this head;
   • the [Explicit] ledger freshness tests, which take tens of minutes;
   • the HCLoader producibility verdicts in the README table, which come from FieldWorks source that is not in this tree.

Public API, target framework, package, or sillsdev/machine.py parity: None verified.

Finding	Severity	Status
F1	Critical	addressed (cc947db)
F2	Important	addressed (bcb3d62)
F3	Important	addressed (bcb3d62)
F4	Important	accepted, unverified
F5	Low	addressed (bcb3d62)
F6	Low	addressed (067de6f)
F7	Low	addressed (0a1212b)
F8	Low	addressed (067de6f)
F9	Low	addressed (067de6f)
F10	Important	addressed (cd0c909)
F11	Low	accepted, unverified
F12	Low	addressed (cd0c909)
F13	Low	addressed (0a1212b)
F14	Low	addressed (0a1212b)
F15	Low	addressed (067de6f)
F16	Low	addressed (067de6f)
F17	Low	addressed (0a1212b)
F18	Low	addressed (0a1212b)
F19	Critical	addressed (e7fe733)
F20	Low	addressed (8b5d74e)
F21	Low	addressed (1c5c3a5)
F22	Low	addressed (a3b4c0e)
F23	Important	addressed (49ba481)
F24	Important	addressed (49ba481)
F25	Low	addressed (9671b28)
F26	Low	addressed (641f613)
F27	Low	addressed (8b5d74e)
F28	Critical	addressed (dd5f00f, 2cd68e8)
F29	Low	addressed (bb42ff4)
F30	Low	addressed (f592706)
F31	Low	addressed (6c6fbb3)
F32	Critical	addressed (cf84539, stacked on #500 and #556)
F33	Low	addressed (b9c7c8a)
F34	Low	addressed (382168a)
F35	Important	addressed (9d7963a)
F36	Low	addressed (19a2979)
F37	Low	addressed (19a2979)
F38	Important	addressed (b84c578)
F39	Low	addressed (ec897fa, 6e1155f)
F40	Low	addressed (d1f545c)
F41	Low	addressed (1358310)
F42	Low	addressed (e21aae4, fixture retired)
F43	Low	addressed (e21aae4, fixture retired)
F44	Low	addressed (6b36f45)
F45	Low	new

Reviewed at 69c4ee0