Repository navigation
Run the FieldWorks parser conformance suite against HermitCrab in CI - #615
Open
johnml1135 wants to merge 15 commits into
Open
johnml1135 wants to merge 15 commits into
johnml1135 wants to merge 15 commits into
Conversation
The tool's batch command parses every word of a list with one grammar and writes one row per word: index, word, milliseconds, status and result signature. SignatureFormat builds the signature, so an external runner compares its engine against the same text. The script mode of the tool now returns the first nonzero command result instead of always 0, and SplitCommandLine is internal so a runner tokenizes a command template the same way. A MorpherTests case pins the Morpher constructor defaults that the signature protocol documents. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
conformance.commit holds the full SHA of the commit of sillsdev/fieldworks-parser-conformance that machine runs. The clone in conformance/ is ignored by git and by csharpier, as is the temporary directory a clone is made in. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The runner checks the conformance suite against the C# HermitCrab engine. Each fixture's words.yaml is compared with what the engine returns, word by word, in process, including traced rules and budget_ms. --adapter runs the same fixtures through an external engine from a command template, as the suite's PROTOCOL.md describes. --propose prints the words.yaml patch for a mismatch, --coverage-report checks the suite's coverage.csv, rules.csv and dead-rule list, and --check-manifest validates every grammar and words.yaml against the suite's DTD and schema and compares the suite's generated manifest with a fresh computation. The runner writes nothing into the suite. Building the runner runs EnsureConformanceSuite.targets, which makes conformance/ a clone of the suite at the commit in conformance.commit: cloned into a temporary directory and renamed when absent, moved to the pin when it is at another commit, left alone when it has uncommitted changes, and never fetched when it is already at the pin. A clone at the pin with local changes gets a warning naming them. A git that hangs is killed after three minutes, and any failure of the target is a warning, never a build failure. The runner warns when conformance/ is not at the pin or has local changes. HC_CONFORMANCE_SUITE=<path> uses another copy instead. SIL.Machine, the HermitCrab library and the tool do not reference the runner, so they never run it. The tests run every fixture, the manifest, the schemas and the FieldWorks-producible shape rules in dotnet test, and fail with the command that fixes it when the suite is missing or not at the pin. A test pins the script-mode exit code of the hc tool, which is the first nonzero result. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Selected with TestCategory=Stress, StressFixtureParityTests runs the 16 fixtures under conformance/stress through the same comparison as the runner. It never runs by default. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The Conformance workflow builds the runner once in Release, then runs the self-check, the stress self-check, the manifest check and the coverage report, and fails unless the commit in conformance.commit is an ancestor of the suite's main. local_check.sh runs the same checks and honours HC_CONFORMANCE_SUITE. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The README and AGENTS.md point to the suite repository. A README beside the runner says how the suite is fetched, how to bump the pin, how to regenerate the files the suite owns, how to run another engine with --adapter, and what to run before pushing. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## fix/signature-drops-id-under-identity-rule #615 +/- ##
==============================================================================
+ Coverage 74.62% 75.10% +0.48%
==============================================================================
Files 453 490 +37
Lines 38345 41265 +2920
Branches 5285 5682 +397
==============================================================================
+ Hits 28614 30993 +2379
- Misses 8568 9048 +480
- Partials 1163 1224 +61 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
This comment has been minimized.
This comment has been minimized.
The dispatcher counted a batch line's arguments after it had parsed --start, and the reset ran only when Run completed. A script line such as "batch --start=2 words.txt" was rejected for its missing output path and left the next batch line resuming at index 2. BatchCommand now counts its own arguments and clears the start index before it rejects a line. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Self-check passed an expect_crash fixture on any exception except the watchdog timeout, so a harness failure such as the rule index rejecting a duplicate rule name read as the expected crash. Adapter mode already credited only EngineCrashException. The engine load and the parse now throw that exception, and the crash match requires it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A grammar that was not well-formed XML made GrammarValidation throw from the reader, and --check-manifest ended in an unhandled exception whose stack trace did not name the fixture. The loop now reports the fixture id and the reader's message and exits 2, as it does for a grammar that fails the DTD. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
RunSelfCheck adds one result per word run, so a stress fixture with a variants list reported two passes and failed the test's expectation of exactly one. The test now requires at least one result and every result to have passed. No current stress fixture has variants. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
local_check reported suite-modified whenever the suite had uncommitted edits, which the targets allow and HC_CONFORMANCE_SUITE is meant for, and it missed files the runner left untracked. It now records the suite's status and diff before the runner steps and fails only if they differ afterwards. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Three diagnostic messages sorted names with the current culture, so the order could differ between machines. They now use StringComparer.Ordinal as the rest of the runner does. Only the message text is affected. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The template is split by the same routine as an hc script line, which
drops quote characters and keeps no empty token, so --grammar="{grammar}"
reached the engine as two arguments. The README now says to write
--grammar {grammar}, and that placeholders are filled after the split so
a path with spaces stays one argument.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Self-check mode runs through Runner.RunSelfCheck, and RunAdapter is reached only with an AdapterEngine, so SelfCheckEngine.Run was never called. Without it the class only carried a name and a capability set into the report header, so Program now passes those to PrintRunReport directly. Comments that named it as an IEngine are corrected. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This was referenced Nov 2, 2026
This comment has been minimized.
This comment has been minimized.
Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
This was referenced Nov 3, 2026
Contributor
|
Verdict: approve. The only change since the last round fixes F8, and no findings remain open. Most important: F8 ( Counts: Critical 0, Important 0, Low 0 open. No new findings. Ran: Not verified: I could not run CSharpier here, so whether F8 is fully fixed depends on the pending ubuntu Build reaching "Check formatting" and passing it. API / parity: None verified changed since the last round. This is not a port from Findings: F1 addressed, F2 addressed, F3 withdrawn, F4 addressed, F5 addressed, F6 addressed, F7 addressed, F8 addressed. Reviewed at 4d9b454 |
johnml1135
added a commit
to sillsdev/fieldworks-parser-conformance
that referenced
this pull request
Nov 3, 2026
… the wrapper script. Removes the machine commit hash, the "last copy" wording and the sillsdev/machine#615 mentions that the previous commit added, and returns PROVENANCE.md to its content on main. The consumer-pin corrections stay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 3, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Quick summary
dotnet testand a new CI workflow now run the FieldWorks parser conformance suite against the C# HermitCrab engine. The suite is not vendored:conformance.commitpins its commit, and the build clones it into an ignoredconformance/. The engine is untouched, but thehctool's script mode now exits with the first failing command's code.Where to look
ConformanceSuite.Root()fails each test with the two commands that fix it.EnsureConformanceSuite.targetsonly warns on failure, kills a git stalled for 3 minutes, and skips packing.ConformanceFixtureGateTestsandconformance.yml, which fails a pin off the suite's main.hcexit code --ToolScriptExitCodeTests.Deliberately not included
--adaptercan run its binary, but CI runs only the C# engine until PanGloss adopts the suite repo.Validation
./local_check.sh --agent-strict-- not run (no cmake); its steps ran by handdotnet test tests/SIL.Machine.Morphology.HermitCrab.Tests-- 270 passed at the tip; also 270 from a plain clone, before a rebase onto Fix a dropped morpheme ID when a zero-width rule wraps one ending at the same node #500 that only renamed testspwsh ./scripts/comment-hygiene.ps1,dotnet tool run csharpier check .-- cleangit diff --checkagainst the Fix a dropped morpheme ID when a zero-width rule wraps one ending at the same node #500 branch -- Machine.sln only, CRLFIssue / porting context
Stacked on #500; #556 and #500 land first. Replaces #480 and #578, both closed. Their grammars are now in sillsdev/fieldworks-parser-conformance.
Behaviour change to the shipped `hc` tool
Script mode (
hc -i grammar.xml -s script) used to exit 0 whatever the commands returned. It now exits with the first nonzero command result, so the runner'sbatchwrapper can see a failure.ToolScriptExitCodeTestspins it. Interactive mode is unchanged.What is kept and dropped
Kept: the runner (self-check and
--adapter),batch, and only the tests that fail when the engine, the runner or the pin changes: fixture gate, manifest, schemas, DTD, FieldWorks-producible shapes, adapter crash classification. Dropped: semantic-coverage ledgers and their ratchets, the counterfactual workflow,eng/,global.json, the grammar-overview test.How the pin behaves
At the pin: no network and no git writes; local changes get a warning, from the build and from the runner. At another commit: moved to the pin, or a build error if it has uncommitted changes. No git, a failed fetch, a git that stalls for 3 minutes or any other failure: a warning, then the tests fail naming the two commands.
HC_CONFORMANCE_SUITE=<dir>uses that directory and skips the clone and the pin check. To bump: editconformance.commit, build, run the runner checks, commit.Verification detail
Plain clone: 270 passed in 44 s. Windows (PowerShell 5.1, Git for Windows 2.51, dotnet 10.0.303): first build cloned and reached the pin, a second build with GitHub unreachable touched nothing, a pin bump moved the suite and back, a two-hour-old read-only temp directory was removed, 270 passed. Linux: a hung git ended the build in 3 minutes with a warning; a failed rename kept the clone and warned. Runner: 143 on 133 fixtures, stress 16/16, manifest and coverage current. CI on this head passed on Ubuntu and Windows, including both conformance suite jobs. Not run:
local_check.sh.🤖 Generated with Claude Code
This change is