Repository navigation
Vocabulary build: type-design follow-ups and silent-pass refusals from #737's review - #744
Merged
Merged
Conversation
… from #737's review Type design (no output moves; _compiled.py is byte-identical): - query.Row stores the exclusion's structure (excluded, prop) and builds `label` only for display; select() no longer parses the label back. - compile_pool/compile_files return a Compiled TypedDict, tied to the CONSTANTS roster by one import-time assert; render reads it by roster. - Reading.acronym and Reading.addresses define each default once, read by _behavior, compile_pool, _refused_by and _check_reading. - Exclusion.reason is a property over props, the object is built only once its table is valid, and Reading and Exclusion are frozen. New refusals, each with a MUTATIONS row (every row's input compiled before this change, measured by stubbing its check): - a text no token can equal: a comma or nakaguro (the tokenizer divides at them), an invisible format character other than ZWNJ/ZWJ (bidi marks are stripped, a zero-width space is kept but unseen), a phrase word with an edge period, and `cased` on a phrase (looked up per word); - a file with no [[word]] entries; - a same_as naming its own word; - a date-time where `decided` or `as_of` expects a date, and an `as_of` with no count. DIVIDERS hand-copies the pipeline's comma and name-dot separators; a test holds it to them and checks every bidi mark the tokenizer strips is refused. A .toml in a subfolder was already refused by read_dir ("no subfolders"), pinned by test_the_folder_is_read_whole. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #744 +/- ##
=======================================
Coverage 99.05% 99.05%
=======================================
Files 47 47
Lines 4642 4642
=======================================
Hits 4598 4598
Misses 44 44 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
- Check 2 also refuses an emoji (stripped by default), an interpunct between two East Asian characters (divides there, rules/T3) and a control character (kept but unseen), and words its comma refusal as what it is: the main stream divides at a comma, and inside a clause rules/M3 depends on the vocabulary holding none. - The phrase-word refusal is a one-key-one-spelling rule, not a never-match one: Lexicon folds each word, so 'z. domu' would have matched as 'z domu'. Comment and message corrected. - The divider test is now driven by the tokenizer's own _ignorable over every codepoint, with EMOJI_RANGES and EAST_ASIAN_RANGES pinned to their sources, so a new separator or strip class fails it. Negative control: hiding the emoji check behind the pinned constant fails the sweep. - Exclusion validates its reason in __post_init__; Row asserts a reading refuses no property; Reading.flag() is the one definition of the may_be_* default. - as_of with a date-time gets its own message; the numbered check list says which checks run per entry and which over the pool. - New rows: control character, emoji, East Asian interpunct, a table-form exclusion reason; a query row pinning whole-name property matching; controls for a Catalan l·l and a CJK text ending in one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sweep accepted any VocabError, so U+0020 passed only because a later check refuses a phrase title (found by the fix-commit review). It now excludes the space, a phrase's word separator, and matches check 2's messages. The _check_tokenizable and Exclusion docstrings said more than the code does; corrected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
derek73
added a commit
that referenced
this pull request
Oct 19, 2026
A guard over a hand copy of a pipeline set went vacuous twice in one review: pinning copies missed an uncopied class (emoji, `dr★`), and a sweep accepting any refusal let an unrelated check answer (space, `dr x`). Amends the mutation-test field note with the fix that held, and names the interpunct as the class the predicate-driven sweep cannot reach, both in the note and in the test's comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3 tasks done
derek73
added a commit
that referenced
this pull request
Oct 19, 2026
* Record #744's hand-copy guard lesson in mechanisms.md's field notes A guard over a hand copy of a pipeline set went vacuous twice in one review: pinning copies missed an uncopied class (emoji, `dr★`), and a sweep accepting any refusal let an unrelated check answer (space, `dr x`). Amends the mutation-test field note with the fix that held, and names the interpunct as the class the predicate-driven sweep cannot reach, both in the note and in the test's comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Name the mutation row that holds the interpunct check, not the pins The pins hold the copied constants and stay green with the check gone; the text-with-an-east-asian-interpunct MUTATIONS row is what fails. Also cites the review by PR number and narrows 'would pass' to the case where a later check happens to refuse the test text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #737: the type-design refactors and low-severity silent-failure findings its review deferred, plus what this PR's own two review rounds found. Scope:
tools/vocab/build.py,tools/vocab/query.py,tests/v2/test_vocab.py, and one AGENTS.md line.No compiled set moves.
nameparser/config/_compiled.pyis byte-identical to master,build.py --checkreports current, andsnapshot.pyreports "matches the snapshot". No shipped TOML entry hits any new check; I measured that over every file before writing the checks.Type design
query.Rowstores the exclusion's structure (excluded,prop) and asserts that a reading refuses no property.labelis display-only, andselect()no longer parses it back.CompiledTypedDict.compile_pool/compile_filesreturn one instead ofdict[str, Any]. One assert at load ties it to theCONSTANTSroster, and the old runtime roster check andMAPPINGSare gone.Reading.acronym,Reading.addressesandReading.flag()(themay_be_*flags) are the only places the defaults are written._behavior,compile_pool,_refused_byand the checks all read them.Exclusion.reasonis a property overprops, and__post_init__refuses a missing or empty reason, so the invariant is enforced rather than documented.ReadingandExclusionare frozen.New refusals
Each has a
MUTATIONSrow.Build check 2:
z. domuwould actually match, sinceLexiconfolds each word, so this is a one-spelling-per-key rule, not a never-match one.Other new refusals:
casedon a phrase. Capitalization is looked up one word at a time, so a phrase'scasedis never read.[[word]]entries.same_asnaming its own word.decidedoras_ofexpects a date. tomllib'sdatetimepassesisinstance(..., date).as_ofwith no count.Guards and controls:
DIVIDERS,EMOJI_RANGES,EAST_ASIAN_RANGES,INTERPUNCT) are pinned to their pipeline sources.test_every_character_the_tokenizer_ignores_is_refusedsweeps every codepoint that_tokenize._ignorabledrops under the default Policy, plus the commas. It requires each one to be refused by check 2 itself.Already in place, unchanged: a
.tomlin a subfolder is refused byread_dir("no subfolders"), pinned bytest_the_folder_is_read_whole.Review
Five agents reviewed (code, silent failures, types, tests, comments), and a further pass reviewed the fix commit alone. Fixed from those rounds:
Not changed: the reviewer noted that a decomposed (NFD) hangul input is not divided at an interpunct, because the flank guard reads raw text. An entry refused by the interpunct rule could match such input. This is a nit, recorded here.
Checks
uv run python tools/vocab/build.py --check: currentuv run python tools/differential/snapshot.py: matches the snapshotuv run pytest: 21708 passed, 123 skipped, 3 xfaileduv run ruff check,uv run mypy,uv run mypy --strict tools/vocab/: clean🤖 Generated with Claude Code