Skip to content

Vocabulary build: type-design follow-ups and silent-pass refusals from #737's review - #744

Merged
derek73 merged 3 commits into
masterfrom
feat/vocab-type-design-and-refusals
Oct 19, 2026
Merged

derek73 merged 3 commits into
masterfrom
feat/vocab-type-design-and-refusals

Conversation

@derek73

@derek73 derek73 commented Oct 19, 2026 •

Copy link
Copy Markdown
Owner

Follow-up to #737: the type-design refactors and low-severity silent-failure findings its review deferred, plus what this PR's own two review rounds found. Scope: tools/vocab/build.py, tools/vocab/query.py, tests/v2/test_vocab.py, and one AGENTS.md line.

No compiled set moves. nameparser/config/_compiled.py is byte-identical to master, build.py --check reports current, and snapshot.py reports "matches the snapshot". No shipped TOML entry hits any new check; I measured that over every file before writing the checks.

Type design

  • query.Row stores the exclusion's structure (excluded, prop) and asserts that a reading refuses no property. label is display-only, and select() no longer parses it back.
  • Compiled TypedDict. compile_pool/compile_files return one instead of dict[str, Any]. One assert at load ties it to the CONSTANTS roster, and the old runtime roster check and MAPPINGS are gone.
  • One definition per default. Reading.acronym, Reading.addresses and Reading.flag() (the may_be_* flags) are the only places the defaults are written. _behavior, compile_pool, _refused_by and the checks all read them.
  • Exclusion. reason is a property over props, and __post_init__ refuses a missing or empty reason, so the invariant is enforced rather than documented. Reading and Exclusion are frozen.

New refusals

Each has a MUTATIONS row.

Build check 2:

  • A comma or nakaguro. The tokenizer divides the main stream at both. Inside a clause a comma is kept, but rules/M3 (a marker followed by a comma is no marker) relies on the vocabulary holding none.
  • An emoji. The tokenizer strips it by default.
  • An interpunct between two East Asian characters. The tokenizer divides the name there (rules/T3).
  • An invisible character other than ZWNJ/ZWJ: a format or control character. ZWNJ/ZWJ are allowed because they are spelling in Persian and the Indic scripts.
  • A phrase word with an edge period. z. domu would actually match, since Lexicon folds each word, so this is a one-spelling-per-key rule, not a never-match one.

Other new refusals:

  • cased on a phrase. Capitalization is looked up one word at a time, so a phrase's cased is never read.
  • A file with no [[word]] entries.
  • A same_as naming its own word.
  • A date-time where decided or as_of expects a date. tomllib's datetime passes isinstance(..., date).
  • An as_of with no count.

Guards and controls:

  • The hand copies (DIVIDERS, EMOJI_RANGES, EAST_ASIAN_RANGES, INTERPUNCT) are pinned to their pipeline sources.
  • test_every_character_the_tokenizer_ignores_is_refused sweeps every codepoint that _tokenize._ignorable drops under the default Policy, plus the commas. It requires each one to be refused by check 2 itself.
  • Negative controls: with any new check stubbed out in memory, its rows' inputs compiled. Hiding the emoji check behind its pinned constant fails the sweep.

Already in place, unchanged: a .toml in a subfolder is refused by read_dir ("no subfolders"), pinned by test_the_folder_is_read_whole.

Review

Five agents reviewed (code, silent failures, types, tests, comments), and a further pass reviewed the fix commit alone. Fixed from those rounds:

  • the emoji, CJK interpunct and control-character gaps;
  • a vacuous divider test, twice: first it pinned only two constants, then the sweep accepted any refusal;
  • a wrong never-match claim about phrase words, plus overclaims in the comments about commas and bidi marks;
  • the Exclusion and Row invariants;
  • a test for a table-form exclusion reason.

Not changed: the reviewer noted that a decomposed (NFD) hangul input is not divided at an interpunct, because the flank guard reads raw text. An entry refused by the interpunct rule could match such input. This is a nit, recorded here.

Checks

  • uv run python tools/vocab/build.py --check: current
  • uv run python tools/differential/snapshot.py: matches the snapshot
  • uv run pytest: 21708 passed, 123 skipped, 3 xfailed
  • uv run ruff check, uv run mypy, uv run mypy --strict tools/vocab/: clean

🤖 Generated with Claude Code

… from #737's review

Type design (no output moves; _compiled.py is byte-identical):
- query.Row stores the exclusion's structure (excluded, prop) and builds
  `label` only for display; select() no longer parses the label back.
- compile_pool/compile_files return a Compiled TypedDict, tied to the
  CONSTANTS roster by one import-time assert; render reads it by roster.
- Reading.acronym and Reading.addresses define each default once, read by
  _behavior, compile_pool, _refused_by and _check_reading.
- Exclusion.reason is a property over props, the object is built only
  once its table is valid, and Reading and Exclusion are frozen.

New refusals, each with a MUTATIONS row (every row's input compiled
before this change, measured by stubbing its check):
- a text no token can equal: a comma or nakaguro (the tokenizer divides
  at them), an invisible format character other than ZWNJ/ZWJ (bidi
  marks are stripped, a zero-width space is kept but unseen), a phrase
  word with an edge period, and `cased` on a phrase (looked up per word);
- a file with no [[word]] entries;
- a same_as naming its own word;
- a date-time where `decided` or `as_of` expects a date, and an `as_of`
  with no count.

DIVIDERS hand-copies the pipeline's comma and name-dot separators; a test
holds it to them and checks every bidi mark the tokenizer strips is
refused. A .toml in a subfolder was already refused by read_dir
("no subfolders"), pinned by test_the_folder_is_read_whole.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.05%. Comparing base (ef03011) to head (82efc03).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #744   +/-   ##
=======================================
  Coverage   99.05%   99.05%           
=======================================
  Files          47       47           
  Lines        4642     4642           
=======================================
  Hits         4598     4598           
  Misses         44       44           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@derek73 derek73 added this to the 2.6 milestone Oct 19, 2026
derek73 and others added 2 commits October 19, 2026 02:26
- Check 2 also refuses an emoji (stripped by default), an interpunct
  between two East Asian characters (divides there, rules/T3) and a
  control character (kept but unseen), and words its comma refusal as
  what it is: the main stream divides at a comma, and inside a clause
  rules/M3 depends on the vocabulary holding none.
- The phrase-word refusal is a one-key-one-spelling rule, not a
  never-match one: Lexicon folds each word, so 'z. domu' would have
  matched as 'z domu'. Comment and message corrected.
- The divider test is now driven by the tokenizer's own _ignorable over
  every codepoint, with EMOJI_RANGES and EAST_ASIAN_RANGES pinned to
  their sources, so a new separator or strip class fails it.
  Negative control: hiding the emoji check behind the pinned constant
  fails the sweep.
- Exclusion validates its reason in __post_init__; Row asserts a
  reading refuses no property; Reading.flag() is the one definition of
  the may_be_* default.
- as_of with a date-time gets its own message; the numbered check list
  says which checks run per entry and which over the pool.
- New rows: control character, emoji, East Asian interpunct, a
  table-form exclusion reason; a query row pinning whole-name property
  matching; controls for a Catalan l·l and a CJK text ending in one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sweep accepted any VocabError, so U+0020 passed only because a
later check refuses a phrase title (found by the fix-commit review).
It now excludes the space, a phrase's word separator, and matches
check 2's messages. The _check_tokenizable and Exclusion docstrings
said more than the code does; corrected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73
derek73 merged commit 40d5b97 into master Oct 19, 2026
12 checks passed
derek73 added a commit that referenced this pull request Oct 19, 2026
A guard over a hand copy of a pipeline set went vacuous twice in one
review: pinning copies missed an uncopied class (emoji, `dr★`), and a
sweep accepting any refusal let an unrelated check answer (space, `dr x`).
Amends the mutation-test field note with the fix that held, and names the
interpunct as the class the predicate-driven sweep cannot reach, both in
the note and in the test's comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
derek73 added a commit that referenced this pull request Oct 19, 2026
* Record #744's hand-copy guard lesson in mechanisms.md's field notes

A guard over a hand copy of a pipeline set went vacuous twice in one
review: pinning copies missed an uncopied class (emoji, `dr★`), and a
sweep accepting any refusal let an unrelated check answer (space, `dr x`).
Amends the mutation-test field note with the fix that held, and names the
interpunct as the class the predicate-driven sweep cannot reach, both in
the note and in the test's comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Name the mutation row that holds the interpunct check, not the pins

The pins hold the copied constants and stay green with the check gone;
the text-with-an-east-asian-interpunct MUTATIONS row is what fails.
Also cites the review by PR number and narrows 'would pass' to the case
where a later check happens to refuse the test text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@derek73
derek73 deleted the feat/vocab-type-design-and-refusals branch October 19, 2026 20:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant