Skip to content

Compile a language file's [rules] into its pack's policy patch (3.0, 4f) - #953

Merged
derek73 merged 3 commits into
masterfrom
feat/v3-4f-rules-table
Oct 30, 2026
Merged

derek73 merged 3 commits into
masterfrom
feat/v3-4f-rules-table

Conversation

@derek73

@derek73 derek73 commented Oct 30, 2026

Copy link
Copy Markdown
Owner

3.0 sub-phase 4f of #754 (and #753): a language file's [rules] table compiles into its shipped Locale's PolicyPatch. RU and TR_AZ are now build output from ru.toml and tr.toml, value-equal to the packs written by hand, and no parse moves.

The schema

# nameparser/vocab/ru.toml
[rules]
patronymic_rules = ["east-slavic"]

Each key is the Policy field of its name, and its value takes that field's shape:

Key Shape Accepted
patronymic_rules set: a non-empty array, each name once "east-slavic", "turkic" (PatronymicRule values)
name_order choice: one name "family-first", "family-first-given-last"
two_family_names flag true

A rule is written only to turn it on. The defaults (name_order = "given-first", two_family_names = false) are refused, because a pack's scalar overrides the caller's own scalar in apply_patch. 4g's compound given name will be a new Policy field plus one row in RULE_KEYS and one in _locale._RULE_VALUES.

What changed

  • tools/vocab/build.py: new tables RULE_KEYS, RULE_DEFAULTS and PACK_RULES (ru → RU, tr → TR_AZ). Build check 12 refuses:

    • a table in a file no pack reads;
    • a rules that is not a table;
    • an empty table or an empty array;
    • an unknown key, rule or order name;
    • a value of the wrong type;
    • a name listed twice;
    • a default;
    • a file a pack reads that has no table.

    The build writes a third generated module, nameparser/config/_rules.py. It holds tuples only, one line per file, and --check exits 1 when it is stale.

  • nameparser._locale.rules_patch(tag): imports config/_rules.py at call time and builds the PolicyPatch. The pack modules still import _locale, _lexicon and _policy only. In test_layering.py, _locale's row gains nameparser.config._rules.

  • locales/ru.py and locales/tr_az.py: now policy=rules_patch("ru") and policy=rules_patch("tr").

  • Tests:

    • 16 mutation rows for check 12;
    • a test holding the build's schema table and the converter table to each other, to PolicyPatch's fields and to Policy()'s defaults, which runs every accepted name through the converter;
    • a stale-_rules.py check;
    • a roster test tying PACK_RULES to locales._REGISTRY;
    • the literal hand-written Locales pinned with ==;
    • the regex sync test now derives which patterns a pack must hold from its compiled patch, so turning on a rule in a file with no DEVIATES for it fails.

Decisions the spec left open

All are recorded in decisions/vocabulary-model's 2026-10-29 sub-phase 4f entry.

  1. The keys are the field names. Weighed against one key per rule.
  2. Writing a default is refused. It would switch a caller's rule off.
  3. Where the patch lives: config/_rules.py, read through _locale.rules_patch. Weighed against generating PolicyPatch(...) source, which would make a config module import _policy, and against a converter in each pack.
  4. Which file a pack reads is a build roster, PACK_RULES. TR_AZ reads tr.toml: there is no az.toml, and the rule is the same for both languages. The table does not choose the file's vocabulary, so languages stays empty for both packs.
  5. The endings stay in the core with their rule kind (_pipeline/_vocab.py). A file turns a rule on; it never supplies what the rule matches.
  6. DEVIATES stays hand-written, now held by a test to the rules the patch turns on. Deriving it outright, by moving each rule's surface predicate beside its enum member, is left Open for 4g.
  7. ZH and JA keep their hand-written patches. Their segment_scripts turn-on depends on a Python surname list (ZH) or the caller's segmenter (JA), so a table would split the pack without making it pure data.

Evidence the packs and parses are unchanged

  • locales.RU and locales.TR_AZ == the hand-written Locales, and their rules are PatronymicRule members.
  • Pickles. Five objects were pickled: RU, TR_AZ, parser_for(RU), parser_for(TR_AZ) and parser_for(RU, TR_AZ).
    • Pickles made under the 2.8.0 wheel (with PYTHONSAFEPATH=1, from a directory outside the worktree; the tell printed 2.8.0 from the uv cache) and under base 23aacddc load in this tree, compare equal and parse alike.
    • Under PYTHONHASHSEED=0, all five of this tree's pickles are byte-identical to the base's.
  • The full suite passes, including test_snapshot.py, test_cldr_agreement.py and the locale agreement records. Neither cldr_agreement_locales.json nor wikidata_native_agreement_locales.json was re-recorded.

Import time and frames

Each figure is the median of 25 fresh processes, Python 3.11.

before after
import nameparser.locales 0.23 ms 0.24 ms
first locales.RU, nameparser already imported 0.50 ms 0.69 ms
first locales.TR_AZ, nameparser already imported 0.50 ms 0.70 ms

The difference is the import of config/_rules.py. Packs still load lazily. The default parse runs no new code, since rules_patch runs once per pack, at the pack's import. The serial benchmark pass (-n0 -m serial, which holds the frame budget) passes.

Checks

  • uv run pytest
  • uv run pytest -n0 -m serial
  • uv run --python 3.15 pytest -q
  • the suite without namedivider, then again with it
  • mypy, ruff
  • build.py --check
  • sphinx-build -b doctest

No release-log bullet: nothing changes for users. The new docs are for contributors: nameparser/vocab/README.md's new "Rules" section and a step in docs/locales.rst's "Contributing a pack".

🤖 Generated with Claude Code

RU's and TR_AZ's PolicyPatch are now build output: ru.toml and tr.toml
carry a [rules] table (patronymic_rules = ["east-slavic"] / ["turkic"]),
tools/vocab/build.py checks it (check 12: RULE_KEYS, PACK_RULES, no
scalar default) and writes nameparser/config/_rules.py, and
nameparser._locale.rules_patch(tag) turns a file's table into the
patch. Both packs compare equal to the Locales written by hand, their
pickles load across the change, and no parse moves.

DEVIATES stays hand-written, now held by a test to the rules each
pack's compiled patch turns on; ZH and JA keep their hand-written
segmentation patches. Decisions: decisions/vocabulary-model's
2026-10-29 sub-phase 4f entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73 derek73 added enhancement design vocabulary Per-word vocabulary model and language packs labels Oct 30, 2026
@derek73 derek73 self-assigned this Oct 30, 2026
derek73 and others added 2 commits October 29, 2026 22:37
- RFC: describe what shipped (tuples in config/_rules.py, built into
  the patch by rules_patch at pack import); list phase 4's landed
  sub-phases in the phase table.
- decisions 4f: give ZH and JA each their own condition; state the
  import figures with nameparser preloaded and the recipe; link the
  Open questions to their issues.
- vocab README: describe the files holding [rules] by PACK_RULES rather
  than a count; one instruction for a new rule kind, matching
  mechanisms.md#LOCALE-PACKS-PURE-DATA (field with ADDED_IN, RULE_KEYS
  and _RULE_VALUES rows, its own mechanisms entry).
- build.py: the no-pack-reads message no longer doubles the quote.
- test_the_rule_tables_agree_with_policy requires a RULE_DEFAULTS entry
  for every scalar key, each converting to Policy()'s value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 4f decisions entry gains a SUPERSEDES line naming item 10's clause
that made ZH's surname list and the RU and TR_AZ endings rule data for
[rules]; items 5 and 7 state the reversal; the 2026-10-27 #753 head
names the superseded part; the RFC's "existing packs fit" sentence
describes what 4f built.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73
derek73 merged commit 4067514 into master Oct 30, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

design enhancement vocabulary Per-word vocabulary model and language packs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant