diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index a0aec9782..b99ff0a75 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -31,7 +31,9 @@ jobs: # stops reporting a line once it is recorded: a line tracer # (sys.settrace, its core below 3.14) is called on every line # every time, and cost CI's 3.12 job most of its ~12 minutes on - # the property grids in tests/v2/test_properties.py. The package + # the property grids in tests/v2/test_properties.py (measured + # 2026-09-29: the three #397 grids took 139s under the settrace core + # on 3.12, 24s under sys.monitoring, 17s with no coverage). The package # has no version-conditional code, so one interpreter's report # stands for all of them. COVERAGE_ON: ${{ matrix.python-version == '3.14' }} diff --git a/AGENTS.md b/AGENTS.md index d75e32279..92243f237 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -34,6 +34,8 @@ Three committed contributor docs carry the parser's normative rules and their re **Writing the user docs (docs/*.rst) has its own AGENTS.md too.** `docs/AGENTS.md` carries the style the docs are written in — tables as indexes, subheadings per task, measured claims, recipe doctests — distilled from PR #588. It loads the same way as the docs/design/ one below; if your tool does not do nested discovery, read it before editing a `.rst` file under docs/. +**A nested AGENTS.md is a file in its folder, and a tool that reads the folder whole reads it too.** Root AGENTS.md is loaded by every session, so guidance that applies to one folder belongs in that folder's AGENTS.md, which loads when a session reads a file there: `nameparser/AGENTS.md` (the sweeps), `nameparser/_pipeline/AGENTS.md` (ambiguity reports), `nameparser/vocab/AGENTS.md` (adding a word), `docs/AGENTS.md` and `docs/design/AGENTS.md`. Before adding one, check what reads its folder: `tools/vocab/build.py` refuses every non-TOML file in `nameparser/vocab/` but the names in its `SKIPPED`, so that a misnamed language file fails loudly, and #911's vocab file failed eleven tests until it joined that set. + **Working on docs/design/ has its own AGENTS.md.** `docs/design/AGENTS.md` carries the landing-a-design distillation checklist, the primary-source review rule, the dated-count convention, and the review axes. Claude Code loads it automatically when a session reads or edits anything under docs/design/; if your tool does not do nested discovery, read it yourself before touching those files or reviewing a change to them. **Triaging a design backlog: shape before value.** When a pile of open questions has to be ordered, relatedness will not partition it — in a parser nearly everything touches particles or suffixes. The line that does is whether a decision changes WHAT THE MODEL CAN EXPRESS or fills in a value in a shape already fixed. Settle the shape questions first, as a batch: they are few, they are usually independent of each other (so their own ordering does not matter), and each one collapses or re-frames a run of the value questions below it. The rest are leaves needing one measurement and one answer apiece, in any order. Worked example, 2026-08-16 (#386): 30 open issues and ~13 open design questions reduced to three shape decisions, which decided or re-framed fourteen of them — and two of the three turned out smaller than their issues claimed, because the issues had gone stale. Check the age of an issue's premises before believing its cost estimate. @@ -251,7 +253,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These ## Extension Patterns -**Adding a word to the vocabulary** — the steps are in `nameparser/vocab/README.md`; this paragraph is the traps they have run into. First check the word's OTHER readings: `uv run python tools/vocab/query.py --word ` lists every reading and exclusion of it in every file, which is where the overlaps between categories now show. Real overlaps exist: `st`/`freiherr` ∈ `PARTICLES` ∩ `TITLES`, and `do`/`mc`/`vd` ∈ `PARTICLES` ∩ `SUFFIX_ACRONYMS`; `abd` = "ABD" ∈ `SUFFIX_ACRONYMS`; `abu` ∈ `PARTICLES` ∩ `BOUND_GIVEN_NAMES` (position-dependent: leading token → given-name join, mid-name → family-name join). Usually position-dependent and harmless, but can force a guard or an exclusion (the `last_base` all-particles guard; `abd`, kept out of `BOUND_GIVEN_NAMES` until decisions/P5 found the credential collision cost nothing at either end). Then check the word against real names, not memory: `uv run python tools/name_usage.py ` counts its bearers as a given name and as a family name (US Census surnames, SSA given names, Wikidata with countries of citizenship), and gives its given-name frequency band per country from Jörg Michael's `nam_dict.txt` (54 countries and an "other countries" column, GFDL, downloaded into the cache and never committed) — the one source that sees a given name outside the US and Wikidata's lower bound, such as Korean `In` (band A in Korea) or Vietnamese `Bay` (6 in Vietnam). A band is a frequency class, not a count, and the dictionary is a selection, so a miss there (`Shehu`) proves nothing; and because a truncated source reads as a run of false zeros, a pinned download is checked against its sha256 on every load from the cache, never against a floor on what it holds — #775's first fix refused fewer than 40,000 names and passed a `nam_dict.txt` missing only its last line (`Zyta`). Pass a name as it is written (`McDonald`): Wikidata labels match exactly, and no rule rebuilds an internal capital from `mcdonald`. A word in a non-Latin script is counted by its romanization too, since the Census, SSA and nam_dict hold only Latin names, and that count weighs only where its bearers write the script: #835 vetted its salutations by their own spellings and its review found `बाबू` flagged while `babu` is a US surname (1,496) and an Indian given-name element (Ram Babu), where the romanization of `মি`, `Mi`, counts Korean and Chinese bearers who never write Bengali. It reports and does not decide — read the counts against the position the set's claim acts on. For a suffix it also has the other side: `--fetch-nppes` counts the US healthcare providers who write the word as a credential (NPPES, the 4.8 million credentialed providers in the pinned August 2026 file, so blind to law, accounting and honours), and `--rank N --unlisted` lists the commonest credentials the vocabulary misses, with where each lands in `John Smith, X` -- the list #797 came from. Where a local corpus is fetched (`tools/differential/local_corpora.py`, mechanisms.md#NAME-DATA-JOBS), a row says how many of its records (for ClinicalTrials.gov, officials) write the word, and how many after a comma: ClinicalTrials.gov's study officials show where a credential actually stands in a name as typed. A word that would peel GLUED (`GLUED_HONORIFICS`) asks a different question, whether any name ENDS in it, and for a CJK word the tool answers that too: an ends-with row from a cached index of Wikidata's CJK name labels (built on first use; 33 minutes on 2026-10-17), plus the names the word ends across the family/given join (阿部 + 長 for 部長). Read the tool's hits by item before counting them, because labels include transcriptions and mislabels: #707's one hit for 二世 was 厄德二世, the given name Odo with a regnal numeral in its Chinese label, borne by 109 Western people named Odo. Then ask a third question the tool cannot: does a common longer word end in it? The peel matches the end of a token, spaced or glued, so a listed tail splits every unlisted compound that ends in it — #707 vetted 議員, 社長 and 医学博士 against names and its review found 衆議院議員, 副社長 and 獣医学博士 cut at the tail (`田中 副社長` → given 副); no corpus held a compound office word until #707's rules examples added two, so the snapshot was silent about the class. List the common compounds whole or record the split as a cost. And check the predicates that test a member's SPELLING rather than its membership: they encode what the set's members happened to look like when they were written. `_pieces.starts_a_credential_run` implemented the credential run's "Latin" (rules/S8 now, part of S2 until #682 split it) as `isascii()`, exact until #661 added `mlađi`, which would have read `mlađi Jones` differently from `stariji Jones`; the design-docs review caught it, after a first draft had credited the test to the wrong clause of S2 and proposed deleting it on a mutation that moved nothing because nothing pinned it. (#679 then dropped the script test on purpose, after measuring what it protected.) Before adding the first member of a new script, or the first accented one, grep the set's consumers for `isascii`, `isalpha` and script tests, and read the rule each one implements. The same goes for the first member holding PUNCTUATION: #700 added `d'` and `az-`, the parse read them right and every snapshot move was an intended one, and `capitalized()` still wrote `D' Ivoi`, because case repair splits a token at its apostrophe or hyphen and asks the vocabulary of the stem. The snapshot records both case views since #826, but only for the names the corpora hold, so parse such a member's names through `capitalized()` and `capitalized(force=True)` in each case before calling it done. The user docs use real words as examples of an UNLISTED one, and listing the word breaks the example: #797 listed `dphil`, and docs/customize.rst's capitalization example (`parse("jane smith dphil").capitalized()` giving `Dphil`) failed in all five CI build jobs while `uv run pytest` passed, because `sphinx-build -b doctest docs` is a separate step. Run it after adding a word, and grep `docs/*.rst` for the word; the example now uses `dclinpsy`, a credential left unlisted on purpose, so list it only after moving the example. Record what the vetting found ON the word, not in a comment or a decisions sentence: `note`, `issue`, a `given`/`family` reading with dated counts, and for a word you decided to keep out an `excluded` entry with its reason, so the next contributor who adds it meets the reason as a build error (decisions/vocabulary-model). A name reading is not only provenance on a word that has a credential, particle or title reading: it IS the clash, and derives the marker (an ambiguous acronym, a particle that may be given, since 2026-10-25, a generation or agnomen that may be a name, rules/S11, since phase 2h the same day, so given-name counts recorded on `junior` would gate it, and a title that may be a given name, rules/H9, since phase 2i), so counts kept as evidence AGAINST marking such a word go in the clashing reading's `note` instead, as `ld`'s do. A word whose reading moves may be the worked example of the class it is leaving: `king` was the rules' and tests' title-that-is-also-a-surname across 35 files when #901 gave it a given reading, and five tests went vacuous before the review found them. Grep rules, decisions Standing lines, the RFCs, AGENTS.md, docs/*.rst, code comments and tests for the word before measuring, and replace it with a stand-in checked to read exactly as the word read on master in every example (`pope` there). A new derivation reads readings other work recorded for another purpose, so list what it would mark before shipping it: phase 2i's first build found thirteen salutations whose given readings #835 had kept as H8's evidence (`mr` among them, so `Mr Smith` would have read given 'Mr'), and moved their counts into notes. +**Adding a word to the vocabulary** — the steps are in `nameparser/vocab/README.md`, and the traps they have run into (other readings that clash, reading the real-name counts, the predicates that test a member's spelling, user-doc examples that use the word, recording the vetting on the word) are in `nameparser/vocab/AGENTS.md`, which loads when a session reads a vocabulary TOML file. First check the word's OTHER readings (`uv run python tools/vocab/query.py --word `), then its bearers (`uv run python tools/name_usage.py `). **Before adding a short/common word to `PARTICLES` globally**, test it mid-string against realistic 3-token names, not just check for English-word collisions: Korean/Vietnamese given names put a short syllable in the middle slot (`Park In Hwan`, `Nguyen To Nga`), and Western names put a bare initial there (`John V. Smith`). A word that looks safe ("nobody is named 'to'") can still swallow a real middle name/initial into the last name once it's a global prefix — confirmed regressions for `to`/`in`/`an`/`ten`/`then` and bare `v` this way (PR #191). @@ -293,7 +295,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These **`_normalize` must reach a fixed point** — storage and match-time share the one fold, and `Lexicon.__setstate__` re-validates, so a value that changes on re-normalization changes under its owner. `strip().strip(".")` alone is not idempotent (`'. a .'` → `' a '` → `'a'`). The loop is the fix; keep any new stripping inside it. **Anything built on `_normalize` must converge too** — `_fold_words` runs `_normalize` per word and DROPS the words that fold away (`_title_key` is that list space-joined, and `_run_addresses_by_given` reads the list itself, so its last-word arm is the last word of the FOLDED key by construction); keeping the empty slot stored `'lt .'` as `'lt '`, a key match-time can never rebuild (so the entry is silently inert) and `__setstate__` rejects on the next round-trip as "not written by this version". -**Perf regressions are caught by the scaling test, not the absolute-time ones** — `tests/v2/test_benchmark.py::test_parse_cost_grows_no_worse_than_linearly` times a repeated unit at n vs 4n over a table of shapes (one per pipeline inner loop) and bounds the ratio; the `_thousand_names` tests use constant-size, delimiter-free input and are structurally blind to a complexity regression. Two rules when touching it: calibrate `_MAX_RATIO` against the WEAKEST quadratic's signal (a mixed quadratic surfaces far below the textbook 16×, so the operating point `_BASE` matters more than the bound), and confirm a planted regression fails it across REPEATED runs — one failure is a coin-flip on a timing test. The shapes cover different dimensions (segment count only via `commas`, intra-piece accumulation only via `particles`/`conjunctions`, non-ASCII input only via `honorifics` — every other unit is pure ASCII, so `script_segment` returns at its bail and the CJK stages go unmeasured, M2's clause view only via `maiden_clause`, the one unit holding a marker (until #601 its order mattered, `MA nee ` reaching nothing; since #601 either order reaches the take) — and connective RUN LENGTH only via `link_run`, whose unit has to be MIXED CASE and hold a connective of the generational class: `i und ` reads the letter as an initial and reaches nothing where `i Und ` reaches everything, and S7's anchor pass only via `credential_run`, whose unit has to hold a Title-case member behind an unambiguous credential: `PhD MA ` reads the same fields and never asks the pass, the capitals deciding each member first); measure before pruning one. **A shape whose input needs a PREFIX cannot be a `_SHAPES` row at all**, since that table repeats a unit and nothing else — a maiden clause needs a name word and a marker before the run it is about, and `"nee i Und " * n` reaches the clause rule not at all, measuring the identical ratio on a broken tree and a fixed one. That one is `test_a_clause_link_run_does_not_cost_quadratically`, which builds its own input and counts FRAMES. A prefixed shape whose cost is Python-level joins `_REREAD_SHAPES` beside it — (input builder, the one function the defect re-runs, reachability probe) rows counted in that function's frames at 8 units against 32 — which holds #558's trailing fixed point, its maiden-clause twin, and #559's particle chain, a shape that grows at both ends and so has no unit to repeat. A prefixed shape whose cost is C-level work — a list scanned by `in`, a tuple re-hashed as a dict key — emits no frame for that count to see, so it goes in `_PREFIXED_SHAPES` instead: (prefix, unit, reachability probe) rows timed on the clock like `_SHAPES`, at a base of their own recorded beside the table (#553, whose run shape read 5.97× once at base 800 on the pre-fix tree, under the bound; py3.11, 2026-09-28). **That table skips under a line tracer** (`sys.settrace`, coverage's core below py3.14), which slows every Python line and not the C-level scan, diluting the quadratic below the bound (5.46–5.52× broken at base 1600 under `coverage run` on py3.11, same date). The `sys.monitoring` core coverage uses from 3.14 does not dilute it (8.87–10.91× broken), and CI collects coverage in its 3.14 build job alone, so every CI job runs the table (in its own `-n0 -m serial` pass: the module is marked `serial`, since a parallel run's load stretches the larger timing more than the smaller -- `maiden_clause` read 6.4× against 6.0 under `-n auto` on a tree that passes serially, 2026-10-23 -- so a new clock-timed test elsewhere takes the mark too); `ja-extra` also sets `NAMEPARSER_REQUIRE_CLOCK_GUARDS` so that a line tracer there fails the rows instead of skipping them. **Coverage is collected on ONE interpreter, the one that uploads it** (`COVERAGE_ON` in `.github/workflows/python-package.yml`): the package has no version-conditional code, so one report stands for all of them, and under a line tracer the property grids in `tests/v2/test_properties.py` had cost the 3.12 job most of its ~12 minutes (measured 2026-09-29: the three #397 grids take 139s under coverage's settrace core on 3.12, 24s under its sys.monitoring core, 17s with none). **A shape the CLOCK cannot reach needs a FRAME-count guard instead**, which is the second scaling test in that file (`test_a_trailing_credential_run_does_not_cost_exponentially`, #531): where the defect is an exponential rather than a quadratic, the input length that separates the curves on a timing test does not finish, so the guard counts frames over 8 units against 16 and bounds THAT ratio. One pair does not see every curve, and the fix round for #531 measured why: at 2× the input the per-member LINEAR work swamps a quadratic (2.08× for a genuine one against 1.73× clean), so that pair guards the exponential alone and a second, longer pair — 16 against 64, where the same quadratic reads 7.42× against 3.53× clean — is what can see one. Assert them in that order: an exponential never returns from the longer run, so the cheap pair has to have failed first. **A frame guard counts ONE function, so it goes blind when the code it watches stops calling that function.** #563's paired-initials guard counted `_normalize`, which segment's scan reached; #613 moved the scan into `_comma.decide`, where it calls `is_suffix_piece`, and the quadratic came back while the guard read 3.6× and passed (review caught it: 120 → 1,620 calls for 4× the input). When code under a frame guard moves, re-take the guard's negative control on the moved code before trusting a pass. Frame counts do not move under load, so this shape needs no repeated-run calibration — but it does need the same reachability assertion `_POLICY_SHAPES` rows carry, since the walk under measurement runs only while every unit still reads as a credential. A stage gated on an opt-in `Policy` field needs a `_POLICY_SHAPES` entry instead, since bare `parse()` never enters it — and that table's rows carry a **reachability probe** run before the measurement, because a precedence change can quietly stop the shape reaching the stage and leave a green test measuring a no-op (`_POLICY_SHAPES` is also asserted non-empty: an empty `parametrize` is a skip, not a failure, so deleting its last row would retire the guard silently). +**Perf regressions are caught by the scaling tests, not the absolute-time ones** — the `_thousand_names` tests use constant-size, delimiter-free input and are structurally blind to a complexity regression. A new inner loop, a policy-gated stage or a shape that needs a prefix gets a row in `tests/v2/test_benchmark.py`: its module docstring says which table, and the comment beside each table carries the measurements behind its rule. Coverage is collected on ONE interpreter, 3.14's (`COVERAGE_ON` in `.github/workflows/python-package.yml`, whose comment says why). **A JSONL file written with `json.dumps(..., ensure_ascii=False)` is cut by `str.splitlines()`** — `json.dumps` leaves U+2028, U+2029 and U+0085 unescaped, and `splitlines()` splits at all three, so one name pasted from a document holding a line separator turns its line into two halves of invalid JSON, and the file into one no reader can load. #801's review found it in the local corpora (a single official among 536,981 would have made the cache unloadable for good), and the review of that fix found it again one file over, in `snapshot.py`'s `dump`/`load`, which the fix had not touched. So it is fixed at the shared source now: `corpus_build.jsonl_line` escapes the three and `corpus_build.jsonl_lines` cuts at `"\n"` only, and `corpus_build`'s corpus and fields writers, `snapshot.py`, `cldr_agreement.py`, `local_corpora.py`, `local_psc.py`'s extracts and `local_agreement.py` go through them. A new JSONL writer or reader in tools/ goes through the same two. Still outside them: the builders that write their own lines (`build_cjk_corpus.py`, `_conventions`, `_shapes`, `_rules`, `_issues`), `compare.py`'s worker stream, and the tests' corpus reads (`_differential_fixtures.py`, `test_corpora.py`); none of the 16 committed JSONL files held one on 2026-10-21. diff --git a/docs/design/decisions/vocabulary-exclusions.md b/docs/design/decisions/vocabulary-exclusions.md index 68b85410f..bafdb5416 100644 --- a/docs/design/decisions/vocabulary-exclusions.md +++ b/docs/design/decisions/vocabulary-exclusions.md @@ -62,7 +62,7 @@ Excluded (TITLES): - The per-word refusals are data since #750 (2026-10-19): each word refused a title reading — the Indic ঠাকুর/ठाकुर, कुमारी/কুমারী, बेगम/বেগম, शेख/শেখ, आचार्य and राजा/रानी (#343, #344), and #606's ông/bà, thiru, puan, pan, rodina, gróf, cik, paron, tikin, Latin janab, marshal and justice, #662's malam (Derek's decline) and bay, #660's shehu, and #789's wm (2026-10-22) — is an `excluded` entry in its language's file with its reason, and with its counts as name readings where they were taken. `uv run python tools/vocab/query.py --excluded title` lists them. A contributor who adds one back meets the reason as a build error. - shehu's counts do not settle a graf-class weighing, and must not be read as if they did: its one leading-position figure, the Wikidata given-name count on its entry, is a lower bound (Shagari's own entry has no given-name statement), and its other counts are family-name bearers, a position C-i does not ask about. The evidence that Shehu leads a name is not a count. It is Shagari himself, and CLDR's Hausa sample data, whose six renderings carrying the word all file it as the given name (`grep '"given": "Shehu"' tools/differential/cldr_fields.jsonl`). It arrived with an early bulk import (1a51555f) and was never put to C-i before #660. Whoever re-argues it counts Hausa given-name use first; the accepted cost is the honorific use, `Shehu Abubakar` reading given Shehu as any untitled name does, and the reading belongs in an opt-in pack if anyone asks for it. malam is the opposite case: CLDR's Hausa data files Malam as the title (`grep '"title": "Malam"' tools/differential/cldr_fields.jsonl`), so its refusal is a decline on the counts, not a shehu-style removal. - The class-level half of the Indic refusals stays here: the divergence on शेख/শেখ is deliberate and is the sri/shri case run backwards — the Latin sheikh cluster SHIPS as given-name titles for the Arabic addressing form, while the Indic spellings name the family. shehu's removal moved three CLDR corpus names (`Shehu Muhammed`, `Muhammed, Shehu`, `Modibbo, Shehu Bello`), all three now agreeing with CLDR on every role (moves: `uv run python tools/differential/snapshot.py --since v2.4.1`). -- The trailing-position rule that must NOT be adopted, and 2026-09-08 says what the line is: TITLES holds hundreds of words in no suffix set, at least nineteen of them ordinary English surnames (king, judge, bishop, baron, sheriff, ...), so a blanket "vocabulary outranks position in the trailing slot" reading would turn "Mary Jane King" into title="King" with the family name gone. The leading half of this argument is AGENTS.md's "Dean is deliberately absent" gotcha; this is the trailing half, and it shadows the family name rather than the given (#316). What #316 settled is that the prohibition is on BARE words: "Mary Jane King" still reads family King, while "John Smith Prof." reads title Prof. **The second half of that argument is RETRACTED 2026-09-09**, in review of the docs commit: "a period-marked trailing word being one nobody writes as a surname" claimed the period keeps the collision out of the trailing slot, and it does not. Measured on the branch tree — "Mary Jane King." reads title "King.", given Mary, family Jane, the family name shadowed exactly as the bare reading would have shadowed it, and "John Smith Judge." reads title "Judge." The reach is the vocabulary rather than a handful: of the 746 titles in no suffix set, 627 read as a trailing title once a period is written behind them, and every one of the 119 that do not is kept out by the abbreviation SHAPE — a digit, a hyphen, an apostrophe, or a script whose letters carry combining marks (rules/H2 records that half) — no plain ASCII-letter title missing at all. So king, judge, bishop, baron and sheriff ARE reachable in the trailing slot; what the prohibition above buys is the bare spelling and nothing more. The period is a WRITING convention the rule can read, not evidence about the word, and reading it is an ACCEPTED COST under the premise that the input is a name (rules.md H Background): someone who ends a name with a period-marked word has written an abbreviation. rules/H5 carries it as an Accepted clause with "Mary Jane King." as the executable example. (Since 2026-10-26 `king` is a title only inside a title phrase, decisions/H9, #752 phase 2j: "Mary Jane King." reads family "King." and the surnames this paragraph names are pope, judge, bishop, baron and sheriff, rules/H5's example "Mary Jane Pope.".) Recompute the reach with `L = Parser().lexicon; ws = [w for w in L.titles - L.suffix_acronyms - L.suffix_words if " " not in w]; len([w for w in ws if parse("Mary Jane %s." % w).title])` against `len(ws)` — 627 of 746 on 2026-09-09 — the digits move as the vocabulary grows and neither argument does. See #H5. The bare prohibition has one exception since #835 (decisions/H8): a listed salutation ending the given part after a family comma is the title, the comma having marked the surname's place and the salutations borne as names being excluded. +- The trailing-position rule that must NOT be adopted, and 2026-09-08 says what the line is: TITLES holds hundreds of words in no suffix set, at least nineteen of them ordinary English surnames (king, judge, bishop, baron, sheriff, ...), so a blanket "vocabulary outranks position in the trailing slot" reading would turn "Mary Jane King" into title="King" with the family name gone. The leading half of this argument is AGENTS.md's "Titles permanently shadow first names" gotcha, whose example is Dean; this is the trailing half, and it shadows the family name rather than the given (#316). What #316 settled is that the prohibition is on BARE words: "Mary Jane King" still reads family King, while "John Smith Prof." reads title Prof. **The second half of that argument is RETRACTED 2026-09-09**, in review of the docs commit: "a period-marked trailing word being one nobody writes as a surname" claimed the period keeps the collision out of the trailing slot, and it does not. Measured on the branch tree — "Mary Jane King." reads title "King.", given Mary, family Jane, the family name shadowed exactly as the bare reading would have shadowed it, and "John Smith Judge." reads title "Judge." The reach is the vocabulary rather than a handful: of the 746 titles in no suffix set, 627 read as a trailing title once a period is written behind them, and every one of the 119 that do not is kept out by the abbreviation SHAPE — a digit, a hyphen, an apostrophe, or a script whose letters carry combining marks (rules/H2 records that half) — no plain ASCII-letter title missing at all. So king, judge, bishop, baron and sheriff ARE reachable in the trailing slot; what the prohibition above buys is the bare spelling and nothing more. The period is a WRITING convention the rule can read, not evidence about the word, and reading it is an ACCEPTED COST under the premise that the input is a name (rules.md H Background): someone who ends a name with a period-marked word has written an abbreviation. rules/H5 carries it as an Accepted clause with "Mary Jane King." as the executable example. (Since 2026-10-26 `king` is a title only inside a title phrase, decisions/H9, #752 phase 2j: "Mary Jane King." reads family "King." and the surnames this paragraph names are pope, judge, bishop, baron and sheriff, rules/H5's example "Mary Jane Pope.".) Recompute the reach with `L = Parser().lexicon; ws = [w for w in L.titles - L.suffix_acronyms - L.suffix_words if " " not in w]; len([w for w in ws if parse("Mary Jane %s." % w).title])` against `len(ws)` — 627 of 746 on 2026-09-09 — the digits move as the vocabulary grows and neither argument does. See #H5. The bare prohibition has one exception since #835 (decisions/H8): a listed salutation ending the given part after a family comma is the title, the comma having marked the surname's place and the salutations borne as names being excluded. Excluded (Policy.script_orders defaults): Script.KATAKANA is deliberately absent — a pure-katakana token is predominantly a transcribed foreign name kept in its source order, so nothing defaults on it (rule W4's boundary). Noted 2026-08-15: of the three Script-keyed axes, this is the one with no force-a-decision guard (mechanisms.md#FORCE-A-DECISION-TABLE), so a new Script member silently gets no order. diff --git a/nameparser/_pipeline/AGENTS.md b/nameparser/_pipeline/AGENTS.md index 1b917d684..9c03b957e 100644 --- a/nameparser/_pipeline/AGENTS.md +++ b/nameparser/_pipeline/AGENTS.md @@ -2,6 +2,10 @@ Loaded when a session reads or edits a pipeline stage. The root AGENTS.md carries the 2.0 conventions every stage follows (one stage per module, the `Consumes:`/`Produces:`/`Reads:` header, the frame budget, checkable stage-attribution claims); this file carries what only work inside the stages needs. The contract behind the ambiguity rules below is mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE, and rules/A1 is its observable side. +## A new inner loop needs a scaling guard + +A loop that grows with the input -- per token, per member, per pair -- needs a row in `tests/v2/test_benchmark.py`, whose module docstring says which table; the absolute-time tests there use constant-size input and are blind to a complexity regression. + ## Ambiguity reports ### Emit at the decision site diff --git a/nameparser/config/capitalization.py b/nameparser/config/capitalization.py index 8a8bc74b9..dcb94bb22 100644 --- a/nameparser/config/capitalization.py +++ b/nameparser/config/capitalization.py @@ -3,7 +3,8 @@ # `cased` property of the word's reading, one per word, and # tools/vocab/build.py refuses one that does not spell its word's letters # and digits (#459) before Lexicon ever sees it. Still a plain dict, the -# one vocabulary constant not frozen (AGENTS.md, "Configuration layer"). +# one vocabulary constant not frozen (AGENTS.md's vocabulary and +# configuration-layer section, "Neither dict was frozen"). from nameparser.config import _compiled CAPITALIZATION_EXCEPTIONS = _compiled.CAPITALIZATION_EXCEPTIONS diff --git a/nameparser/vocab/AGENTS.md b/nameparser/vocab/AGENTS.md new file mode 100644 index 000000000..29a306430 --- /dev/null +++ b/nameparser/vocab/AGENTS.md @@ -0,0 +1,31 @@ +# Adding a word to nameparser/vocab/ + +Loaded when a session reads a vocabulary TOML file. The steps for adding a word are in `nameparser/vocab/README.md`, the reference for anyone editing these files; this file holds the traps those steps have run into, filed under the README step each belongs to. Each names the PR that found it. + +## Step 1: its other readings + +First check the word's OTHER readings: `uv run python tools/vocab/query.py --word ` lists every reading and exclusion of it in every file, which is where the overlaps between categories now show. Real overlaps exist: `st`/`freiherr` ∈ `PARTICLES` ∩ `TITLES`, and `do`/`mc`/`vd` ∈ `PARTICLES` ∩ `SUFFIX_ACRONYMS`; `abd` = "ABD" ∈ `SUFFIX_ACRONYMS`; `abu` ∈ `PARTICLES` ∩ `BOUND_GIVEN_NAMES` (position-dependent: leading token → given-name join, mid-name → family-name join). Usually position-dependent and harmless, but can force a guard or an exclusion (the `last_base` all-particles guard; `abd`, kept out of `BOUND_GIVEN_NAMES` until decisions/P5 found the credential collision cost nothing at either end). + +## Step 2: real names, not memory + +Then check the word against real names, not memory: `uv run python tools/name_usage.py ` counts its bearers as a given name and as a family name (US Census surnames, SSA given names, Wikidata with countries of citizenship), and gives its given-name frequency band per country from Jörg Michael's `nam_dict.txt` (54 countries and an "other countries" column, GFDL, downloaded into the cache and never committed) — the one source that sees a given name outside the US and Wikidata's lower bound, such as Korean `In` (band A in Korea) or Vietnamese `Bay` (6 in Vietnam). A band is a frequency class, not a count, and the dictionary is a selection, so a miss there (`Shehu`) proves nothing; and because a truncated source reads as a run of false zeros, a pinned download is checked against its sha256 on every load from the cache, never against a floor on what it holds — #775's first fix refused fewer than 40,000 names and passed a `nam_dict.txt` missing only its last line (`Zyta`). Pass a name as it is written (`McDonald`): Wikidata labels match exactly, and no rule rebuilds an internal capital from `mcdonald`. A word in a non-Latin script is counted by its romanization too, since the Census, SSA and nam_dict hold only Latin names, and that count weighs only where its bearers write the script: #835 vetted its salutations by their own spellings and its review found `बाबू` flagged while `babu` is a US surname (1,496) and an Indian given-name element (Ram Babu), where the romanization of `মি`, `Mi`, counts Korean and Chinese bearers who never write Bengali. It reports and does not decide — read the counts against the position the set's claim acts on. For a suffix it also has the other side: `--fetch-nppes` counts the US healthcare providers who write the word as a credential (NPPES, the 4.8 million credentialed providers in the pinned August 2026 file, so blind to law, accounting and honours), and `--rank N --unlisted` lists the commonest credentials the vocabulary misses, with where each lands in `John Smith, X` -- the list #797 came from. Where a local corpus is fetched (`tools/differential/local_corpora.py`, mechanisms.md#NAME-DATA-JOBS), a row says how many of its records (for ClinicalTrials.gov, officials) write the word, and how many after a comma: ClinicalTrials.gov's study officials show where a credential actually stands in a name as typed. + +## Step 2, for a glued word: does any name, or any longer word, end in it? + +A word that would peel GLUED (`GLUED_HONORIFICS`) asks a different question, whether any name ENDS in it, and for a CJK word the tool answers that too: an ends-with row from a cached index of Wikidata's CJK name labels (built on first use; 33 minutes on 2026-10-17), plus the names the word ends across the family/given join (阿部 + 長 for 部長). Read the tool's hits by item before counting them, because labels include transcriptions and mislabels: #707's one hit for 二世 was 厄德二世, the given name Odo with a regnal numeral in its Chinese label, borne by 109 Western people named Odo. Then ask a third question the tool cannot: does a common longer word end in it? The peel matches the end of a token, spaced or glued, so a listed tail splits every unlisted compound that ends in it — #707 vetted 議員, 社長 and 医学博士 against names and its review found 衆議院議員, 副社長 and 獣医学博士 cut at the tail (`田中 副社長` → given 副); no corpus held a compound office word until #707's rules examples added two, so the snapshot was silent about the class. List the common compounds whole or record the split as a cost. + +## Step 3: the predicates that read a member's spelling + +Check the predicates that test a member's SPELLING rather than its membership: they encode what the set's members happened to look like when they were written. `_pieces.starts_a_credential_run` implemented the credential run's "Latin" (rules/S8 now, part of S2 until #682 split it) as `isascii()`, exact until #661 added `mlađi`, which would have read `mlađi Jones` differently from `stariji Jones`; the design-docs review caught it, after a first draft had credited the test to the wrong clause of S2 and proposed deleting it on a mutation that moved nothing because nothing pinned it. (#679 then dropped the script test on purpose, after measuring what it protected.) Before adding the first member of a new script, or the first accented one, grep the set's consumers for `isascii`, `isalpha` and script tests, and read the rule each one implements. The same goes for the first member holding PUNCTUATION: #700 added `d'` and `az-`, the parse read them right and every snapshot move was an intended one, and `capitalized()` still wrote `D' Ivoi`, because case repair splits a token at its apostrophe or hyphen and asks the vocabulary of the stem. The snapshot records both case views since #826, but only for the names the corpora hold, so parse such a member's names through `capitalized()` and `capitalized(force=True)` in each case before calling it done. + +## Step 4: record the vetting on the word + +Record what the vetting found ON the word, not in a comment or a decisions sentence: `note`, `issue`, a `given`/`family` reading with dated counts, and for a word you decided to keep out an `excluded` entry with its reason, so the next contributor who adds it meets the reason as a build error (decisions/vocabulary-model). A name reading is not only provenance on a word that has a credential, particle or title reading: it IS the clash, and derives the marker (an ambiguous acronym, a particle that may be given, since 2026-10-25, a generation or agnomen that may be a name, rules/S11, since phase 2h the same day, so given-name counts recorded on `junior` would gate it, and a title that may be a given name, rules/H9, since phase 2i), so counts kept as evidence AGAINST marking such a word go in the clashing reading's `note` instead, as `ld`'s do. + +## Step 6: the user docs' doctests + +The user docs use real words as examples of an UNLISTED one, and listing the word breaks the example: #797 listed `dphil`, and docs/customize.rst's capitalization example (`parse("jane smith dphil").capitalized()` giving `Dphil`) failed in all five CI build jobs while `uv run pytest` passed, because `sphinx-build -b doctest docs` is a separate step. Run it after adding a word, and grep `docs/*.rst` for the word; the example now uses `dclinpsy`, a credential left unlisted on purpose, so list it only after moving the example. + +## When a word's reading moves, or a new derivation reads old readings + +A word whose reading moves may be the worked example of the class it is leaving: `king` was the rules' and tests' title-that-is-also-a-surname across 35 files when #901 gave it a given reading, and five tests went vacuous before the review found them. Grep rules, decisions Standing lines, the RFCs, AGENTS.md, docs/*.rst, code comments and tests for the word before measuring, and replace it with a stand-in checked to read exactly as the word read on master in every example (`pope` there). A new derivation reads readings other work recorded for another purpose, so list what it would mark before shipping it: phase 2i's first build found thirteen salutations whose given readings #835 had kept as H8's evidence (`mr` among them, so `Mr Smith` would have read given 'Mr'), and moved their counts into notes. diff --git a/nameparser/vocab/README.md b/nameparser/vocab/README.md index 0266b78a8..f1949b85e 100644 --- a/nameparser/vocab/README.md +++ b/nameparser/vocab/README.md @@ -246,7 +246,7 @@ Record a count on the word as a `given` or `family` reading, with the date you t ## Adding a word 1. **Look up the word's other readings.** `uv run python tools/vocab/query.py --word ` lists every reading and exclusion of it in every file. An exclusion there is a decision already made; a reading in another category is a clash you are about to create, which is fine when position settles it (`do`) and needs an argument when it does not. -2. **Check it against real names, not memory.** `uv run python tools/name_usage.py ` gives the counts (add `--fetch-nppes` for a suffix, to see how often it is written as a credential), and [decisions/vocabulary-collisions](../../docs/design/decisions/vocabulary-collisions.md) is the criterion they are read against: a word that is a real name in the position the reading acts on needs a name reading with those counts, which marks the clash (see [Derived markers](#derived-markers)), or should stay out. The root `AGENTS.md`'s "Adding a word to the vocabulary" lists the traps that have bitten before. +2. **Check it against real names, not memory.** `uv run python tools/name_usage.py ` gives the counts (add `--fetch-nppes` for a suffix, to see how often it is written as a credential), and [decisions/vocabulary-collisions](../../docs/design/decisions/vocabulary-collisions.md) is the criterion they are read against: a word that is a real name in the position the reading acts on needs a name reading with those counts, which marks the clash (see [Derived markers](#derived-markers)), or should stay out. `nameparser/vocab/AGENTS.md` lists the traps that have bitten before, filed under these steps. 3. **Check how it is written.** The entry's `text` is the folded spelling (`mbchb`), and `capitalized()` writes an acronym credential in capitals unless the word has a `cased` mask, so one that people write in mixed case (`MBChB`, `DPhil`, `ScD`) needs the mask: `cased = "MBChB"`. Nothing in the build can see this, since no source it reads records case. The ClinicalTrials.gov row of `name_usage.py` lists the spellings as typed ("written MBChB 1,442, MBCHB 72, MbChB 28"), once that local corpus is fetched (`uv run python tools/differential/local_corpora.py fetch ctgov-officials`). Where a mask would cost a name its rendering, refuse it instead, with the reason, as `edd` does (`'edd smith'` would render `'EdD Smith'`). #806 added twelve mixed-case credentials without their masks, and review caught it. 4. **Write the entry** in the file of the language whose names it occurs in (`und.toml` when that is not known), in its place in the [word order](#word-order), with what the checks found on it: `note`, `issue`, a dated name reading, a `cased` mask, or an `excluded` entry if you decided against it. 5. **Rebuild**: `uv run python tools/vocab/build.py`. It refuses what it can detect, naming the word and, where one file is at fault, the file, and rewrites `nameparser/config/_compiled.py`. Commit both. diff --git a/tests/v2/test_benchmark.py b/tests/v2/test_benchmark.py index 515e0a2ed..1ac15dc1c 100644 --- a/tests/v2/test_benchmark.py +++ b/tests/v2/test_benchmark.py @@ -21,6 +21,44 @@ _PREFIXED_SHAPES exists for a shape that needs a prefix before its repeated run, which _SHAPES cannot express (#553). + +Which guard a new shape needs (the comment beside each table carries +the measurements behind its rule): + +- A unit repeated at n and 4n that a bare parse() reaches: a _SHAPES + row, timed on the clock. The rows cover different dimensions, and + the comments at the table say which dimension each row alone covers + and what its unit must hold to reach the code -- measure before + pruning one. Calibrate _MAX_RATIO against the WEAKEST quadratic's + signal, and confirm a planted regression fails across REPEATED runs. +- A stage gated on an opt-in Policy field: a _POLICY_SHAPES row, with + the reachability probe every row carries. +- A shape that needs a PREFIX before its run, whose cost is + Python-level: a _REREAD_SHAPES row, counted in the frames of the one + function the defect re-runs, or a frame test of its own + (test_a_clause_link_run_does_not_cost_quadratically). +- A prefixed shape whose cost is C-level work (a list scanned by `in`, + a tuple re-hashed as a key), which emits no frame: a _PREFIXED_SHAPES + row, on the clock at its own base. That table skips under a line + tracer, and NAMEPARSER_REQUIRE_CLOCK_GUARDS turns the skip into a + failure (the comment at the table says why CI still runs it). +- An exponential, whose separating input length never finishes on the + clock: a frame-count pair, cheap pair first, then a longer pair that + can see a quadratic + (test_a_trailing_credential_run_does_not_cost_exponentially). + +A frame guard either counts the whole parse (the exponential pair, the +clause-link test, _REREAD_SHAPES' `lead` row) or isolates ONE function +with `only=`. An isolated guard goes blind when the code under it stops +calling that function, so when that code moves, move `only=` with it +and re-take its negative control (#563, where #613's move left the +guard reading 3.6x on a returned quadratic). A whole-parse guard's +ratio includes the parse's linear work, which its calibration accounts +for. Either kind needs the same reachability assertion a _POLICY_SHAPES +row carries. Frame +counts do not move under load, so they need no repeated-run +calibration. The module is marked `serial` (tests/conftest.py says +why), and a new clock-timed test elsewhere takes the mark too. """ import gc import os @@ -225,7 +263,7 @@ def test_a_thousand_names_still_parse_in_reasonable_time( # before pruning one that looks redundant: # # dimension covered by -# token count all ten +# token count every row # piece count unmatched_*, plain_tokens, commas, titles # SEGMENT count commas ONLY -- deleting it leaves every # segment-keyed regression unguarded @@ -277,6 +315,16 @@ def test_a_thousand_names_still_parse_in_reasonable_time( # family. 'PhD MA ' reads the # same and never asks the pass at all, the # capitals deciding each member first +# gated-word front scan agnomens (#890), carried_agnomens (the +# scan through carried members) +# S13 clause pass bracket_members (#810) +# +# The last three rows were added after the census above was taken; on +# 2026-10-27 none of them reached S7's anchor pass or the honorific +# peel (counted in calls to credential_anchors and +# _peel_honorific_tail at 20 units), and none holds a comma, a +# non-ASCII character, a bound given name, a maiden marker or a +# connective, so the ONLY claims above still hold. _SHAPES = { "delimiter_pairs": "(a) ", # extract: matched pairs -> masked spans "quote_pairs": '"a" ', # extract: the open==close path @@ -551,7 +599,7 @@ def test_prefixed_cost_grows_no_worse_than_linearly( # Every _SHAPES entry runs bare parse(), and a stage gated on an opt-in # Policy field is dead there -- #329's clause loop exits on its first # line when maiden_delimiters is empty, so the quadratic it shipped -# with was invisible to all ten shapes above even though "(a) " was +# with was invisible to every shape above even though "(a) " was # already one of them. The unit is the SAME string as delimiter_pairs; # only the policy differs, which is the whole point. # diff --git a/tests/v2/test_vocab.py b/tests/v2/test_vocab.py index 6daece877..24f7c3ba8 100644 --- a/tests/v2/test_vocab.py +++ b/tests/v2/test_vocab.py @@ -108,7 +108,7 @@ def test_the_toml_ships_as_package_data() -> None: globs = pyproject["tool"]["setuptools"]["package-data"]["nameparser"] assert "vocab/*.toml" in globs assert not [p for p in build.VOCAB.rglob("*") - if p.is_file() and p.suffix != ".toml" and p.name != build.README] + if p.is_file() and p.suffix != ".toml" and p.name not in build.SKIPPED] # --------------------------------------------- controls on the shipped data @@ -789,8 +789,10 @@ def test_an_empty_word_is_a_usage_error() -> None: ({}, "no vocabulary files"), ({"EN.TOML": OK}, "BCP 47 tag"), # misnamed files are read, then refused ({"en.toml.orig": OK}, "BCP 47 tag"), - ({"readme.md": OK}, "BCP 47 tag"), # only README.md, by its exact name, is skipped + ({"readme.md": OK}, "BCP 47 tag"), # only README.md and AGENTS.md, by exact name, are skipped ({"README.md": OK}, "no vocabulary files"), + ({"agents.md": OK}, "BCP 47 tag"), + ({"AGENTS.md": OK}, "no vocabulary files"), ({"sub/en.toml": OK}, "no subfolders"), ({"en.toml": "->nowhere.toml"}, "a link to nothing"), ]) diff --git a/tools/vocab/build.py b/tools/vocab/build.py index bc9d58206..561f140e4 100644 --- a/tools/vocab/build.py +++ b/tools/vocab/build.py @@ -160,8 +160,13 @@ def _alnum(text: str) -> str: VOCAB = ROOT / "nameparser" / "vocab" OUTPUT = ROOT / "nameparser" / "config" / "_compiled.py" REFRESH = "uv run python tools/vocab/build.py" -#: The folder's reference for editors, the one file read_dir skips by name. +#: The folder's reference for editors. README = "README.md" +#: The agent notes beside it (nameparser/vocab/AGENTS.md), loaded by +#: coding agents that read a file in the folder. +AGENTS = "AGENTS.md" +#: The files read_dir skips, by exact name. +SKIPPED = frozenset({README, AGENTS}) #: The format version this compiler reads. A new kind or property is #: additive and does not bump it; a change to what an existing key means @@ -1347,15 +1352,16 @@ def compile_pool(pool: Pool) -> Compiled: def read_dir(path: Path = VOCAB) -> dict[str, str]: """Every file in the vocabulary folder, by name. Every entry but a - dotfile and README.md, the folder's reference, is read, so a - misnamed one (EN.TOML, en.toml.orig, readme.md) reaches check 1's + dotfile and the two by-name skips (README.md, the folder's reference, + and AGENTS.md, its agent notes) is read, so a misnamed one + (EN.TOML, en.toml.orig, readme.md, agents.md) reaches check 1's file-name check instead of being skipped, and a missing or empty folder is an error rather than an empty vocabulary.""" if not path.is_dir(): raise VocabError(f"{path}: no vocabulary folder there") files = {} for f in sorted(path.iterdir()): - if f.name.startswith(".") or f.name == README: + if f.name.startswith(".") or f.name in SKIPPED: continue if f.is_symlink() and not f.exists(): raise VocabError(f"{f.name}: a link to nothing ({f.readlink()})")