Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ Quick Start Example
>>> russian.parse("Сидоров Иван Петрович").family
'Сидоров'
>>> locales.available()
('ja', 'ne', 'ru', 'tr_az', 'zh')
('cs', 'hr', 'ja', 'ne', 'ru', 'sk', 'sl', 'tr_az', 'zh')

Learn more
----------
Expand Down
2 changes: 1 addition & 1 deletion docs/design/decisions/M3-marker-led-clause.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
- A rule that extends where a vocabulary claim acts re-asks C-i of the whole set it reads; M3 made the marker vocabulary act on the first word of any bracketed clause, and the re-ask removed `roz` (`Rosalind Roz Smith` reads no maiden name, middle 'Roz' under the default order) — 2026-08-26 #335, M3 makes.
- Declaring `maiden_delimiters` is a declaration about the data, still needed for markerless and one-word clauses: `Cherice J. (Johnson) Williams` reads nickname 'Johnson' by default — 2026-08-26 #335, what configuring.

Elsewhere: a marker followed by a comma, standing alone or glued to either word, is no marker (`Jane (née , Jones) Smith` → nickname 'née , Jones') — decisions/N1's 2026-10-12 #653 entry; the delimiters trumping every reading inside the clause (`Jane Doe (nee Smith MA)` → maiden 'Smith MA') — decisions/M2's 2026-09-19 (Derek), #533 review — THE DELIMITERS TRUMP entry; the bare take's first-word floor, which reads `Jane Smith née V` as maiden 'V' — decisions/M2's 2026-10-04 (Derek), #601 entry; `roz`'s exclusion, now on the word (`uv run python tools/vocab/query.py --word roz`) — decisions/vocabulary-exclusions' Excluded (MAIDEN_MARKERS) block.
Elsewhere: a marker followed by a comma, standing alone or glued to either word, is no marker (`Jane (née , Jones) Smith` → nickname 'née , Jones') — decisions/N1's 2026-10-12 #653 entry; the delimiters trumping every reading inside the clause (`Jane Doe (nee Smith MA)` → maiden 'Smith MA') — decisions/M2's 2026-09-19 (Derek), #533 review — THE DELIMITERS TRUMP entry; the bare take's first-word floor, which reads `Jane Smith née V` as maiden 'V' — decisions/M2's 2026-10-04 (Derek), #601 entry; `roz`, refused as a default marker and since 3.0 a reading off by default that the Czech pack turns on (`uv run python tools/vocab/query.py --word roz`) — decisions/vocabulary-exclusions' Excluded (MAIDEN_MARKERS) block.

Open: the fullwidth-colon marker (`旧姓:佐藤` arriving as one word) is decisions/M2's Open line for #317.

Expand Down
3 changes: 2 additions & 1 deletion docs/design/decisions/deviates-registry.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Standing

- A locale pack declares its deviations from the defaults as a module-level `DEVIATES` callable, acceptance metadata kept off `Locale`, whose value equality and pickling a callable field would break; `test_registry_is_the_pack_contract` fails unless every registered pack ships `DEVIATES` and a rotator list, which is what lets mechanisms.md#LOCALE-PACKS-PURE-DATA promise packs no code paths of their own. A field on `Locale` is to be revisited only if a runtime consumer of the declaration, such as an `explain()`, appears, and none has — 2026-07-18. A vocabulary pack's deviations are derivable instead, from the reading index (`_reading_index.changed_words`), which tests/v2/test_locales.py's choice gate reads; the gate ors the derived surface into every registered pack with `languages`, so a hand `DEVIATES` declares only what a fragment or a patch beside a choice changes; the contract test refuses a `DEVIATES` on a pack that is nothing but a choice of languages (`locales.NE`, the first) and requires one of every other pack — 2026-10-29 #753, phase 3b; 2026-10-30 #753, phase 3d.
- A locale pack declares its deviations from the defaults as a module-level `DEVIATES` callable, acceptance metadata kept off `Locale`, whose value equality and pickling a callable field would break; `test_registry_is_the_pack_contract` fails unless every registered policy or segmentation pack ships `DEVIATES`, and every registered pack a rotator list, which is what lets mechanisms.md#LOCALE-PACKS-PURE-DATA promise packs no code paths of their own. A field on `Locale` is to be revisited only if a runtime consumer of the declaration, such as an `explain()`, appears, and none has — 2026-07-18. A vocabulary pack's deviations are derivable instead, from the reading index (`_reading_index.changed_words`), which tests/v2/test_locales.py's choice gate reads; the gate ors the derived surface into every registered pack with `languages`, so a hand `DEVIATES` declares only what a fragment or a patch beside a choice changes; the contract test refuses a `DEVIATES` on a pack that is nothing but a choice of languages (`locales.NE`, the first, and `CS`, `SK`, `SL` and `HR`), and a choice pack whose choice changes no word, and requires one of every other pack — 2026-10-29 #753, phase 3b; 2026-10-30 #753, phase 3d; 2026-10-30 #753, phase 3f.

Open: none in this file.

Expand All @@ -11,3 +11,4 @@ Open: none in this file.
- 2026-07-18 (d4aaafa; the DEVIATES design note) — a `deviates` field ON Locale was rejected: callables break value equality and pickle-by-reference, and DEVIATES is acceptance metadata, not runtime behavior (revisit only if a runtime consumer like explain() appears). The REGISTRY became the contract instead: test_registry_is_the_pack_contract fails structurally unless every registered pack ships DEVIATES and its rotator list. This is why LOCALE-PACKS-PURE-DATA can promise "no code paths of their own" and mean it.
- 2026-10-29 #753, phase 3b — A VOCABULARY PACK'S DEVIATIONS ARE DERIVED, NOT DECLARED. `Locale.languages` selects among the default's readings, so the words a choice can move are computable: `_reading_index.changed_words(chosen)` is every word joining or leaving a compiled constant under the choice, and tests/v2/test_locales.py's `test_non_interference_each_choice` requires the default's reading for every corpus name holding none of them, with a negative control (`test_the_choice_gate_fails_when_its_surface_misses_a_word`). No pack ships in 3b, so the gate runs over choices of the files' tags; the registry contract test still asks every REGISTERED pack for `DEVIATES` and a rotator list, and learns to read a derived surface for a pack with `languages` in the PR that registers the first one (3d). The policy and segmentation packs (JA, RU, TR_AZ, ZH) keep theirs, decisions/vocabulary-model's 2026-10-27 #753 entry, item 7.
- 2026-10-30 #753, phase 3d — THE CONTRACT READS A DERIVED SURFACE. `locales.NE` registers the first pack whose content is a choice of languages and nothing else (`Locale.lexicon` empty, `Locale.policy` the empty patch). tests/v2/test_locales.py's `_deviates(code)` gives the gate, the rotator test and the combined gate every pack's predicate: the surface derived from its `languages` through `changed_words` and the 3b choice gate's `_surface_deviates`, or'd with its `DEVIATES`, so a pack with a fragment or a patch beside a choice needs to declare only that part (`test_a_pack_with_a_choice_and_a_patch_declares_only_the_patch`; nothing refuses a hand copy of the choice's words there); `test_registry_is_the_pack_contract` refuses a `DEVIATES` on a pack that is nothing but its choice, so a hand copy cannot drift beside the derived one, and still requires one of every other pack. Its rotator list stays required, as the gate's positive side. decisions/vocabulary-model's 2026-10-30 phase 3d entry, item 2.
- 2026-10-30 #753, phase 3f — A CHOICE PACK MUST CHANGE A WORD. `locales.CS`, `SK`, `SL` and `HR` register as choice packs under phase 3d's contract, with no `DEVIATES`. `test_registry_is_the_pack_contract` adds one refusal: a choice pack whose choice changes no word (an empty `changed_words`), since its gate would pass on nothing; and `test_choice_rotators_cover_every_changed_word` holds each choice pack's rotator list to a name per changed word, so the gate's positive side reaches every word. The combined gate reads the surface of the union of every registered pack's languages, since `parser_for` applies one choice, not one per pack, or'd with every pack's hand `DEVIATES` whatever the pack, since a mixed pack carries both. decisions/vocabulary-model's 2026-10-30 phase 3f entry.
3 changes: 2 additions & 1 deletion docs/design/decisions/salutation-titles.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Closes #606 (#662 later added owura and bapak, its bullet below, and #847 cllr).
- A word that addresses or ranks a person and is written in FRONT of the name ships as a title unless it is borne as an ordinary name in the leading position, the position test of vocabulary-collisions' C-i, TITLES having no ambiguous subset to mark a collision in, so the only answers are ship and do not ship — 2026-10-04 #606.
- One exception to C-i, for TITLES alone: a word borne as a name only RARELY as the first word of one ships on the precedent of `graf` (`herra`, `batoni`, `hrabia`, `kreivi` and the rest of the graf class, each with its counts on its title reading's `note`), and no word in the particle or suffix sets, and no later title, takes it without its own counts — 2026-10-04 #606.
- The cost falls on a name that BEGINS with such a word: `Greve Anna` reads title 'Greve', family 'Anna' and `Herra Wijaya` title 'Herra', family 'Wijaya'; a declared `FAMILY_FIRST` does not help (`Hrabia Anna` reads title 'Hrabia' under it); the family comma keeps the word as the family where no name word stands behind it (`Hrabia, Anna` reads given 'Anna', family 'Hrabia', and so does `Hrabia Jr., Anna`), while a name word behind it makes it the title by H6 (`Hrabia Nowak, Anna` reads title 'Hrabia', family 'Nowak'); a name that ends in the word keeps it (`Gladys Knight`, `Knight, Gladys` read family 'Knight'), unless a period follows it (`Gladys Knight.` reads title 'Knight.', family 'Gladys', by H5, accepted rather than prevented) — 2026-10-04 #606.
- Words judged a COMMON leading-position name stay out (`thiru`, `puan`, `pan`, `rodina`, and `ông` and its pair `bà`), and words with no evidence either way are not shipped (`gróf`, `cik`, `paron`, `tikin`, `janab`), each an `excluded` entry in its language's file (`uv run python tools/vocab/query.py --excluded title --issue 606`) — 2026-10-04 #606.
- Words judged a COMMON leading-position name stay out (`thiru`, `puan`, `pan`, `rodina`, and `ông` and its pair `bà`), and words with no evidence either way are not shipped (`gróf`, `cik`, `paron`, `tikin`, `janab`), each an `excluded` entry in its language's file (`uv run python tools/vocab/query.py --excluded title --issue 606`) — 2026-10-04 #606. `pan` alone has since become a `salutation` reading written `default = false` in cs.toml and pl.toml, which a Czech or Polish choice turns on and the default does not read — 2026-10-30 #753, phase 3f.
- Accents are not folded, so `paní` and `härra` match only their accented spellings (`Paní Nováková` reads title 'Paní', `Pani Anna` reads given 'Pani'); folding the lexicon would turn both into collisions, and whoever proposes it re-argues the entry — 2026-10-04 #606.
- `gravos` was dropped, not excluded, since nothing showed it is the Greek word for Count; `κόμης`, a Greek surname written first in official lists, is neither added nor excluded, and adding it is a decision still to make — 2026-10-04 #606.
- `knt` (Knight, beside `kt`) is a suffix acronym with a `Knt` case mask: `Mary Knt` reads suffix 'Knt' — 2026-10-04 #606.
Expand All @@ -28,3 +28,4 @@ Elsewhere: `justice` and `marshal`, excluded here as common leading-position nam
- **2026-10-17 #662 — owura and bapak join TITLES; malam and bay are excluded.** Akan and Indonesian Mr., found in the CLDR corpus (`Owura Bertram…`, `Bapak Dwi Putro bin Adinata MP` read the salutation as the given name). Counts from `tools/name_usage.py` on 2026-10-17: owura and bapak each under 100 US Census bearers and none found on Wikidata (a zero proves nothing, per the tool). The two exclusions are `excluded` entries in ha.toml and tr.toml, with their counts (`uv run python tools/vocab/query.py --issue 662`). Measurement: two corpus names move, both in corpus_cldr.jsonl and both toward CLDR's fields (`Owura Bertram Wilberforce Henry Robert Wooster MP` and `Bapak Dwi Putro bin Adinata MP`, salutation from given to title); recompute with `uv run python tools/differential/snapshot.py --since <the commit before this entry>`. The release-log reading was checked against the 1.4.0, 2.0.0 and 2.4.2 wheels.
- **2026-10-22 #840 — the Italian abbreviated salutations join TITLES with case masks; sig and ing are excluded.** #838 made case repair write each full-stop chunk of a word as a word, so an unlisted `SIG.RA` repaired to 'Sig.Ra' where every release gave 'Sig.ra', and the remedy is a listed casing. Thirteen forms whose letters after the full stop are the word's own ending: `sig.ra` and `sig.na` (salutations, spellings of `signora` and `signorina`), `dott.ssa` and `prof.ssa` (academic), and the letter-heading styles `gent.mo`, `gent.ma`, `gent.le`, `ill.mo`, `ill.ma`, `ch.mo`, `ch.ma`, `preg.mo`, `preg.ma`, each with a `cased` mask. Listing them also settles the parse, which had read them by accident: `Sig.ra` and `Prof.ssa` were titles through a piece their joined word matched, and `Dott.ssa Anna Bianchi` read given 'Dott.ssa', middle 'Anna' on 1.4.0 and 2.5.0 (wheels, measured 2026-10-22); it reads title 'Dott.ssa', given 'Anna' now, under all three orders and every comma shape. None of the thirteen spells a name, each being an Italian abbreviation, so C-i's position test finds no bearer to weigh (a word with an interior full stop can be one in general: `J.R. Smith` reads given 'J.R.', rules/H2; the nearest shadow found is joined initials spelling `ch.ma`, `CH.MA ROSSI` now reading title 'CH.MA', with no bearer found); the committed corpora hold none (the snapshot moved only rules/R4's own `SIG.RA` row), and the class rests on standard Italian usage rather than a count. The bare `sig` (US SSA 50+ male, Wikidata 11+ given) and `ing` (nam_dict band 5 in Sweden, Wikidata 10+ given, US Census 1,814 family) are excluded with their counts (tools/name_usage.py, 2026-10-22); `Sig.` and `Ing.` already read as titles by H2's shape. A cost the `ing` exclusion does not decide: ClinicalTrials.gov officials write a bare `Ing` AFTER a comma, as a credential (52 of 536,981 write it, 43 after a comma, the same tool run), and 9 of the 28 distinct names holding it put it in the given field (`Viera Sevcikova, Ing` reads given 'Ing', family 'Viera Sevcikova'), where the dotted `Mario Rossi, Ing.` reads title 'Ing.'. That is a suffix question, a credential reading of `ing` beside the comma, and a title exclusion neither causes nor settles it. A trailing `Anna Bianchi Dott.ssa` keeps 'Dott.ssa' as the family, since H5 reads a trailing title only when it ends in a full stop, as `John Smith Sir` keeps 'Sir'; after a comma (`Maria Rossi, Sig.ra`) it is the title.
- **2026-10-22 #847 — `cllr` joins TITLES as `councillor` abbreviated.** British style writes it without the period, so `Cllr John Smith` read given 'Cllr', middle 'John' where `Cllr. John Smith` read title by shape. `tools/name_usage.py Cllr`: US Census surnames under 100, SSA under 5 births a year, not in nam_dict (a selection, so a miss proves little), and 0 on Wikidata as given or family (a lower bound: a zero proves nothing, per the tool), so no bearer was found as a name, and a five-letter vowelless abbreviation is not a plausible given name. It opens 508 of 15,214 names in `ukec-individuals` (cache data date 2026-10-23T03:45:09Z, same corpus sha as the first fetch of the issue), all bare; with `cllr` removed from `Lexicon.default()` the same 508 `as_dict()` readings differ and no others, 505 of them also moving the old middle into given. `Duke` and `Earl` stay out as borne names, by the caution above: they open 8 and 7 of those names (`name_usage.py` counts 8 and 15 names that write either anywhere). The accepted cost is `John Smith Cllr.`, whose period-marked trailing `Cllr.` was family and is now a title (rules/H5).
- **2026-10-30 #753, phase 3f — `pan` is a salutation off by default, for a Czech or Polish choice to turn on.** The 2026-10-04 exclusions in cs.toml and pl.toml become `salutation` readings written `default = false`, the collision with the Chinese surname Pan, which leads, kept in their notes (US Census 2010 12,664 bearers). The default does not read the word, so no parse moves; under `locales.CS` `Pan Jan Novák` reads title 'Pan' and `Pan Wei` title 'Pan', family 'Wei', the cost of choosing the language. decisions/vocabulary-model's phase 3f entry holds the measurement.
Loading
Loading