diff --git a/README.rst b/README.rst index e625c5cd..5d040ae4 100644 --- a/README.rst +++ b/README.rst @@ -82,7 +82,7 @@ Quick Start Example >>> russian.parse("Сидоров Иван Петрович").family 'Сидоров' >>> locales.available() - ('ja', 'ne', 'ru', 'tr_az', 'zh') + ('cs', 'hr', 'ja', 'ne', 'ru', 'sk', 'sl', 'tr_az', 'zh') Learn more ---------- diff --git a/docs/design/decisions/M3-marker-led-clause.md b/docs/design/decisions/M3-marker-led-clause.md index b35a4d00..044127ea 100644 --- a/docs/design/decisions/M3-marker-led-clause.md +++ b/docs/design/decisions/M3-marker-led-clause.md @@ -11,7 +11,7 @@ - A rule that extends where a vocabulary claim acts re-asks C-i of the whole set it reads; M3 made the marker vocabulary act on the first word of any bracketed clause, and the re-ask removed `roz` (`Rosalind Roz Smith` reads no maiden name, middle 'Roz' under the default order) — 2026-08-26 #335, M3 makes. - Declaring `maiden_delimiters` is a declaration about the data, still needed for markerless and one-word clauses: `Cherice J. (Johnson) Williams` reads nickname 'Johnson' by default — 2026-08-26 #335, what configuring. -Elsewhere: a marker followed by a comma, standing alone or glued to either word, is no marker (`Jane (née , Jones) Smith` → nickname 'née , Jones') — decisions/N1's 2026-10-12 #653 entry; the delimiters trumping every reading inside the clause (`Jane Doe (nee Smith MA)` → maiden 'Smith MA') — decisions/M2's 2026-09-19 (Derek), #533 review — THE DELIMITERS TRUMP entry; the bare take's first-word floor, which reads `Jane Smith née V` as maiden 'V' — decisions/M2's 2026-10-04 (Derek), #601 entry; `roz`'s exclusion, now on the word (`uv run python tools/vocab/query.py --word roz`) — decisions/vocabulary-exclusions' Excluded (MAIDEN_MARKERS) block. +Elsewhere: a marker followed by a comma, standing alone or glued to either word, is no marker (`Jane (née , Jones) Smith` → nickname 'née , Jones') — decisions/N1's 2026-10-12 #653 entry; the delimiters trumping every reading inside the clause (`Jane Doe (nee Smith MA)` → maiden 'Smith MA') — decisions/M2's 2026-09-19 (Derek), #533 review — THE DELIMITERS TRUMP entry; the bare take's first-word floor, which reads `Jane Smith née V` as maiden 'V' — decisions/M2's 2026-10-04 (Derek), #601 entry; `roz`, refused as a default marker and since 3.0 a reading off by default that the Czech pack turns on (`uv run python tools/vocab/query.py --word roz`) — decisions/vocabulary-exclusions' Excluded (MAIDEN_MARKERS) block. Open: the fullwidth-colon marker (`旧姓：佐藤` arriving as one word) is decisions/M2's Open line for #317. diff --git a/docs/design/decisions/deviates-registry.md b/docs/design/decisions/deviates-registry.md index f899e3cd..9eabf393 100644 --- a/docs/design/decisions/deviates-registry.md +++ b/docs/design/decisions/deviates-registry.md @@ -2,7 +2,7 @@ ## Standing -- A locale pack declares its deviations from the defaults as a module-level `DEVIATES` callable, acceptance metadata kept off `Locale`, whose value equality and pickling a callable field would break; `test_registry_is_the_pack_contract` fails unless every registered pack ships `DEVIATES` and a rotator list, which is what lets mechanisms.md#LOCALE-PACKS-PURE-DATA promise packs no code paths of their own. A field on `Locale` is to be revisited only if a runtime consumer of the declaration, such as an `explain()`, appears, and none has — 2026-07-18. A vocabulary pack's deviations are derivable instead, from the reading index (`_reading_index.changed_words`), which tests/v2/test_locales.py's choice gate reads; the gate ors the derived surface into every registered pack with `languages`, so a hand `DEVIATES` declares only what a fragment or a patch beside a choice changes; the contract test refuses a `DEVIATES` on a pack that is nothing but a choice of languages (`locales.NE`, the first) and requires one of every other pack — 2026-10-29 #753, phase 3b; 2026-10-30 #753, phase 3d. +- A locale pack declares its deviations from the defaults as a module-level `DEVIATES` callable, acceptance metadata kept off `Locale`, whose value equality and pickling a callable field would break; `test_registry_is_the_pack_contract` fails unless every registered policy or segmentation pack ships `DEVIATES`, and every registered pack a rotator list, which is what lets mechanisms.md#LOCALE-PACKS-PURE-DATA promise packs no code paths of their own. A field on `Locale` is to be revisited only if a runtime consumer of the declaration, such as an `explain()`, appears, and none has — 2026-07-18. A vocabulary pack's deviations are derivable instead, from the reading index (`_reading_index.changed_words`), which tests/v2/test_locales.py's choice gate reads; the gate ors the derived surface into every registered pack with `languages`, so a hand `DEVIATES` declares only what a fragment or a patch beside a choice changes; the contract test refuses a `DEVIATES` on a pack that is nothing but a choice of languages (`locales.NE`, the first, and `CS`, `SK`, `SL` and `HR`), and a choice pack whose choice changes no word, and requires one of every other pack — 2026-10-29 #753, phase 3b; 2026-10-30 #753, phase 3d; 2026-10-30 #753, phase 3f. Open: none in this file. @@ -11,3 +11,4 @@ Open: none in this file. - 2026-07-18 (d4aaafa; the DEVIATES design note) — a `deviates` field ON Locale was rejected: callables break value equality and pickle-by-reference, and DEVIATES is acceptance metadata, not runtime behavior (revisit only if a runtime consumer like explain() appears). The REGISTRY became the contract instead: test_registry_is_the_pack_contract fails structurally unless every registered pack ships DEVIATES and its rotator list. This is why LOCALE-PACKS-PURE-DATA can promise "no code paths of their own" and mean it. - 2026-10-29 #753, phase 3b — A VOCABULARY PACK'S DEVIATIONS ARE DERIVED, NOT DECLARED. `Locale.languages` selects among the default's readings, so the words a choice can move are computable: `_reading_index.changed_words(chosen)` is every word joining or leaving a compiled constant under the choice, and tests/v2/test_locales.py's `test_non_interference_each_choice` requires the default's reading for every corpus name holding none of them, with a negative control (`test_the_choice_gate_fails_when_its_surface_misses_a_word`). No pack ships in 3b, so the gate runs over choices of the files' tags; the registry contract test still asks every REGISTERED pack for `DEVIATES` and a rotator list, and learns to read a derived surface for a pack with `languages` in the PR that registers the first one (3d). The policy and segmentation packs (JA, RU, TR_AZ, ZH) keep theirs, decisions/vocabulary-model's 2026-10-27 #753 entry, item 7. - 2026-10-30 #753, phase 3d — THE CONTRACT READS A DERIVED SURFACE. `locales.NE` registers the first pack whose content is a choice of languages and nothing else (`Locale.lexicon` empty, `Locale.policy` the empty patch). tests/v2/test_locales.py's `_deviates(code)` gives the gate, the rotator test and the combined gate every pack's predicate: the surface derived from its `languages` through `changed_words` and the 3b choice gate's `_surface_deviates`, or'd with its `DEVIATES`, so a pack with a fragment or a patch beside a choice needs to declare only that part (`test_a_pack_with_a_choice_and_a_patch_declares_only_the_patch`; nothing refuses a hand copy of the choice's words there); `test_registry_is_the_pack_contract` refuses a `DEVIATES` on a pack that is nothing but its choice, so a hand copy cannot drift beside the derived one, and still requires one of every other pack. Its rotator list stays required, as the gate's positive side. decisions/vocabulary-model's 2026-10-30 phase 3d entry, item 2. +- 2026-10-30 #753, phase 3f — A CHOICE PACK MUST CHANGE A WORD. `locales.CS`, `SK`, `SL` and `HR` register as choice packs under phase 3d's contract, with no `DEVIATES`. `test_registry_is_the_pack_contract` adds one refusal: a choice pack whose choice changes no word (an empty `changed_words`), since its gate would pass on nothing; and `test_choice_rotators_cover_every_changed_word` holds each choice pack's rotator list to a name per changed word, so the gate's positive side reaches every word. The combined gate reads the surface of the union of every registered pack's languages, since `parser_for` applies one choice, not one per pack, or'd with every pack's hand `DEVIATES` whatever the pack, since a mixed pack carries both. decisions/vocabulary-model's 2026-10-30 phase 3f entry. diff --git a/docs/design/decisions/salutation-titles.md b/docs/design/decisions/salutation-titles.md index 68c30933..361365a7 100644 --- a/docs/design/decisions/salutation-titles.md +++ b/docs/design/decisions/salutation-titles.md @@ -7,7 +7,7 @@ Closes #606 (#662 later added owura and bapak, its bullet below, and #847 cllr). - A word that addresses or ranks a person and is written in FRONT of the name ships as a title unless it is borne as an ordinary name in the leading position, the position test of vocabulary-collisions' C-i, TITLES having no ambiguous subset to mark a collision in, so the only answers are ship and do not ship — 2026-10-04 #606. - One exception to C-i, for TITLES alone: a word borne as a name only RARELY as the first word of one ships on the precedent of `graf` (`herra`, `batoni`, `hrabia`, `kreivi` and the rest of the graf class, each with its counts on its title reading's `note`), and no word in the particle or suffix sets, and no later title, takes it without its own counts — 2026-10-04 #606. - The cost falls on a name that BEGINS with such a word: `Greve Anna` reads title 'Greve', family 'Anna' and `Herra Wijaya` title 'Herra', family 'Wijaya'; a declared `FAMILY_FIRST` does not help (`Hrabia Anna` reads title 'Hrabia' under it); the family comma keeps the word as the family where no name word stands behind it (`Hrabia, Anna` reads given 'Anna', family 'Hrabia', and so does `Hrabia Jr., Anna`), while a name word behind it makes it the title by H6 (`Hrabia Nowak, Anna` reads title 'Hrabia', family 'Nowak'); a name that ends in the word keeps it (`Gladys Knight`, `Knight, Gladys` read family 'Knight'), unless a period follows it (`Gladys Knight.` reads title 'Knight.', family 'Gladys', by H5, accepted rather than prevented) — 2026-10-04 #606. -- Words judged a COMMON leading-position name stay out (`thiru`, `puan`, `pan`, `rodina`, and `ông` and its pair `bà`), and words with no evidence either way are not shipped (`gróf`, `cik`, `paron`, `tikin`, `janab`), each an `excluded` entry in its language's file (`uv run python tools/vocab/query.py --excluded title --issue 606`) — 2026-10-04 #606. +- Words judged a COMMON leading-position name stay out (`thiru`, `puan`, `pan`, `rodina`, and `ông` and its pair `bà`), and words with no evidence either way are not shipped (`gróf`, `cik`, `paron`, `tikin`, `janab`), each an `excluded` entry in its language's file (`uv run python tools/vocab/query.py --excluded title --issue 606`) — 2026-10-04 #606. `pan` alone has since become a `salutation` reading written `default = false` in cs.toml and pl.toml, which a Czech or Polish choice turns on and the default does not read — 2026-10-30 #753, phase 3f. - Accents are not folded, so `paní` and `härra` match only their accented spellings (`Paní Nováková` reads title 'Paní', `Pani Anna` reads given 'Pani'); folding the lexicon would turn both into collisions, and whoever proposes it re-argues the entry — 2026-10-04 #606. - `gravos` was dropped, not excluded, since nothing showed it is the Greek word for Count; `κόμης`, a Greek surname written first in official lists, is neither added nor excluded, and adding it is a decision still to make — 2026-10-04 #606. - `knt` (Knight, beside `kt`) is a suffix acronym with a `Knt` case mask: `Mary Knt` reads suffix 'Knt' — 2026-10-04 #606. @@ -28,3 +28,4 @@ Elsewhere: `justice` and `marshal`, excluded here as common leading-position nam - **2026-10-17 #662 — owura and bapak join TITLES; malam and bay are excluded.** Akan and Indonesian Mr., found in the CLDR corpus (`Owura Bertram…`, `Bapak Dwi Putro bin Adinata MP` read the salutation as the given name). Counts from `tools/name_usage.py` on 2026-10-17: owura and bapak each under 100 US Census bearers and none found on Wikidata (a zero proves nothing, per the tool). The two exclusions are `excluded` entries in ha.toml and tr.toml, with their counts (`uv run python tools/vocab/query.py --issue 662`). Measurement: two corpus names move, both in corpus_cldr.jsonl and both toward CLDR's fields (`Owura Bertram Wilberforce Henry Robert Wooster MP` and `Bapak Dwi Putro bin Adinata MP`, salutation from given to title); recompute with `uv run python tools/differential/snapshot.py --since `. The release-log reading was checked against the 1.4.0, 2.0.0 and 2.4.2 wheels. - **2026-10-22 #840 — the Italian abbreviated salutations join TITLES with case masks; sig and ing are excluded.** #838 made case repair write each full-stop chunk of a word as a word, so an unlisted `SIG.RA` repaired to 'Sig.Ra' where every release gave 'Sig.ra', and the remedy is a listed casing. Thirteen forms whose letters after the full stop are the word's own ending: `sig.ra` and `sig.na` (salutations, spellings of `signora` and `signorina`), `dott.ssa` and `prof.ssa` (academic), and the letter-heading styles `gent.mo`, `gent.ma`, `gent.le`, `ill.mo`, `ill.ma`, `ch.mo`, `ch.ma`, `preg.mo`, `preg.ma`, each with a `cased` mask. Listing them also settles the parse, which had read them by accident: `Sig.ra` and `Prof.ssa` were titles through a piece their joined word matched, and `Dott.ssa Anna Bianchi` read given 'Dott.ssa', middle 'Anna' on 1.4.0 and 2.5.0 (wheels, measured 2026-10-22); it reads title 'Dott.ssa', given 'Anna' now, under all three orders and every comma shape. None of the thirteen spells a name, each being an Italian abbreviation, so C-i's position test finds no bearer to weigh (a word with an interior full stop can be one in general: `J.R. Smith` reads given 'J.R.', rules/H2; the nearest shadow found is joined initials spelling `ch.ma`, `CH.MA ROSSI` now reading title 'CH.MA', with no bearer found); the committed corpora hold none (the snapshot moved only rules/R4's own `SIG.RA` row), and the class rests on standard Italian usage rather than a count. The bare `sig` (US SSA 50+ male, Wikidata 11+ given) and `ing` (nam_dict band 5 in Sweden, Wikidata 10+ given, US Census 1,814 family) are excluded with their counts (tools/name_usage.py, 2026-10-22); `Sig.` and `Ing.` already read as titles by H2's shape. A cost the `ing` exclusion does not decide: ClinicalTrials.gov officials write a bare `Ing` AFTER a comma, as a credential (52 of 536,981 write it, 43 after a comma, the same tool run), and 9 of the 28 distinct names holding it put it in the given field (`Viera Sevcikova, Ing` reads given 'Ing', family 'Viera Sevcikova'), where the dotted `Mario Rossi, Ing.` reads title 'Ing.'. That is a suffix question, a credential reading of `ing` beside the comma, and a title exclusion neither causes nor settles it. A trailing `Anna Bianchi Dott.ssa` keeps 'Dott.ssa' as the family, since H5 reads a trailing title only when it ends in a full stop, as `John Smith Sir` keeps 'Sir'; after a comma (`Maria Rossi, Sig.ra`) it is the title. - **2026-10-22 #847 — `cllr` joins TITLES as `councillor` abbreviated.** British style writes it without the period, so `Cllr John Smith` read given 'Cllr', middle 'John' where `Cllr. John Smith` read title by shape. `tools/name_usage.py Cllr`: US Census surnames under 100, SSA under 5 births a year, not in nam_dict (a selection, so a miss proves little), and 0 on Wikidata as given or family (a lower bound: a zero proves nothing, per the tool), so no bearer was found as a name, and a five-letter vowelless abbreviation is not a plausible given name. It opens 508 of 15,214 names in `ukec-individuals` (cache data date 2026-10-23T03:45:09Z, same corpus sha as the first fetch of the issue), all bare; with `cllr` removed from `Lexicon.default()` the same 508 `as_dict()` readings differ and no others, 505 of them also moving the old middle into given. `Duke` and `Earl` stay out as borne names, by the caution above: they open 8 and 7 of those names (`name_usage.py` counts 8 and 15 names that write either anywhere). The accepted cost is `John Smith Cllr.`, whose period-marked trailing `Cllr.` was family and is now a title (rules/H5). +- **2026-10-30 #753, phase 3f — `pan` is a salutation off by default, for a Czech or Polish choice to turn on.** The 2026-10-04 exclusions in cs.toml and pl.toml become `salutation` readings written `default = false`, the collision with the Chinese surname Pan, which leads, kept in their notes (US Census 2010 12,664 bearers). The default does not read the word, so no parse moves; under `locales.CS` `Pan Jan Novák` reads title 'Pan' and `Pan Wei` title 'Pan', family 'Wei', the cost of choosing the language. decisions/vocabulary-model's phase 3f entry holds the measurement. diff --git a/docs/design/decisions/vocabulary-exclusions.md b/docs/design/decisions/vocabulary-exclusions.md index b73d2133..2bb618ad 100644 --- a/docs/design/decisions/vocabulary-exclusions.md +++ b/docs/design/decisions/vocabulary-exclusions.md @@ -75,7 +75,7 @@ Excluded (MAIDEN_MARKERS, the birth_marker readings): - "born" — never shipped: a release-log drafting invention, caught by the 2.0 milestone audit and corrected (5ccf9f3). Recorded so nobody "restores" it; if ever proposed for real, Max Born is the counterexample to analyze. - Scandinavian "f." — collides with the initial F.; only the full participles (født/fødd/född) are safe. Data since #751 (2026-10-20): `f` in da.toml carries `excluded = { birth_marker = … }`. Czech masculine "rozený" awaits the same vetting, on its own merits rather than following the feminine form in. - Polish "z domu" — INCLUDED 2026-08-26 as the set's first phrase entry, and the contrast to draw against the roz removal below rather than against any included word. Its words fail C-i separately and it passes as a pair; the argument is in the #434 entry under #M2. The point to carry away is that the entry below could not have been rescued the same way: roz is a whole marker as written, so there is no longer unit to move the claim onto. -- Czech/Slovak "roz." — SHIPPED through 2.1 and removed 2026-08-26 (Derek's call). Roz is an ordinary English diminutive of Rosalind, and matching is whole-token, case-folded and period-insensitive, so "Roz", "roz" and "roz." are one string to this set. The collision is not theoretical and it is not new: on the bare path, which predates the 2.2 work entirely, "Rosalind Roz Smith" read maiden "Smith" and NO family name at all, and "Rosalind Roz Jones Smith" read maiden "Jones Smith", because M2 hands the marker every word after it. Grounded in #vocabulary-collisions C-i — a word belongs in its set's ambiguous subset iff it is borne as an ordinary name IN THE POSITION THE VOCABULARY CLAIM ACTS ON, and under uncertainty default to ambiguous. Roz is borne exactly there: the word after a given name, which is the only position M2's claim acts on. What C-i cannot do for this set is the marking: MAIDEN_MARKERS has no ambiguous subset, and no code path consults one, so the only two expressions of C-i available here are ship and do not ship. That is why the remedy is removal rather than an ambiguous marking, and it is worth knowing before someone proposes marking the next collision instead. The full participle rozená stays and is unambiguous; a Czech caller who needs the abbreviation adds it to their own Lexicon, which is the same answer this section gives for every locale-specific vocabulary. The cost is accepted and stated: "Anna Nováková roz. Svobodová" now reads middle "Nováková roz.", family "Svobodová" — which is how 1.4.0 read it — and "Rosalind Roz Smith" is back to 1.4.0's given/middle/family. Pinned by tests/v2/cases.py::diminutive_that_was_a_marker_keeps_the_family and ::full_participle_marker_still_consumes. The refusal is data since #751 (2026-10-20): `roz` in cs.toml carries `excluded = { birth_marker = … }` with its reason, measured again that day (`--excluded birth_marker`). +- Czech/Slovak "roz." — SHIPPED through 2.1 and removed 2026-08-26 (Derek's call). Roz is an ordinary English diminutive of Rosalind, and matching is whole-token, case-folded and period-insensitive, so "Roz", "roz" and "roz." are one string to this set. The collision is not theoretical and it is not new: on the bare path, which predates the 2.2 work entirely, "Rosalind Roz Smith" read maiden "Smith" and NO family name at all, and "Rosalind Roz Jones Smith" read maiden "Jones Smith", because M2 hands the marker every word after it. Grounded in #vocabulary-collisions C-i — a word belongs in its set's ambiguous subset iff it is borne as an ordinary name IN THE POSITION THE VOCABULARY CLAIM ACTS ON, and under uncertainty default to ambiguous. Roz is borne exactly there: the word after a given name, which is the only position M2's claim acts on. What C-i cannot do for this set is the marking: MAIDEN_MARKERS has no ambiguous subset, and no code path consults one, so the only two expressions of C-i available here are ship and do not ship. That is why the remedy is removal rather than an ambiguous marking, and it is worth knowing before someone proposes marking the next collision instead. The full participle rozená stays and is unambiguous; a Czech caller who needs the abbreviation adds it to their own Lexicon, which is the same answer this section gives for every locale-specific vocabulary. The cost is accepted and stated: "Anna Nováková roz. Svobodová" now reads middle "Nováková roz.", family "Svobodová" — which is how 1.4.0 read it — and "Rosalind Roz Smith" is back to 1.4.0's given/middle/family. Pinned by tests/v2/cases.py::diminutive_that_was_a_marker_keeps_the_family and ::full_participle_marker_still_consumes. The refusal is data since #751 (2026-10-20): `roz` in cs.toml carries `excluded = { birth_marker = … }` with its reason, measured again that day (`--excluded birth_marker`). Since 2026-10-30 (#753, phase 3f) it is no refusal but a `birth_marker` reading written `default = false` in cs.toml, which the Czech pack turns on and the default does not read, the Roz collision its note; decisions/vocabulary-model's phase 3f entry. Excluded (CAPITALIZATION_EXCEPTIONS — meng, edd, lac, ded, left out of the 2026-09-24 masks, #459): diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 0a8e4753..252671f9 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -343,3 +343,11 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **Open: a period-joined token whose only title chunk the choice makes possibly given reads as a name word, in both directions.** `_vocab.period_joined_vocab` makes a token a title where ANY chunk is in `titles`, and under `ne` `श्री` has moved to `titles_ambiguous`. That costs `सु.श्री.` (सुश्री, Ms., written with its periods), title under the default and given under `ne`, 2 nepal-candidates names; and it fixes a given name glued behind the word, `श्री.राम थापा`, title `श्री.राम` under the default and given `श्री.राम` under `ne`. Whether a possibly-given chunk counts there is open; a fix would move the default for every word of `TITLES_AMBIGUOUS`, and its grid holds both `सु.श्री.` and `श्री.X`. - **Recompute:** the three commands above from a checkout of this change; the CLDR record with `cldr_agreement.py`; the word counts with `tools/name_usage.py guru गुरु श्री`. - **2026-10-30 #754, sub-phase 4g — THE COMPOUND GIVEN NAME IS A RULE KIND: `Policy.two_given_names`, rules/O7, OFF BY DEFAULT AND A `[rules]` KEY; NO DEFAULT PARSE MOVES.** Carries out item 7 of the 2026-10-28 #754 output-model entry and the 4g row of its item 9. The rule kind is built as the 4f entry's item 1 says a new one is: a `Policy` field with `ADDED_IN` (`3.0.0`, so a 2.8 `Policy` pickle loads with it off, tests/v2/data/policy_v28.pickle), a `flag` row in `tools/vocab/build.py`'s `RULE_KEYS` with its default `false` refused, a row in `_locale._RULE_VALUES`, and a mechanisms.md entry, DECLARED-SPAN, which it shares with rules/O6's switch, whose mirror it is. No file turns it on: whether `es` and `pt` do is 3g's. What the rule reads, its precedence against O6 and O3, and the measurements are decisions/O7's 2026-10-30 entry; the `Maria do Carmo Silva` shape the RFC's table names stays unread, the particle chain running to the end of the name, and is Open there. Item 6 of the 4f entry, deriving `DEVIATES` from the rules, stays open: this rule has no surface in the name for a pack to declare. +- **2026-10-30 #753, phase 3f — THE CZECH, SLOVAK, SLOVENIAN AND CROATIAN PACKS: `locales.CS`, `SK`, `SL` AND `HR`, ONE `Locale` PER LANGUAGE, EACH, LIKE `locales.NE`, A CHOICE OF ITS ONE TAG; `pan` AND `roz` BECOME READINGS OFF BY DEFAULT; THE DEFAULT DOES NOT MOVE.** Carries out 3f of item 11 of the 2026-10-27 #753 entry. + - **One pack per language,** as the RFC's API section already had it ("each language is a pack of its own"), and for three reasons measured here: the files hold different words (sl and hr only the generations, sk `ArtD.`, cs `pan` and `roz`), so a shared pack would turn on Czech `pan` and `roz` for a Slovenian caller; the choice is a set, so a caller with data in two of them passes both (`parser_for(locales.CS, locales.SK)`); and a pack is then nothing but its tag, `Locale(code="cs", lexicon=Lexicon.empty(), languages=frozenset({"cs"}))`, auditable as its file. Each module ships no `DEVIATES`, as phase 3d's contract has it for a pack that is nothing but a choice: its deviations are `changed_words` of its choice, and the contract test now also refuses a choice pack whose choice changes no word (decisions/deviates-registry's 2026-10-30 phase 3f entry). + - **The readings, all written `default = false`, so `_compiled.py` is unchanged.** 3c's generations `ml` and `st` in all four. In cs and sk, the academic titles written before the name as `academic` readings with their case masks (`mudr`, `mddr`, `mvdr`, `judr`, `phdr`, `rndr`, `paeddr`, `pharmdr`, `thdr`, `mgr`; cs also `mga` and `bca`) and the degrees written after it as `credential` readings (`drsc`; sk also `artd`): #848's open question 1, whether `Mgr.`, `MUDr.` and `PhDr.` follow `Ing.` to the suffix after a comma, is answered for the pack by Czech usage, title, since they stand before the name. `mgr` changes no compiled set, en and fr reading it as Monsignor already; under the choice its kind is `academic`. cs's and pl's `pan` exclusions become `salutation` readings, and cs's `roz` exclusion a `birth_marker` reading, each keeping its collision in its note. So `Jan Novák, MUDr` reads title `MUDr`, given `Jan` under `locales.CS` where the default reverses it (given `MUDr`, family `Jan Novák`), `Jan Novák MUDr.` reads the title at the end (rules/H5), `Jan Novák, DrSc` suffix `DrSc`, `Jana Nováková roz. Svobodová` maiden `Svobodová`, `Janez Novak st.` under `locales.SL` suffix `st.`, reported suffix-or-name, and `capitalized()` writes `MUDr.` for `mudr.`. Every reading was swept over the four comma shapes, the three orders, dotted, bare and one-case spellings and a family comma behind a `Freiherr` head; tests/v2/test_locales.py's `test_the_west_and_south_slavic_packs_read_what_the_default_refuses` pins a row of each, default and pack side by side. + - **Left out, with the reason:** `bc` (bakalář) is an exclusion, measured: under a Czech choice it moved 23 of ctgov-officials' 536,981 officials, one a Czech `Bc` and the rest the US nursing credential Board Certified or initials (`Lora L. Beebe, PhD, PMHNP, BC` read title `BC`; `ACNP-BC` recased `ACNP-Bc`), and `Bc.` with its period is a title at the head by its shape anyway. `ing` stays mul.toml's ambiguous credential: an `academic` reading beside its `given` reading would be a title only inside a title phrase (rules/H9), moving nothing but `Ing` in a phrase, and its dotted spelling is a title by shape. `doc` (docent) is an English nickname (US SSA 1,391 given-name births, nam_dict band 4), recorded as no reading of any file; an `academic` reading would need that `given` reading beside it, which makes it a title only inside a title phrase (rules/H9), so it would move nothing a pack wants, and the dotted `doc.` is a title by shape already. Slovak `rod.` and `rodená`, the masculine `rozený`, the Slovene and Croatian academic abbreviations (`dr. sc.`, `mag.`, `univ. dipl. inž.`) and Slovak `pán` and `pani` are unvetted, not refused. + - **3c's three open questions.** (1) **The Saint readings of `st` leave under all four, accepted.** Czech, Slovak, Slovenian and Croatian write Saint `sv.`, so a caller choosing one of them meets `St.` as Saint only in another language's names: `St Clair Smith` reads given `St`, `Jill St. John` middle `St.`, family `John`, and the 11 committed entries 3c counted. Two remedies were weighed and declined: moving en's and fr's Saint readings to mul.toml, which prefer never drops, would keep them under every choice but say they are every language's, and changes `Reading.languages` for the default's `st` from `('en', 'fr')` to `('mul',)`; and a rule that a generation reading never displaces a title is the prefer semantics item 2 of the 2026-10-27 entry rejected for no evidence. The way to keep them is to choose a language that holds them too: `parser_for(locales.CS, Locale(code="fr", …, languages=frozenset({"fr"})))` reads `St Clair Smith` title `St` and `Jiří Dienstbier st.` suffix `st.`, and fr's choice changes only `père` besides. (2) **The ungated `ml` stays ungated.** `Novak, ML` reads suffix `ML` with nothing reported, as the default reads `Smith, JR`: no name is `ml`, so there is no name reading to gate it, and initials are the clash, which a pack chooser accepts (ctgov's `Clements ML` reads suffix `ML`, reporting given-or-family). (3) **The inert one-letter `h` and `i` are re-deferred** to the packs that turn them on, es (3g) and de, which no sub-phase schedules; neither letter is in these four files. + - **Measured** 2026-10-30 on `40675146` plus this change, Python 3.11.17. Committed corpora, `snapshot.py --languages TAG`, entries moved (held only by `corpus_cldr.jsonl` among them): `cs` 17 (6), `sk` 15 (4), `sl` 15 (4), `hr` 15 (4), `pl` 11 (2). Fifteen of each are `st` and `ml` moves, 3c's fourteen and rules/S11's new `[cs]` example `Jiří Dienstbier st.`, which this change adds to the rules corpus; cs and pl add `Pan Tomasz Nowak` and `Pan Nowak Tomasz Jan` (under `FAMILY_FIRST`), both now reading title `Pan`; pl's other 9 are its conjunction `i`, which choosing pl already moved. The CLDR locale view, newly recorded for the four packs in `cldr_agreement_locales.json`: whole renderings agreeing cs 14/24, sk 15/24, sl 16/24, hr 18/24, default and pack alike; the pack moves two sl renderings and no other, each `ml.` from family to suffix (`g. Janez Peter Novak ml., univ. dipl. inž.`, family `Novak`), both still disagreeing elsewhere. Local corpora against the base's default readings (`snapshot.py --local SOURCE --base --languages TAG`), counted by the first changed word each moved name holds: `ctgov-officials` cs 178 (st 69, mudr 44, pan 17, drsc 15 and 2 `Dr. Sc.`, phdr 12, rndr 7, pharmdr 6, ml 3, and one each of paeddr, mddr, mvdr), sk 161 (the same less pan), sl and hr 72 (st 69, ml 3); `orcid-researchers` cs 245 (st 124, pan 46, phdr 16, ml 14, mudr 11, judr 8, drsc 6, mga 5, paeddr 5, rndr 4, roz 3, and one each of thdr, pharmdr, bca), sk 191, sl and hr 138 (st 124, ml 14). Read by kind: the titles and `drsc` are fixes wherever they move a field (`Denisa Wilkes, MUDr` stops reversing, `Christian Salmon, DrSc` keeps its given name, `Prof. MUDr.` reads one title run); every `pan` move is a Chinese name read with a title (`PAN DAI` → title `PAN`), no Czech salutation among them; `roz` fixes one (`Lenka Šenkárová (roz. Veverková)` → maiden `Veverková`) and costs two given names `Roz` (`ELLA ROZ PERRY` → maiden `PERRY`); `st` is 3c's (Saint names, and `PhD St.`, a PhD student, read as the generation). The fielded scores, `local_agreement.read` over the records of the four countries: ORCID's 56,808 records from CZ, SK, SI and HR agree 52,239 under the default and 52,240 under cs and sk (`dr.sc. Sara Soldo`, HR, fixed), unchanged under sl and hr; PSC's 3,578 Czech, Slovak, Slovenian and Croatian filers agree 3,505 under every pack, cs and sk fixing `Mudr Jozef Stec` and moving `Mr Judr Branislav Fridich`, whose filer typed `Judr` in the forename box. The same run over `psc-individuals`: cs 416 (st 395, pan 16, mudr 2, and one each of judr, mvdr and roz, `Miss Naila Roz Roz` → maiden `Roz`), sk 399, sl 395 (all st; hr not run, its choice changing the same two words the same way); `tse-donors` cs 9 (st 5, roz 2, pan 2, `PAN SHISHEN` → title `PAN`); `ukec-individuals` cs 9 (st 8, and `Victor MGA Killearn`, only its forced case repair moving, `MGA` → `MgA`). Recompute: the commands above, and `uv run python -c "from nameparser import _reading_index as ri; print(sorted(ri.changed_words(ri.selected(['cs']))))"` for each tag's words. + - **Found, not changed:** a gated generation behind a particle run after a family comma stays in the given part unreported, `Novák, Freiherr von Berg st.` reading given `von Berg st.` under these packs as the default reads `Dumas, Freiherr von Berg fils` (rules/S11); the default reading of the same `st.` is stranger, family `st. Novák`. Pinned in the 3f table. And with one word before a family comma, `Novák, st.` reads title `st.` by its period shape (rules/H2) under all four packs, reporting suffix-or-name with the generation as the alternative though the reading taken is a title, as the default reads `Dumas, fils.`; the ungated `Novák, ml.` reads suffix `ml.`. `Novák st., Jan` keeps family `Novák st.`, as the default keeps `Dumas fils, Alexandre`: the gated word before a family comma is a name word. Both pinned in the 3f table. + - **Open:** `pan`, kept because the work order names it: the corpora show its cost (81 Chinese names across ctgov, ORCID, PSC and TSE read with the title) and no gain, the shape that excluded `bc`, and as a title on its own it also takes a lone given name after a family comma (`Smith, Pan` reads title `Pan` with no given name, a bare `Pan` title `Pan`), so whether it stays in the Czech pack is a question for the reviewer. Whether the uncontested titles (`mudr`, `judr`, `phdr`, `rndr` and the rest, no name reading and no clash found) belong in the default rather than a pack is a default change, measured on ctgov, outside phase 3's rule that nothing moves the default. diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 6a7ac2b7..9f2ca5f6 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -376,7 +376,7 @@ print(len(MAIDEN_MARKERS), 'members ->', len(cls), 'structural classes')" ### FORCE-A-DECISION-TABLE — no silent defaults on growth -**Contract statement.** Where adding an enum member or a file must not silently inherit a default, a local table's key set is asserted equal to the population, so growth fails the suite until someone decides — against a local table, not the constant under test. Exemplar: tests/v2/pipeline/test_vocab.py's per-script initials check; reused for the pack registry (test_registry_is_the_pack_contract: every registered pack must ship DEVIATES and its rotator list or the suite fails structurally — decisions/deviates-registry). Known gap it exposes: DEFAULT_SCRIPT_ORDERS has no such guard. +**Contract statement.** Where adding an enum member or a file must not silently inherit a default, a local table's key set is asserted equal to the population, so growth fails the suite until someone decides — against a local table, not the constant under test. Exemplar: tests/v2/pipeline/test_vocab.py's per-script initials check; reused for the pack registry (test_registry_is_the_pack_contract: every registered policy or segmentation pack must ship DEVIATES, a pack that is nothing but a choice of languages must not, its surface being derived, and its choice must change a word, a choice beside a fragment or a patch declares only the non-choice part, and every pack needs its rotator list, or the suite fails structurally — decisions/deviates-registry). Known gap it exposes: DEFAULT_SCRIPT_ORDERS has no such guard. ### SELF-EXPIRING-GUARD — a decline keyed to a measured defect diff --git a/docs/design/rules/M-maiden-names.md b/docs/design/rules/M-maiden-names.md index 7aaba010..318ed68e 100644 --- a/docs/design/rules/M-maiden-names.md +++ b/docs/design/rules/M-maiden-names.md @@ -1,4 +1,4 @@ ## Maiden names (M) -Background: a maiden name is written beside the current name, set off by a marker or by enclosure. Markers are attested across French née/né and the unaccented nee English writing uses for both, German geb./geborene, Dutch geboren, Czech/Slovak rozená (the abbreviation roz. shipped through 2.1 and was removed in 2.2 -- it collides with the English diminutive Roz, and a caller who needs it adds it to their own Lexicon), Scandinavian født/fødd/född, Russian урожд. and its full participles урождённая/урождённый (the participles in both the ё and е spellings, which case normalization does not fold), Japanese 旧姓, and Polish z domu — both grammatical genders where attested. Every one of these is a marker wherever it stands, the unaccented nee included: an enclosure holding a word past it reads as the maiden name and the marker is dropped, so a configured pair reading "(Nee Jones)" gives maiden Jones and not Nee Jones. A marker need not be one word. z domu is two, and a phrase marker is recognized only whole and only where its words stand together: neither of its words is a marker standing alone, and neither is the pair once a bracketed clause or a comma divides them. That is what makes shipping it safe where shipping its words would not be — z is an ordinary Polish preposition, and a name that merely contains one keeps its family name. Japanese more often writes the marker with a fullwidth colon (旧姓：佐藤), which is no separator, so marker and name arrive as a single word. Which enclosures mean "maiden" rather than "nickname" is a caller convention, so the maiden reading of a delimiter pair is opt-in — except where the clause announces itself. A clause of two words or more led by a marker word has said which convention it means, and reads as the maiden name inside a nickname pair as well (M3) — unless its content is suffix-shaped, which S1 takes ahead of both. A lone marker has said nothing, and neither has one the colon spelling above glues to the name. A marker also says something about the name standing beside it: it announces a surname the bearer no longer uses, and that is only worth writing where there is a current one to tell it apart from (M4). +Background: a maiden name is written beside the current name, set off by a marker or by enclosure. Markers are attested across French née/né and the unaccented nee English writing uses for both, German geb./geborene, Dutch geboren, Czech/Slovak rozená (the abbreviation roz. shipped through 2.1 and was removed in 2.2 -- it collides with the English diminutive Roz -- and since 3.0 the Czech pack turns it back on, a reading off by default in cs.toml), Scandinavian født/fødd/född, Russian урожд. and its full participles урождённая/урождённый (the participles in both the ё and е spellings, which case normalization does not fold), Japanese 旧姓, and Polish z domu — both grammatical genders where attested. Every one of these is a marker wherever it stands, the unaccented nee included: an enclosure holding a word past it reads as the maiden name and the marker is dropped, so a configured pair reading "(Nee Jones)" gives maiden Jones and not Nee Jones. A marker need not be one word. z domu is two, and a phrase marker is recognized only whole and only where its words stand together: neither of its words is a marker standing alone, and neither is the pair once a bracketed clause or a comma divides them. That is what makes shipping it safe where shipping its words would not be — z is an ordinary Polish preposition, and a name that merely contains one keeps its family name. Japanese more often writes the marker with a fullwidth colon (旧姓：佐藤), which is no separator, so marker and name arrive as a single word. Which enclosures mean "maiden" rather than "nickname" is a caller convention, so the maiden reading of a delimiter pair is opt-in — except where the clause announces itself. A clause of two words or more led by a marker word has said which convention it means, and reads as the maiden name inside a nickname pair as well (M3) — unless its content is suffix-shaped, which S1 takes ahead of both. A lone marker has said nothing, and neither has one the colon spelling above glues to the name. A marker also says something about the name standing beside it: it announces a surname the bearer no longer uses, and that is only worth writing where there is a current one to tell it apart from (M4). diff --git a/docs/design/rules/S11-suffix-word-that-is-also-a-name.md b/docs/design/rules/S11-suffix-word-that-is-also-a-name.md index 73912150..2e18038f 100644 --- a/docs/design/rules/S11-suffix-word-that-is-also-a-name.md +++ b/docs/design/rules/S11-suffix-word-that-is-also-a-name.md @@ -20,6 +20,7 @@ "Pedro Alves Neto, PhD" → suffix="Neto, PhD" "Pedro Alves Neto Júnior" → suffix="Neto Júnior" "Alexandre Dumas fils" → suffix="fils" +"Jiří Dienstbier st." [cs] → suffix="st." "Alves, Pedro Neto" → suffix="Neto" "Ana Neto" → family="Neto" · boundary "Ana Neto" → ambiguities=("suffix-or-name",) · boundary diff --git a/docs/design/rules/S2-suffix-word-ends-a-name-and-never-opens-the-string.md b/docs/design/rules/S2-suffix-word-ends-a-name-and-never-opens-the-string.md index ebcc5dc8..fae9b3c8 100644 --- a/docs/design/rules/S2-suffix-word-ends-a-name-and-never-opens-the-string.md +++ b/docs/design/rules/S2-suffix-word-ends-a-name-and-never-opens-the-string.md @@ -29,6 +29,7 @@ "Alexandre Dumas fils" → suffix="fils" "Marie Fils" → family="Fils" · boundary "Jiří Dienstbier ml." → family="ml." · boundary +"Jiří Dienstbier ml." [cs] → suffix="ml." "Pavlović, Antonio mlađi" → suffix="mlađi" "John Smith PhD" → suffix="PhD" ``` diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 1848472c..00260702 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -164,7 +164,7 @@ Each sub-phase is one PR, measured on its own base: the committed snapshot read ## Phase 3: packs that prefer or restrict -Status: **agreed** (proposed 2026-10-27, #753; Derek accepted every item on 2026-10-28, six and then the other six); **3a landed 2026-10-29**, the reading index, the compiler moved into the package and `default = false` on name readings (decisions/vocabulary-model's 2026-10-29 entry, which records what 3a decided and measured, and the index's lazy load, which Derek chose on 2026-10-29 over loading it with `Lexicon.default()`; "Where the provenance lives at runtime" below describes it). **3b landed 2026-10-29**, the choice: `Locale.languages`, its tag selection, turning on, prefer and the shape clash of `may_be_initial`, the derived deviation surface, `snapshot.py --languages`, and mechanisms.md#LOCALE-PACKS-PURE-DATA amended (decisions/vocabulary-model's 2026-10-29 phase 3b entry, which records what it decided, measured and left open). **3c landed 2026-10-29**, the eight pack-shaped exclusions (cs `ml`, hr, sk and sl `st` and `ml`, fr `père`) written as `generation` readings `default = false`, the default's compile unchanged (decisions/vocabulary-model's 2026-10-29 phase 3c entry, which records what each choice now moves). **3d landed 2026-10-30**, the first language pack, `locales.NE`, a choice of `ne` turning on `given` readings of `श्री`, `guru` and `गुरु` written `default = false` in `ne.toml`, its deviations derived rather than declared, and `local_corpora.py alike --languages` for the pair measure (decisions/vocabulary-model's 2026-10-30 phase 3d entry, which records what it moves and the one defect left open). The record is decisions/vocabulary-model's 2026-10-27 #753 entry; this section describes its items. One design inside the work order is still open: 3g's connective (below), which accepting the work order did not decide. Phase 3 is 3.0 work (re-planned 2026-10-28): 2.8 is the last 2.x feature release, and 3.0 ships phases 3 and 4 with the facade's removal. Packs stay opt-in, so a caller who chooses no language parses as the default does, and nothing in this phase moves the default's parses. One standing text is amended with it, in 3b's PR, which builds the selection: mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack adds, and selects among the default's readings by language, its deviations derived for a vocabulary pack). decisions/3-0-reevaluations' line that removing base vocabulary through a pack is part of 3.0's pack design is carried out rather than moved. +Status: **agreed** (proposed 2026-10-27, #753; Derek accepted every item on 2026-10-28, six and then the other six); **3a landed 2026-10-29**, the reading index, the compiler moved into the package and `default = false` on name readings (decisions/vocabulary-model's 2026-10-29 entry, which records what 3a decided and measured, and the index's lazy load, which Derek chose on 2026-10-29 over loading it with `Lexicon.default()`; "Where the provenance lives at runtime" below describes it). **3b landed 2026-10-29**, the choice: `Locale.languages`, its tag selection, turning on, prefer and the shape clash of `may_be_initial`, the derived deviation surface, `snapshot.py --languages`, and mechanisms.md#LOCALE-PACKS-PURE-DATA amended (decisions/vocabulary-model's 2026-10-29 phase 3b entry, which records what it decided, measured and left open). **3c landed 2026-10-29**, the eight pack-shaped exclusions (cs `ml`, hr, sk and sl `st` and `ml`, fr `père`) written as `generation` readings `default = false`, the default's compile unchanged (decisions/vocabulary-model's 2026-10-29 phase 3c entry, which records what each choice now moves). **3d landed 2026-10-30**, the first language pack, `locales.NE`, a choice of `ne` turning on `given` readings of `श्री`, `guru` and `गुरु` written `default = false` in `ne.toml`, its deviations derived rather than declared, and `local_corpora.py alike --languages` for the pair measure (decisions/vocabulary-model's 2026-10-30 phase 3d entry, which records what it moves and the one defect left open). **3f landed 2026-10-30**, the Czech, Slovak, Slovenian and Croatian packs, `locales.CS`, `SK`, `SL` and `HR`, one `Locale` per language as the API section below has it, each, like `NE`, a choice of its one tag and nothing else: 3c's generations, cs and pl `pan` and cs `roz` converted from exclusions, and the Czech and Slovak academic titles and post-nominals, all readings written `default = false`, the default's compile unchanged (decisions/vocabulary-model's 2026-10-30 phase 3f entry, which records what each pack moves and settles 3c's three open questions). The record is decisions/vocabulary-model's 2026-10-27 #753 entry; this section describes its items. One design inside the work order is still open: 3g's connective (below), which accepting the work order did not decide. Phase 3 is 3.0 work (re-planned 2026-10-28): 2.8 is the last 2.x feature release, and 3.0 ships phases 3 and 4 with the facade's removal. Packs stay opt-in, so a caller who chooses no language parses as the default does, and nothing in this phase moves the default's parses. One standing text is amended with it, in 3b's PR, which builds the selection: mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack adds, and selects among the default's readings by language, its deviations derived for a vocabulary pack). decisions/3-0-reevaluations' line that removing base vocabulary through a pack is part of 3.0's pack design is carried out rather than moved. **Phase 4's shape questions came first.** The output model decides what a pack can express: a patronymic field, a second surname, credentials apart from generation suffixes (#326), and what `middle` means (#704). A pack reading of a kind with no field to land in is a reading nothing can surface, so those questions were to be settled before or with phase 3's mechanism, by the same shape-before-value rule, and the output model was agreed on 2026-10-28 (decisions/vocabulary-model's 2026-10-28 #754 entry): the seven roles stay, and `generation`, `credentials`, `patronymic` and `surname2` are views read from `Token.reading` or rules/O6's tag. Of the work order below, 3a (the reading index and the shared compiler) and 3c (the pack-shaped exclusions converted) are independent of the output model: the index carries whatever kinds the files hold, and 3c moves data between two encodings that compile alike. 3b's selection semantics are independent too, and its API's dependency, what `Locale` and `Policy` look like once the facade is gone, is answered (below). The packs depend on the output model in part, each on the phase 4 sub-phase that builds its view: `cs`'s generations and `pt-BR`'s agnomens on 4b's `generation` (3f and 3h), `es` and `pt` on 4d's `surname2` and 4g's compound given name (3g); `ne`'s readings do not. diff --git a/docs/locales.rst b/docs/locales.rst index 260c17d0..83296554 100644 --- a/docs/locales.rst +++ b/docs/locales.rst @@ -233,6 +233,84 @@ credential: >>> sorted(parser_for(en, base=Parser(lexicon=mine)).lexicon.particles) ['van'] +.. _west-south-slavic-packs: + +Czech, Slovak, Slovenian and Croatian names +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +These four languages tell a father from a son of the same name with a +word written after the name: ``ml.`` for the younger and ``st.`` for +the elder (Czech *mladší* and *starší*, Croatian *mlađi* and +*stariji*). Czech and Slovak also write academic degrees as titles +before the name, run together in their own capitalization (``MUDr.``, +``JUDr.``, ``PhDr.``, ``Mgr.``), and a few degrees after it +(``DrSc.``, Slovak ``ArtD.``). The default cannot read most of these +words: ``ml.`` is also a pair of initials, ``st.`` is Saint in +English and French, and ``Pan`` and ``Roz`` are names in other +languages. So they wait for a pack, and each of the +four languages is a pack of its own, which chooses that one language +and nothing else: + +.. list-table:: + :header-rows: 1 + + * - Pack + - Turns on + * - ``locales.CS`` + - ``ml.``, ``st.``; the titles ``MUDr.``, ``MDDr.``, ``MVDr.``, + ``JUDr.``, ``PhDr.``, ``RNDr.``, ``PaedDr.``, ``PharmDr.``, + ``ThDr.``, ``Mgr.``, ``MgA.``, ``BcA.``; ``DrSc.`` + after the name; ``Pan`` (Mr.); ``roz.``, which routes a maiden + name as ``rozená`` does + * - ``locales.SK`` + - ``ml.``, ``st.``; the same titles but ``MgA.`` and ``BcA.``; + ``ArtD.`` and ``DrSc.`` after the name + * - ``locales.SL``, ``locales.HR`` + - ``ml.``, ``st.`` + +The dotted titles at the head of a name, and after a comma, are titles +without a pack too, by their shape, and ``Mgr.`` is a title everywhere +already, the default listing it as Monsignor. For the titles the +default does not list (``MUDr.``, ``JUDr.``, ``PhDr.`` and the rest), +what a pack changes is the bare spelling, anywhere, and the dotted one +at the end of the name, where the default reads each as a name word. + +.. doctest:: + + >>> czech = parser_for(locales.CS) + >>> parse("Jiří Dienstbier ml.").family + 'ml.' + >>> czech.parse("Jiří Dienstbier ml.").suffix + 'ml.' + >>> parse("Jan Novák, MUDr").family + 'Jan Novák' + >>> czech.parse("Jan Novák, MUDr").title + 'MUDr' + >>> czech.parse("Jana Nováková roz. Svobodová").maiden + 'Svobodová' + +A word read through a pack names no language file in its +:attr:`~nameparser.Reading.languages`, which lists only the files whose +reading is on by default, whatever the parser chose: ``ml.`` above names +none, and ``Mgr.`` names ``en`` and ``fr``, whose Monsignor reading the +Czech choice dropped for its own. + +Choosing the language costs the other readings of these words. ``Pan +Wei`` reads title ``Pan``, ``Rosalind Roz Smith`` takes ``Smith`` for +a maiden name, and ``st`` loses its Saint readings, so ``St Clair +Smith`` reads given ``St``. A caller whose names also hold the English +Saint chooses French too, whose only other word the choice changes is +``père``: + +.. doctest:: + + >>> czech.parse("St Clair Smith").given + 'St' + >>> french = Locale(code="fr", lexicon=Lexicon.empty(), + ... languages=frozenset({"fr"})) + >>> parser_for(locales.CS, french).parse("St Clair Smith").title + 'St' + Finding packs by code ~~~~~~~~~~~~~~~~~~~~~ @@ -243,7 +321,7 @@ look one up dynamically by its lowercase code with .. doctest:: >>> locales.available() - ('ja', 'ne', 'ru', 'tr_az', 'zh') + ('cs', 'hr', 'ja', 'ne', 'ru', 'sk', 'sl', 'tr_az', 'zh') >>> locales.get("ru") is locales.RU True @@ -260,6 +338,12 @@ Shipped packs * - Code - Turns on + * - ``cs``, ``sk``, ``sl``, ``hr`` + - Czech, Slovak, Slovenian and Croatian vocabulary — a choice of + that one language, which turns on the generations ``ml.`` and + ``st.``, and in Czech and Slovak the academic titles written + before the name and the degrees written after it (see + :ref:`west-south-slavic-packs`). * - ``ja`` - Japanese segmentation — activates division for unspaced Japanese names, which needs a segmenter to act: install @@ -301,10 +385,12 @@ Shipped packs Japanese name read with it can split after its first character (``安藤一郎`` → family ``安``), so use ``ja`` for Japanese data. -``ja``, ``ru`` and ``tr_az`` are policy-only — they carry no vocabulary -of their own. ``ne`` is a choice of languages and nothing else: the -readings it turns on are in the default vocabulary's ``ne.toml``, off -until a caller chooses Nepali. ``zh`` is both halves at once: a surname list, plus the +``cs``, ``hr``, ``ne``, ``sk`` and ``sl`` are each a choice of languages +and nothing else: the readings each turns on are the default +vocabulary's own, written in its language's file (``ne.toml`` for +Nepali) and off until a caller chooses the language. ``ja``, ``ru`` and +``tr_az`` are policy-only — they carry no vocabulary of their own. +``zh`` is both halves at once: a surname list, plus the one policy field that turns segmentation on for the script it covers. See :ref:`config-containers` for how that split (language vocabulary vs. behavior) is drawn, and `Contributing a pack to nameparser`_ for which half a @@ -486,9 +572,9 @@ by ``tests/v2/test_locales.py``: ``ru`` and ``tr_az`` do. ``nameparser/vocab/README.md``'s "Rules" section lists the keys. A rule the table does not list is a change to the parser, not to a pack. -#. Declare a module-level ``DEVIATES(name)`` predicate: given a name - string, return whether *this pack alone* might parse it differently - from the default parser. Over-declaring is safe; under-declaring is +#. Declare a module-level ``DEVIATES(name)`` predicate. Given a name + string, ``DEVIATES`` returns whether *this pack alone* might parse + it differently from the default parser. Over-declaring is safe; under-declaring is not — when in doubt, ``DEVIATES`` should say yes. A pack whose scope is a *script* builds the predicate with ``_script_matcher`` from ``nameparser/_policy.py``, the way ``zh`` and ``ja`` do — @@ -496,9 +582,10 @@ by ``tests/v2/test_locales.py``: docstring explains how the pack-contract test enforces this. A pack's choice of languages is never declared: its deviations are derived from the words the choice changes, so a pack that is nothing - but a choice (``ne``) declares no ``DEVIATES``, and one with a - fragment or a patch beside its choice declares only what those - change. + but a choice (``ne``, ``cs``) declares no ``DEVIATES`` — the + contract test refuses one, and refuses a choice that changes no + word — and one with a fragment or a patch beside its choice + declares only what those change. #. Add a rotator list to ``tests/v2/test_locales.py``. Every pack needs one, but what it has to contain follows from how the pack declares its scope: @@ -512,9 +599,11 @@ by ``tests/v2/test_locales.py``: the same weight by hand — the unspaced names the pack must split, one per shape of the vocabulary it ships: single surname, compound surname, and any spelling variant it means to cover. - - **By a choice of languages** (``ne``): a name for each word the - choice changes, in each script it is written in and each position - where the default reads it otherwise. + - **By a choice of languages** (``ne``, ``cs``, ``sk``, ``sl``, + ``hr``): a name for each word the choice changes, in each script it + is written in and each position where the default reads it + otherwise — ``test_choice_rotators_cover_every_changed_word`` fails + until every word is held. - **With no vocabulary at all**: list the shapes its *segmenter* must divide instead, and mark the rotator tests to skip when the optional dependency is absent, so the contract tests still run diff --git a/docs/release_log.rst b/docs/release_log.rst index 7fa0a3e3..83da3c0e 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -14,6 +14,7 @@ Release Log - **Add a switch for names with a compound given name.** ``Parser(policy=Policy(two_given_names=True)).parse("Juan Carlos Pérez")`` gives given ``Juan Carlos``, where 1.4.0 through 2.8.0 gave given ``Juan``, middle ``Carlos``. Off by default: under it a middle name beside the given name joins it (``John Quincy Adams`` gives given ``John Quincy``). With ``two_family_names`` on too, the family takes its word first, so ``María José García López`` reads given ``María José``, family ``García López``. A ``Policy`` pickled by 2.8 loads with it off. :issue:`704` - **Add Token.reading, the reading the parse took for each word.** ``parse("John Smith PhD Jones").tokens[-1].reading`` is ``Reading('credential' as 'suffix', 'position')``, the reading 2.7's ``Ambiguity.alternative`` gave only for the other side of a fork, and ``ReadingKind`` gains ``patronymic``. The field is not compared, and a ``ParsedName`` pickled by 2.x loads with no readings. :issue:`754` - **Add Reading.languages, the language files that list a word the way it was read.** ``parse("Dr. Juan de la Vega III").tokens[2].reading.languages`` is ``('es', 'fr', 'it', 'nl', 'pt')`` for the particle ``de``, and ``Ambiguity.alternative`` names its files too; a reading by shape or position names none. It is not compared, and a ``Reading`` pickled by 2.7 or 2.8 loads with none. The first parse in a process that meets a listed word loads the reading index once: about 1 ms, or about 16 ms where Python writes no bytecode cache. :issue:`754`, :issue:`753` + - **Add Czech, Slovak, Slovenian and Croatian packs, locales.CS, SK, SL and HR.** ``parser_for(locales.CS).parse("Jan Novák, MUDr")`` gives title ``MUDr``, given ``Jan``, family ``Novák``, where 1.4.0 through 2.8.0 gave given ``MUDr``, family ``Jan Novák``. Each pack chooses its one language and turns on words the default does not read: ``ml.`` and ``st.`` after the name in all four (``Jiří Dienstbier ml.``), the Czech and Slovak academic titles and ``DrSc.``, and Czech ``Pan`` and ``roz.``. Choosing the language drops ``st``'s Saint readings (``St Clair Smith`` reads given ``St``), which choosing ``fr`` as well keeps. :issue:`753` **Breaking Changes** diff --git a/nameparser/config/_index.py b/nameparser/config/_index.py index 2e4db339..f4d06a15 100644 --- a/nameparser/config/_index.py +++ b/nameparser/config/_index.py @@ -126,6 +126,7 @@ ('army', (('en', 'modifier', ()),)), ('arranger', (('en', 'occupation', ()),)), ('arrc', (('en', 'credential', ()),)), + ('artd', (('sk', 'credential', (('cased', 'ArtD'), ('default', False))),)), ('as-', (('ar', 'article', ()),)), ('asa', (('en', 'credential', ()), ('en', 'given', ()), ('en', 'family', ()))), ('asc', (('en', 'credential', ()),)), @@ -177,7 +178,7 @@ ('bat', (('he', 'patronymic', ()), ('he', 'given', ()))), ('batoni', (('ka', 'salutation', (('after_given', True),)),)), ('bay', (('tr', 'family', ()),)), - ('bca', (('en', 'credential', ()),)), + ('bca', (('cs', 'academic', (('cased', 'BcA'), ('default', False))), ('en', 'credential', ()))), ('bcl', (('en', 'credential', ()),)), ('bcss', (('en', 'credential', ()),)), ('bds', (('en', 'credential', ()),)), @@ -611,6 +612,7 @@ ('drb', (('en', 'credential', ()),)), ('drmp', (('en', 'credential', ()),)), ('drph', (('en', 'credential', (('cased', 'DrPH'),)),)), + ('drsc', (('cs', 'credential', (('cased', 'DrSc'), ('default', False))), ('sk', 'credential', (('cased', 'DrSc'), ('default', False))))), ('druid', (('en', 'religious', ()),)), ('drummer', (('en', 'occupation', ()),)), ('dsc', (('en', 'credential', (('cased', 'DSc'),)),)), @@ -896,6 +898,7 @@ ('jr', (('en', 'generation', ()),)), ('judge', (('en', 'office', ()),)), ('judicial', (('en', 'modifier', ()),)), + ('judr', (('cs', 'academic', (('cased', 'JUDr'), ('default', False))), ('sk', 'academic', (('cased', 'JUDr'), ('default', False))))), ('juffrouw', (('nl', 'salutation', (('after_given', True),)),)), ('junior', (('en', 'generation', ()),)), ('jurist', (('en', 'occupation', ()),)), @@ -1035,6 +1038,7 @@ ('mda', (('de', 'credential', ()),)), ('mdb', (('de', 'credential', ()),)), ('mdbb', (('de', 'credential', ()),)), + ('mddr', (('cs', 'academic', (('cased', 'MDDr'), ('default', False))), ('sk', 'academic', (('cased', 'MDDr'), ('default', False))))), ('mdep', (('de', 'credential', ()),)), ('mdhb', (('de', 'credential', ()),)), ('mdiv', (('en', 'credential', (('cased', 'MDiv'),)),)), @@ -1054,7 +1058,8 @@ ('mevrouwe', (('nl', 'salutation', (('after_given', True),)),)), ('mfa', (('en', 'credential', ()),)), ('mg', (('en', 'military', ()),)), - ('mgr', (('en', 'religious', ()), ('fr', 'religious', ()))), + ('mga', (('cs', 'academic', (('cased', 'MgA'), ('default', False))),)), + ('mgr', (('cs', 'academic', (('default', False),)), ('en', 'religious', ()), ('fr', 'religious', ()), ('sk', 'academic', (('default', False),)))), ('mgysgt', (('en', 'military', ()),)), ('mhsc', (('en', 'credential', (('cased', 'MHSc'),)),)), ('micp', (('en', 'credential', ()),)), @@ -1114,12 +1119,14 @@ ('mt', (('en', 'credential', ()),)), ('mt-bc', (('en', 'credential', ()),)), ('mts', (('en', 'credential', ()),)), + ('mudr', (('cs', 'academic', (('cased', 'MUDr'), ('default', False))), ('sk', 'academic', (('cased', 'MUDr'), ('default', False))))), ('mufti', (('ar', 'religious', ()),)), ('mullah', (('fa', 'religious', ()),)), ('municipal', (('en', 'modifier', ()),)), ('murshid', (('ar', 'religious', ()),)), ('musician', (('en', 'occupation', ()),)), ('musicologist', (('en', 'occupation', ()),)), + ('mvdr', (('cs', 'academic', (('cased', 'MVDr'), ('default', False))), ('sk', 'academic', (('cased', 'MVDr'), ('default', False))))), ('mvo', (('en', 'credential', ()),)), ('mx', (('en', 'salutation', (('after_given', True),)),)), ('mystery', (('en', 'modifier', ()),)), @@ -1185,8 +1192,10 @@ ('owura', (('ak', 'salutation', (('after_given', True),)),)), ('pa', (('en', 'credential', ()), ('en', 'given', ()), ('en', 'family', ()))), ('pa-c', (('en', 'credential', ()),)), + ('paeddr', (('cs', 'academic', (('cased', 'PaedDr'), ('default', False))), ('sk', 'academic', (('cased', 'PaedDr'), ('default', False))))), ('painter', (('en', 'occupation', ()),)), ('paleontologist', (('en', 'occupation', ()),)), + ('pan', (('cs', 'salutation', (('default', False),)), ('pl', 'salutation', (('default', False),)))), ('paní', (('cs', 'salutation', ()), ('cs', 'family', ()))), ('pastor', (('en', 'religious', ()),)), ('patriarch', (('en', 'religious', ()),)), @@ -1203,8 +1212,10 @@ ('pgmp', (('en', 'credential', ()),)), ('pharaoh', (('en', 'nobility', ()),)), ('pharmd', (('en', 'credential', (('cased', 'PharmD'),)),)), + ('pharmdr', (('cs', 'academic', (('cased', 'PharmDr'), ('default', False))), ('sk', 'academic', (('cased', 'PharmDr'), ('default', False))))), ('phc', (('en', 'credential', (('cased', 'PhC'),)),)), ('phd', (('en', 'credential', (('cased', 'PhD'),)),)), + ('phdr', (('cs', 'academic', (('cased', 'PhDr'), ('default', False))), ('sk', 'academic', (('cased', 'PhDr'), ('default', False))))), ('philantropist', (('en', 'occupation', ()),)), ('philosopher', (('en', 'occupation', ()),)), ('photographer', (('en', 'occupation', ()),)), @@ -1331,10 +1342,12 @@ ('rls', (('en', 'credential', ()),)), ('rmsks', (('en', 'credential', ()),)), ('rn', (('en', 'credential', ()),)), + ('rndr', (('cs', 'academic', (('cased', 'RNDr'), ('default', False))), ('sk', 'academic', (('cased', 'RNDr'), ('default', False))))), ('robt', (('en', 'given', (('same_as', 'robert'),)),)), ('rock', (('en', 'modifier', ()),)), ('rouva', (('fi', 'salutation', (('after_given', True),)),)), ('royal', (('en', 'modifier', (('addresses', 'given'),)), ('en', 'given', ()), ('en', 'family', ()))), + ('roz', (('cs', 'birth_marker', (('default', False),)),)), ('rozená', (('cs', 'birth_marker', ()),)), ('rp', (('en', 'credential', ()),)), ('rpa', (('en', 'credential', ()),)), @@ -1483,6 +1496,7 @@ ('tenor', (('en', 'occupation', ()),)), ('ter', (('nl', 'preposition', ()),)), ('thd', (('en', 'credential', (('cased', 'ThD'),)),)), + ('thdr', (('cs', 'academic', (('cased', 'ThDr'), ('default', False))), ('sk', 'academic', (('cased', 'ThDr'), ('default', False))))), ('the', (('en', 'conjunction', ()),)), ('theater', (('en', 'modifier', ()),)), ('theatre', (('en', 'modifier', ()),)), diff --git a/nameparser/locales/__init__.py b/nameparser/locales/__init__.py index 40d20e51..4647a0ad 100644 --- a/nameparser/locales/__init__.py +++ b/nameparser/locales/__init__.py @@ -23,9 +23,13 @@ #: attribute name -> (module name, module attribute). Codes are the #: lowercase module names; attribute constants are uppercase. _REGISTRY = { + "CS": ("nameparser.locales.cs", "CS"), + "HR": ("nameparser.locales.hr", "HR"), "JA": ("nameparser.locales.ja", "JA"), "NE": ("nameparser.locales.ne", "NE"), "RU": ("nameparser.locales.ru", "RU"), + "SK": ("nameparser.locales.sk", "SK"), + "SL": ("nameparser.locales.sl", "SL"), "TR_AZ": ("nameparser.locales.tr_az", "TR_AZ"), "ZH": ("nameparser.locales.zh", "ZH"), } diff --git a/nameparser/locales/cs.py b/nameparser/locales/cs.py new file mode 100644 index 00000000..ed50ca0b --- /dev/null +++ b/nameparser/locales/cs.py @@ -0,0 +1,30 @@ +"""The Czech locale pack: a choice of languages, ``cs``, and nothing else +-- no lexicon fragment and no policy patch (3.0, phase 3f, #753; +mechanisms.md#LOCALE-PACKS-PURE-DATA). Choosing it turns on +nameparser/vocab/cs.toml's readings written ``default = false``: the +generations ``ml.`` (mladší) and ``st.`` (starší), the salutation +``pan``, the birth marker ``roz.``, the academic titles written before +the name (``MUDr.``, ``JUDr.``, ``PhDr.``, ``RNDr.``, ``Mgr.`` and the +rest) and the post-nominal ``DrSc.``; on a word cs.toml holds a +vocabulary reading of, other languages' vocabulary readings leave +(`st`'s Saint title and particle, so ``St Clair`` is a given name under +this pack; a caller whose names also hold the English Saint chooses +``fr`` too, whose only other word the choice changes is ``père``, as +decisions/vocabulary-model's 2026-10-30 phase 3f entry records). + +Data sources: the readings' own entries in nameparser/vocab/cs.toml, +each with its issue and the vetting on its note; nothing is listed +here. + +Deviations: derived, not declared. The words a choice of ``cs`` +can move are ``_reading_index.changed_words``, which the +non-interference gate in tests/v2/test_locales.py reads +(decisions/deviates-registry), so this module carries no DEVIATES. +""" +from __future__ import annotations + +from nameparser._lexicon import Lexicon +from nameparser._locale import Locale + +CS = Locale(code="cs", lexicon=Lexicon.empty(), + languages=frozenset({"cs"})) diff --git a/nameparser/locales/hr.py b/nameparser/locales/hr.py new file mode 100644 index 00000000..7ff138da --- /dev/null +++ b/nameparser/locales/hr.py @@ -0,0 +1,28 @@ +"""The Croatian locale pack: a choice of languages, ``hr``, and nothing +else -- no lexicon fragment and no policy patch (3.0, phase 3f, #753; +mechanisms.md#LOCALE-PACKS-PURE-DATA). Choosing it turns on +nameparser/vocab/hr.toml's readings written ``default = false``: the +generations ``ml.`` (mlađi) and ``st.`` (stariji); on a word hr.toml +holds a vocabulary reading of, other languages' vocabulary readings +leave (`st`'s Saint title and particle, so ``St Clair`` is a given name +under this pack; a caller whose names also hold the English Saint +chooses ``fr`` too, whose only other word the choice changes is +``père``, as decisions/vocabulary-model's 2026-10-30 phase 3f entry +records). + +Data sources: the readings' own entries in nameparser/vocab/hr.toml, +each with its issue and the vetting on its note; nothing is listed +here. + +Deviations: derived, not declared. The words a choice of ``hr`` +can move are ``_reading_index.changed_words``, which the +non-interference gate in tests/v2/test_locales.py reads +(decisions/deviates-registry), so this module carries no DEVIATES. +""" +from __future__ import annotations + +from nameparser._lexicon import Lexicon +from nameparser._locale import Locale + +HR = Locale(code="hr", lexicon=Lexicon.empty(), + languages=frozenset({"hr"})) diff --git a/nameparser/locales/sk.py b/nameparser/locales/sk.py new file mode 100644 index 00000000..f929911f --- /dev/null +++ b/nameparser/locales/sk.py @@ -0,0 +1,29 @@ +"""The Slovak locale pack: a choice of languages, ``sk``, and nothing else +-- no lexicon fragment and no policy patch (3.0, phase 3f, #753; +mechanisms.md#LOCALE-PACKS-PURE-DATA). Choosing it turns on +nameparser/vocab/sk.toml's readings written ``default = false``: the +generations ``ml.`` and ``st.``, the academic titles written before the +name (``MUDr.``, ``JUDr.``, ``PhDr.``, ``RNDr.``, ``Mgr.`` and the rest) +and the post-nominals ``ArtD.`` and ``DrSc.``; on a word sk.toml holds a +vocabulary reading of, other languages' vocabulary readings leave +(`st`'s Saint title and particle, so ``St Clair`` is a given name under +this pack; a caller whose names also hold the English Saint chooses +``fr`` too, whose only other word the choice changes is ``père``, as +decisions/vocabulary-model's 2026-10-30 phase 3f entry records). + +Data sources: the readings' own entries in nameparser/vocab/sk.toml, +each with its issue and the vetting on its note; nothing is listed +here. + +Deviations: derived, not declared. The words a choice of ``sk`` +can move are ``_reading_index.changed_words``, which the +non-interference gate in tests/v2/test_locales.py reads +(decisions/deviates-registry), so this module carries no DEVIATES. +""" +from __future__ import annotations + +from nameparser._lexicon import Lexicon +from nameparser._locale import Locale + +SK = Locale(code="sk", lexicon=Lexicon.empty(), + languages=frozenset({"sk"})) diff --git a/nameparser/locales/sl.py b/nameparser/locales/sl.py new file mode 100644 index 00000000..9885f38d --- /dev/null +++ b/nameparser/locales/sl.py @@ -0,0 +1,28 @@ +"""The Slovenian locale pack: a choice of languages, ``sl``, and nothing +else -- no lexicon fragment and no policy patch (3.0, phase 3f, #753; +mechanisms.md#LOCALE-PACKS-PURE-DATA). Choosing it turns on +nameparser/vocab/sl.toml's readings written ``default = false``: the +generations ``ml.`` (mlajši) and ``st.`` (starejši); on a word sl.toml +holds a vocabulary reading of, other languages' vocabulary readings +leave (`st`'s Saint title and particle, so ``St Clair`` is a given name +under this pack; a caller whose names also hold the English Saint +chooses ``fr`` too, whose only other word the choice changes is +``père``, as decisions/vocabulary-model's 2026-10-30 phase 3f entry +records). + +Data sources: the readings' own entries in nameparser/vocab/sl.toml, +each with its issue and the vetting on its note; nothing is listed +here. + +Deviations: derived, not declared. The words a choice of ``sl`` +can move are ``_reading_index.changed_words``, which the +non-interference gate in tests/v2/test_locales.py reads +(decisions/deviates-registry), so this module carries no DEVIATES. +""" +from __future__ import annotations + +from nameparser._lexicon import Lexicon +from nameparser._locale import Locale + +SL = Locale(code="sl", lexicon=Lexicon.empty(), + languages=frozenset({"sl"})) diff --git a/nameparser/vocab/cs.toml b/nameparser/vocab/cs.toml index c40f11e1..91021433 100644 --- a/nameparser/vocab/cs.toml +++ b/nameparser/vocab/cs.toml @@ -2,7 +2,7 @@ schema = 1 [[word]] text = "pan" -excluded = { salutation = { reason = "Czech and Polish Mr., and the Chinese surname Pan, which leads; the accented paní ships", issue = 606, decided = 2026-10-04 } } +salutation = { default = false, issue = [606, 753], gender = "m", note = "Czech and Polish Mr., off by default because Pan is the Chinese surname 潘, which leads (US Census 2010 12,664 bearers; 285 of ClinicalTrials.gov's 536,981 officials write the word and 488 of ORCID's 1,831,533 records, data 2026-10-21 and 2025-10-17, nearly all of them the surname): under a Czech or Polish choice Pan Wei reads title 'Pan'. The accented paní ships" } [[word]] text = "paní" @@ -17,6 +17,58 @@ excluded = { salutation = { reason = "Czech 'family', and a Russian surname, oft text = "slečna" salutation = { issue = 606, gender = "f", after_given = true } +[[word]] +text = "bc" +excluded = { academic = { reason = "bakalář, the Czech and Slovak Bachelor, but BC is the US nursing credential Board Certified, written after the name (APRN, BC; ACNP-BC): read as a title under a Czech choice it moved 23 of ClinicalTrials.gov's 536,981 officials, one a Czech Bc (Dagmar Koveslygetyova, Bc) and the rest that credential or initials (Lora L. Beebe, PhD, PMHNP, BC read title 'BC'; ACNP-BC recased 'ACNP-Bc'), measured 2026-10-30; written Bc. at the head of a name it reads as a title by its shape without a reading (rules/H2)", issue = 753, decided = 2026-10-30 } } + +[[word]] +text = "bca" +academic = { default = false, cased = "BcA", issue = [753, 848], expands = "bakalář umění, Bachelor of Arts" } + +[[word]] +text = "judr" +academic = { default = false, cased = "JUDr", issue = [753, 848], expands = "doktor práv, Juris utriusque doctor" } + +[[word]] +text = "mddr" +academic = { default = false, cased = "MDDr", issue = [753, 848], expands = "doktor zubního lékařství, the dentistry doctorate" } + +[[word]] +text = "mga" +academic = { default = false, cased = "MgA", issue = [753, 848], expands = "magistr umění, Master of Arts" } + +[[word]] +text = "mgr" +academic = { default = false, issue = [753, 848], expands = "magistr, Master", note = "en and fr read mgr as Monsignor (religious), on by default, so the Czech choice changes only the kind its title reading names; 44 of ClinicalTrials.gov's 536,981 officials write it, all after a comma (data 2026-10-21)" } + +[[word]] +text = "mudr" +academic = { default = false, cased = "MUDr", issue = [753, 848], expands = "doktor medicíny, Medicinae universae doctor", note = "45 of ClinicalTrials.gov's 536,981 officials write it, 42 after a comma, 3 bare (data 2026-10-21)" } + +[[word]] +text = "mvdr" +academic = { default = false, cased = "MVDr", issue = [753, 848], expands = "doktor veterinární medicíny, the veterinary doctorate" } + +[[word]] +text = "paeddr" +academic = { default = false, cased = "PaedDr", issue = [753, 848], expands = "doktor pedagogiky, Paedagogiae doctor" } + +[[word]] +text = "pharmdr" +academic = { default = false, cased = "PharmDr", issue = [753, 848], expands = "doktor farmacie, the pharmacy doctorate" } + +[[word]] +text = "phdr" +academic = { default = false, cased = "PhDr", issue = [753, 848], expands = "doktor filozofie, Philosophiae doctor" } + +[[word]] +text = "rndr" +academic = { default = false, cased = "RNDr", issue = [753, 848], expands = "doktor přírodních věd, Rerum naturalium doctor" } + +[[word]] +text = "thdr" +academic = { default = false, cased = "ThDr", issue = [753, 848], expands = "doktor teologie, Theologiae doctor" } + [[word]] text = "ml" generation = { default = false, expands = "mladší, the Younger: Jiří Dienstbier ml.", issue = [661, 679, 684, 753], note = "real and standard, but ML is a pair of initials in other languages written in Latin script, and nothing in its writing tells the two apart, so no default reads it; a Czech pack turns it on" } @@ -33,15 +85,21 @@ generation = { default = false, expands = "starší, the Elder: Jiří Dienstbie text = "starší" generation = { issue = 661 } +[[word]] +text = "drsc" +credential = { default = false, cased = "DrSc", issue = [753, 848], expands = "doktor věd, Doctor scientiarum, written after the name", note = "16 of ClinicalTrials.gov's 536,981 officials write it, all after a comma (data 2026-10-21)" } + [[word]] text = "roz" -[word.excluded.birth_marker] -reason = "Roz, the English diminutive of Rosalind, stands exactly where a marker acts: with it listed, 'Rosalind Roz Smith' reads maiden 'Smith' and takes the given name Rosalind for the family (measured 2026-10-20), a marker taking every word after it; the full rozená stays" -decided = 2026-08-26 +[word.birth_marker] +default = false +issue = [274, 753] +expands = "rozená or rozený, née or né, abbreviated" +note = "off by default: Roz, the English diminutive of Rosalind (US SSA 64 female births; US Census 2010 154 surnames), stands exactly where a marker acts, so with it read 'Rosalind Roz Smith' reads maiden 'Smith' and takes the given name Rosalind for the family (measured 2026-10-20); the Czech choice accepts that, and the full rozená ships" [[word]] text = "rozená" [word.birth_marker] issue = 274 gender = "f" -note = "the masculine rozený awaits its own vetting; the abbreviation roz., shared by both, is refused beside it" +note = "the masculine rozený awaits its own vetting; the abbreviation roz., shared by both, is a reading off by default beside it" diff --git a/nameparser/vocab/pl.toml b/nameparser/vocab/pl.toml index eb4d903d..9b9e0ae4 100644 --- a/nameparser/vocab/pl.toml +++ b/nameparser/vocab/pl.toml @@ -2,7 +2,7 @@ schema = 1 [[word]] text = "pan" -excluded = { salutation = { reason = "Polish and Czech Mr., and the Chinese surname Pan, which leads", issue = 606, decided = 2026-10-04 } } +salutation = { default = false, issue = [606, 753], gender = "m", note = "Polish and Czech Mr., off by default because Pan is the Chinese surname 潘, which leads (US Census 2010 12,664 bearers); a Polish or Czech choice turns it on" } [[word]] text = "hrabia" diff --git a/nameparser/vocab/sk.toml b/nameparser/vocab/sk.toml index 0c4dbf4d..a766b891 100644 --- a/nameparser/vocab/sk.toml +++ b/nameparser/vocab/sk.toml @@ -1,5 +1,45 @@ schema = 1 +[[word]] +text = "judr" +academic = { default = false, cased = "JUDr", issue = [753, 848], expands = "doktor práv, Juris utriusque doctor" } + +[[word]] +text = "mddr" +academic = { default = false, cased = "MDDr", issue = [753, 848], expands = "doktor zubného lekárstva, the dentistry doctorate" } + +[[word]] +text = "mgr" +academic = { default = false, issue = [753, 848], expands = "magister, Master", note = "en and fr read mgr as Monsignor (religious), on by default, so the Slovak choice changes only the kind its title reading names" } + +[[word]] +text = "mudr" +academic = { default = false, cased = "MUDr", issue = [753, 848], expands = "doktor medicíny, Medicinae universae doctor" } + +[[word]] +text = "mvdr" +academic = { default = false, cased = "MVDr", issue = [753, 848], expands = "doktor veterinárnej medicíny, the veterinary doctorate" } + +[[word]] +text = "paeddr" +academic = { default = false, cased = "PaedDr", issue = [753, 848], expands = "doktor pedagogiky, Paedagogiae doctor" } + +[[word]] +text = "pharmdr" +academic = { default = false, cased = "PharmDr", issue = [753, 848], expands = "doktor farmácie, the pharmacy doctorate" } + +[[word]] +text = "phdr" +academic = { default = false, cased = "PhDr", issue = [753, 848], expands = "doktor filozofie, Philosophiae doctor" } + +[[word]] +text = "rndr" +academic = { default = false, cased = "RNDr", issue = [753, 848], expands = "doktor prírodných vied, Rerum naturalium doctor" } + +[[word]] +text = "thdr" +academic = { default = false, cased = "ThDr", issue = [753, 848], expands = "doktor teológie, Theologiae doctor" } + [[word]] text = "ml" generation = { default = false, expands = "mladší, the Younger", issue = [661, 679, 684, 753], note = "real and standard, but ML is a pair of initials in other languages written in Latin script, and nothing in its writing tells the two apart, so no default reads it; a Slovak pack turns it on" } @@ -23,3 +63,11 @@ generation = { issue = 679 } [[word]] text = "starší" generation = { issue = 661 } + +[[word]] +text = "artd" +credential = { default = false, cased = "ArtD", issue = [753, 848], expands = "doktor umenia, artis doctor, written after the name" } + +[[word]] +text = "drsc" +credential = { default = false, cased = "DrSc", issue = [753, 848], expands = "doktor vied, Doctor scientiarum, written after the name" } diff --git a/tests/v2/test_locales.py b/tests/v2/test_locales.py index c35162c6..4d84cd1d 100644 --- a/tests/v2/test_locales.py +++ b/tests/v2/test_locales.py @@ -46,6 +46,15 @@ def _pack_modules() -> dict[str, ModuleType]: _PACKS = _pack_modules() + +def _is_vocabulary_pack(pack: Locale) -> bool: + """Whether `pack` is nothing but its choice of languages, so that its + deviation surface is derived (mechanisms.md#LOCALE-PACKS-PURE-DATA).""" + return (bool(pack.languages) and pack.lexicon == Lexicon.empty() + and pack.policy == PolicyPatch()) + + + # ja is the one pack whose behavior is not pack data: it activates # segmentation and supplies nothing to segment WITH, so every parse # through it needs the optional nameparser[ja] segmenter. Probed once @@ -103,8 +112,10 @@ def test_locales_module_attribute_access() -> None: def test_locales_get_and_available() -> None: assert locales.get("ru") is locales.RU - assert set(locales.available()) == {"ja", "ne", "ru", "tr_az", "zh"} - with pytest.raises(KeyError, match="ja, ne, ru, tr_az, zh"): + assert set(locales.available()) == { + "cs", "hr", "ja", "ne", "ru", "sk", "sl", "tr_az", "zh"} + with pytest.raises(KeyError, + match="cs, hr, ja, ne, ru, sk, sl, tr_az, zh"): locales.get("xx") # the segmenter FACTORY is not a locale: it is an ordinary function # on the package, so it must not appear among the codes @@ -1001,6 +1012,65 @@ def _default_corpus() -> list[str]: "佐々木健", # the iteration mark 々 inside a top-20 surname ] +# The choice packs (3f): one name per word of the derived surface, each +# in a shape the default reads differently, so the gate's positive side +# reaches every word the choice changes (held by +# test_choice_rotators_cover_every_changed_word below). +_ROTATORS["cs"] = [ + "BcA Tomáš Krejčí", # bca + "Jan Novák, JUDr", # judr, bare after a comma + "MDDr Eva Dvořáková", # mddr + "MgA Petra Malá", # mga + "MUDr Jan Novák", # mudr + "MVDr Petr Svoboda", # mvdr + "PaedDr Jana Králová", # paeddr + "PharmDr Jana Králová", # pharmdr + "Jan Novák PhDr.", # phdr, dotted at the end + "Novák, RNDr", # rndr, after a one-word head + "ThDr Jan Hus", # thdr + "Jan Novák, DrSc", # drsc, a credential after a comma + "Pan Jan Novák", # pan + "Jana Nováková roz. Svobodová", # roz, the birth marker + "Jiří Dienstbier ml.", # ml + "Jiří Dienstbier st.", # st +] +_ROTATORS["sk"] = [ + "Jozef Mrkva, ArtD.", # artd, a credential after a comma + "Jozef Mrkva, JUDr", # judr + "MDDr Eva Kováčová", # mddr + "MUDr Jozef Mrkva", # mudr + "MVDr Peter Horváth", # mvdr + "PaedDr Jana Kováčová", # paeddr + "PharmDr Jana Kováčová", # pharmdr + "Jozef Mrkva PhDr.", # phdr + "Mrkva, RNDr", # rndr + "ThDr Ján Chryzostom", # thdr + "Jozef Mrkva, DrSc", # drsc + "Ján Mrkva ml.", # ml + "Ján Mrkva st.", # st +] +_ROTATORS["sl"] = [ + "Janez Novak ml.", # ml + "Janez Novak st.", # st +] +_ROTATORS["hr"] = [ + "Ivan Horvat ml.", # ml + "Ivan Horvat st.", # st +] + + +@pytest.mark.parametrize("code", sorted(c for c in _PACKS + if _is_vocabulary_pack( + locales.get(c)))) +def test_choice_rotators_cover_every_changed_word(code: str) -> None: + # the choice packs' counterpart of the marker-branch sweep below: + # a word joining a choice's surface fails here until a rotator + # name holds it, so the gate's positive side cannot thin out + surface = _ri.changed_words(_ri.selected(locales.get(code).languages)) + missing = sorted(w for w in surface if not any( + _surface_deviates(frozenset({w}))(name) for name in _ROTATORS[code])) + assert surface and missing == [] + def _marker_regexes(module: ModuleType) -> list[re.Pattern]: return [v for v in vars(module).values() if isinstance(v, re.Pattern)] @@ -1046,6 +1116,10 @@ def test_registry_is_the_pack_contract() -> None: assert not hasattr(module, "DEVIATES"), ( f"vocabulary pack {code!r} declares DEVIATES by hand; its " f"surface is derived from its choice") + # an empty surface would make its gate pass by parsing + # nothing differently + assert _ri.changed_words(_ri.selected(pack.languages)), ( + f"vocabulary pack {code!r} changes no word") continue assert callable(getattr(module, "DEVIATES", None)), ( f"pack {code!r} does not declare DEVIATES") @@ -1144,9 +1218,17 @@ def test_non_interference_all_packs_combined() -> None: with pytest.warns(UserWarning, match="ja_segmenter"): packed = parser_for( *(locales.get(code) for code in sorted(_PACKS))) + # the packs' languages union into ONE choice, whose surface is + # derived from the union rather than from each pack's own; every + # hand DEVIATES is or'd in whatever the pack, since a pack with a + # choice beside a fragment or a patch carries both + chosen = frozenset().union(*(locales.get(c).languages for c in _PACKS)) + choice_deviates = _surface_deviates( + _ri.changed_words(_ri.selected(chosen))) + hand = [m.DEVIATES for m in _PACKS.values() if hasattr(m, "DEVIATES")] declared = _assert_non_interference( packed, - lambda n: any(_deviates(code)(n) for code in _PACKS), + lambda n: choice_deviates(n) or any(d(n) for d in hand), corpus) assert declared >= len(all_rotators) @@ -1193,13 +1275,6 @@ def deviates(name: str) -> bool: return deviates -def _is_vocabulary_pack(pack: Locale) -> bool: - """Whether `pack` is nothing but its choice of languages, so that its - deviation surface is derived (mechanisms.md#LOCALE-PACKS-PURE-DATA).""" - return (bool(pack.languages) and pack.lexicon == Lexicon.empty() - and pack.policy == PolicyPatch()) - - @functools.cache def _deviates(code: str) -> Callable[[str], bool]: """The registered pack `code`'s deviation predicate: the surface @@ -1364,6 +1439,157 @@ def test_no_exclusion_is_pack_shaped_any_more() -> None: assert vocab.is_dir() and found == [] +# -- 3f: the Czech, Slovak, Slovenian and Croatian packs -- +# +# Each row: a pack (or a choice of tags joined by +), an order, a name, +# then (title, given, middle, family, suffix, maiden, ambiguity kinds) +# under the default and under the pack. Every row but the last two +# groups is a case the default refuses and the pack reads; those two +# pin what choosing the language costs, and what it leaves alone +# (decisions/vocabulary-model's 2026-10-30 #753, phase 3f entry). +_ORDERS = {None: Policy(), "FAMILY_FIRST": Policy(name_order=FAMILY_FIRST), + "FAMILY_FIRST_GIVEN_LAST": Policy( + name_order=FAMILY_FIRST_GIVEN_LAST)} + + +@pytest.mark.parametrize("tags, order, name, default, packed", [ + # an academic title written bare, in each comma shape and order; + # its dotted spelling at the head is a title by its shape in both + ("cs", None, "MUDr Jan Novák", + ("", "MUDr", "Jan", "Novák", "", "", ()), + ("MUDr", "Jan", "", "Novák", "", "", ())), + ("cs", None, "Jan Novák, MUDr", + ("", "MUDr", "", "Jan Novák", "", "", ()), + ("MUDr", "Jan", "", "Novák", "", "", ())), + ("cs", None, "Novák, MUDr", + ("", "MUDr", "", "Novák", "", "", ()), + ("MUDr", "", "", "Novák", "", "", ())), + ("cs", None, ", MUDr Jan", + ("", "MUDr", "Jan", "", "", "", ()), + ("MUDr", "", "", "Jan", "", "", ())), + ("cs", "FAMILY_FIRST", "MUDr Novák Jan", + ("", "Novák", "Jan", "MUDr", "", "", ()), + ("MUDr", "Jan", "", "Novák", "", "", ())), + ("sk", None, "PhDr Jozef Mrkva", + ("", "PhDr", "Jozef", "Mrkva", "", "", ()), + ("PhDr", "Jozef", "", "Mrkva", "", "", ())), + # dotted at the end, a listed title there (rules/H5), in mixed case + # and in one case + ("cs", None, "Jan Novák MUDr.", + ("", "Jan", "Novák", "MUDr.", "", "", ()), + ("MUDr.", "Jan", "", "Novák", "", "", ())), + ("cs", None, "JAN NOVÁK MUDR.", + ("", "JAN", "NOVÁK", "MUDR.", "", "", ()), + ("MUDR.", "JAN", "", "NOVÁK", "", "", ())), + # the post-nominals: a credential after a comma, where the default + # reverses the bare one and takes the dotted one for a title + ("cs", None, "Jan Novák, DrSc", + ("", "DrSc", "", "Jan Novák", "", "", ()), + ("", "Jan", "", "Novák", "DrSc", "", ())), + ("cs", None, "prof. MUDr. Jan Novák, DrSc.", + ("prof. MUDr. DrSc.", "Jan", "", "Novák", "", "", ()), + ("prof. MUDr.", "Jan", "", "Novák", "DrSc.", "", ())), + ("sk", None, "Jozef Mrkva, ArtD.", + ("ArtD.", "Jozef", "", "Mrkva", "", "", ()), + ("", "Jozef", "", "Mrkva", "ArtD.", "", ())), + # the salutation and the birth marker, each converted from an + # exclusion; pl's pan with cs's + ("cs", None, "Pan Jan Novák", + ("", "Pan", "Jan", "Novák", "", "", ()), + ("Pan", "Jan", "", "Novák", "", "", ())), + ("pl", None, "Pan Jan Kowalski", + ("", "Pan", "Jan", "Kowalski", "", "", ()), + ("Pan", "Jan", "", "Kowalski", "", "", ())), + ("cs", None, "Jana Nováková roz. Svobodová", + ("", "Jana", "Nováková roz.", "Svobodová", "", "", ()), + ("", "Jana", "", "Nováková", "", "Svobodová", ())), + # the generations 3c wrote, turned on by each language's pack; the + # gated st reports the fork, the ungated ml does not + ("sl", None, "Janez Novak st.", + ("st.", "Janez", "", "Novak", "", "", ()), + ("", "Janez", "", "Novak", "st.", "", ("SUFFIX_OR_NAME",))), + ("hr", None, "Ivan Horvat ml.", + ("", "Ivan", "Horvat", "ml.", "", "", ()), + ("", "Ivan", "", "Horvat", "ml.", "", ())), + # behind a title-particle head after a family comma, the trailing + # peel takes ml.; the gated st. stays in the given part unreported, + # as the default reads fils there (Dumas, Freiherr von Berg fils) + ("cs", None, "Novák, Freiherr von Berg ml.", + ("Freiherr", "von Berg ml.", "", "Novák", "", "", ()), + ("Freiherr", "von Berg", "", "Novák", "ml.", "", ())), + # one word before a family comma: the dotted st. is a title by H2's + # shape, reported suffix-or-name as the default reports Dumas, fils.; + # the ungated ml. is the suffix + ("cs", None, "Novák, st.", + ("st.", "", "", "Novák", "", "", ()), + ("st.", "", "", "Novák", "", "", ("SUFFIX_OR_NAME",))), + ("cs", None, "Novák, ml.", + ("ml.", "", "", "Novák", "", "", ()), + ("", "", "", "Novák", "ml.", "", ())), + ("cs", None, "Novák, Freiherr von Berg st.", + ("Freiherr", "von Berg", "", "st. Novák", "", "", + ("PARTICLE_OR_GIVEN",)), + ("Freiherr", "von Berg st.", "", "Novák", "", "", ())), + # the costs: Pan the surname, Roz the given name, and the Saint + # readings of st, which a choice of fr keeps + ("cs", None, "Smith, Pan", + ("", "Pan", "", "Smith", "", "", ()), + ("Pan", "", "", "Smith", "", "", ())), + ("cs", None, "Pan Wei", + ("", "Pan", "", "Wei", "", "", ()), + ("Pan", "", "", "Wei", "", "", ())), + ("cs", None, "Rosalind Roz Smith", + ("", "Rosalind", "Roz", "Smith", "", "", ()), + ("", "", "", "Rosalind", "", "Smith", ())), + ("cs", None, "St Clair Smith", + ("St", "Clair", "", "Smith", "", "", ()), + ("", "St", "Clair", "Smith", "", "", ())), + ("cs+fr", None, "St Clair Smith", + ("St", "Clair", "", "Smith", "", "", ()), + ("St", "Clair", "", "Smith", "", "", ())), + # left alone: a bare title ending the given part is a name word + # (rules/H5), the title Mgr was one already, and BC, excluded for + # the US credential it spells, stays one (cs.toml's bc) + ("cs", "FAMILY_FIRST_GIVEN_LAST", "Novák, Jan MUDr", + ("", "Jan", "MUDr", "Novák", "", "", ()), + ("", "Jan", "MUDr", "Novák", "", "", ())), + ("cs", None, "Mgr Jana Nováková", + ("Mgr", "Jana", "", "Nováková", "", "", ()), + ("Mgr", "Jana", "", "Nováková", "", "", ())), + ("cs", None, "Lora L. Beebe, PhD, PMHNP, BC", + ("", "Lora", "L.", "Beebe", "PhD, PMHNP, BC", "", + ("COMMA_STRUCTURE", "COMMA_STRUCTURE")), + ("", "Lora", "L.", "Beebe", "PhD, PMHNP, BC", "", + ("COMMA_STRUCTURE", "COMMA_STRUCTURE"))), +]) +def test_the_west_and_south_slavic_packs_read_what_the_default_refuses( + tags: str, order: str | None, name: str, + default: tuple[object, ...], packed: tuple[object, ...]) -> None: + base = Parser(policy=_ORDERS[order]) + chosen = frozenset(tags.split("+")) + # a shipped pack where one is registered; pl, and two tags, as a + # choice of languages of the same shape + pack = (locales.get(tags) if tags in locales.available() else + Locale(code="choice", lexicon=Lexicon.empty(), languages=chosen)) + assert pack.languages == chosen + + def fields(n: ParsedName) -> tuple[object, ...]: + return (n.title, n.given, n.middle, n.family, n.suffix, n.maiden, + tuple(a.kind.name for a in n.ambiguities)) + assert (fields(base.parse(name)), + fields(parser_for(pack, base=base).parse(name))) == ( + default, packed) + + +def test_the_west_and_south_slavic_packs_repair_their_titles_case() -> None: + # the readings' cased masks join with them, so capitalized() writes + # the title as Czech does under the pack and leaves it otherwise + cs = parser_for(locales.CS) + assert str(cs.capitalized(cs.parse("mudr. jan novák"))) == ( + "MUDr. Jan Novák") + assert str(parse("mudr. jan novák").capitalized()) == "Mudr. Jan Novák" + + # -- #269: non-Latin default vocabulary (Cyrillic, Greek, Arabic, # Hebrew, Devanagari, Bengali) -- # diff --git a/tools/differential/cldr_agreement.py b/tools/differential/cldr_agreement.py index 28348f69..8ec89b0c 100644 --- a/tools/differential/cldr_agreement.py +++ b/tools/differential/cldr_agreement.py @@ -141,7 +141,8 @@ #: keyed like the default one. A new pack adds its files here, or its #: name to NO_CLDR; test_cldr_agreement.py fails on a pack in neither. LOCALE_FILES = {"zh": "ZH", "zh_Hant": "ZH", "yue": "ZH", "yue_Hans": "ZH", - "ja": "JA", "ne": "NE", "ru": "RU", "tr": "TR_AZ", "az": "TR_AZ"} + "ja": "JA", "ne": "NE", "ru": "RU", "tr": "TR_AZ", "az": "TR_AZ", + "cs": "CS", "sk": "SK", "sl": "SL", "hr": "HR"} #: Packs with no CLDR file to measure them on. NO_CLDR: frozenset[str] = frozenset() #: The locale view of the native-script Wikidata source: stratum label -> diff --git a/tools/differential/cldr_agreement_locales.json b/tools/differential/cldr_agreement_locales.json index a68dbdbf..95aba3a3 100644 --- a/tools/differential/cldr_agreement_locales.json +++ b/tools/differential/cldr_agreement_locales.json @@ -125,6 +125,206 @@ }, "text": "Sinbad" }, + "cs/foreignFull/givenFirst": { + "surname2": { + "expected": "González Domingo", + "got": "" + }, + "text": "Prof. Dr. Ada Cornelia César Martín von Brühl González Domingo, M.D. DDS." + }, + "cs/foreignFull/sorting": { + "suffix": { + "expected": "", + "got": "M.D. DDS." + }, + "text": "Brühl, Ada Cornelia César Martín von (Prof. Dr., M.D. DDS.)", + "title": { + "expected": "", + "got": "Prof. Dr." + } + }, + "cs/foreignFull/surnameFirst": { + "family": { + "expected": "von Brühl González Domingo", + "got": "von" + }, + "family_base": { + "expected": "Brühl González Domingo", + "got": "von" + }, + "family_particles": { + "expected": "von", + "got": "" + }, + "given_names": { + "expected": "Ada Cornelia César Martín", + "got": "Brühl González Domingo Ada Cornelia César Martín" + }, + "surname2": { + "expected": "González Domingo", + "got": "" + }, + "text": "Prof. Dr. von Brühl González Domingo Ada Cornelia César Martín, M.D. DDS." + }, + "cs/foreignG/sorting": { + "family": { + "expected": "", + "got": "Jack" + }, + "given_names": { + "expected": "Jack", + "got": "" + }, + "text": "Jack" + }, + "cs/foreignG/surnameFirst": { + "family": { + "expected": "", + "got": "Jack" + }, + "given_names": { + "expected": "Jack", + "got": "" + }, + "text": "Jack" + }, + "cs/nativeFull/givenFirst": { + "family": { + "expected": "Machová Ondřejová", + "got": "Ondřejová" + }, + "family_base": { + "expected": "Machová Ondřejová", + "got": "Ondřejová" + }, + "given_names": { + "expected": "Alexandra Zuzana", + "got": "Alexandra Zuzana Machová" + }, + "surname2": { + "expected": "Ondřejová", + "got": "" + }, + "text": "paní Alexandra Zuzana Machová Ondřejová, Ph.D." + }, + "cs/nativeFull/sorting": { + "given_names": { + "expected": "Alexandra Zuzana", + "got": "Alexandra Zuzana paní" + }, + "suffix": { + "expected": "", + "got": "Ph.D." + }, + "text": "Machová, Alexandra Zuzana (paní, Ph.D.)" + }, + "cs/nativeFull/surnameFirst": { + "family": { + "expected": "Machová Ondřejová", + "got": "Machová" + }, + "family_base": { + "expected": "Machová Ondřejová", + "got": "Machová" + }, + "given_names": { + "expected": "Alexandra Zuzana", + "got": "Ondřejová Alexandra Zuzana" + }, + "surname2": { + "expected": "Ondřejová", + "got": "" + }, + "text": "paní Machová Ondřejová Alexandra Zuzana, Ph.D." + }, + "cs/nativeG/sorting": { + "family": { + "expected": "", + "got": "Jiří" + }, + "given_names": { + "expected": "Jiří", + "got": "" + }, + "text": "Jiří" + }, + "cs/nativeG/surnameFirst": { + "family": { + "expected": "", + "got": "Jiří" + }, + "given_names": { + "expected": "Jiří", + "got": "" + }, + "text": "Jiří" + }, + "hr/foreignG/sorting": { + "family": { + "expected": "", + "got": "Sinbad" + }, + "given_names": { + "expected": "Sinbad", + "got": "" + }, + "text": "Sinbad" + }, + "hr/foreignG/surnameFirst": { + "family": { + "expected": "", + "got": "Sinbad" + }, + "given_names": { + "expected": "Sinbad", + "got": "" + }, + "text": "Sinbad" + }, + "hr/nativeFull/givenFirst": { + "suffix": { + "expected": "mlađi prof.", + "got": "mlađi" + }, + "text": "dr. sc. Branko Hrvoje Marić mlađi, prof.", + "title": { + "expected": "dr. sc.", + "got": "dr. sc. prof." + } + }, + "hr/nativeFull/surnameFirst": { + "suffix": { + "expected": "mlađi prof.", + "got": "mlađi" + }, + "text": "Marić, dr. sc. Branko Hrvoje mlađi, prof.", + "title": { + "expected": "dr. sc.", + "got": "dr. sc. prof." + } + }, + "hr/nativeG/sorting": { + "family": { + "expected": "", + "got": "Zrinka" + }, + "given_names": { + "expected": "Zrinka", + "got": "" + }, + "text": "Zrinka" + }, + "hr/nativeG/surnameFirst": { + "family": { + "expected": "", + "got": "Zrinka" + }, + "given_names": { + "expected": "Zrinka", + "got": "" + }, + "text": "Zrinka" + }, "ja/foreignFull/givenFirst": { "family": { "expected": "フォン スミス", @@ -545,6 +745,249 @@ }, "text": "Сергей" }, + "sk/foreignFull/givenFirst": { + "surname2": { + "expected": "Nováková Predná", + "got": "" + }, + "text": "Mgr. Ing. Anna Karolína Eva Kristína von Kant Nováková Predná, CSc." + }, + "sk/foreignFull/sorting": { + "family": { + "expected": "Kant von", + "got": "Kant" + }, + "family_particles": { + "expected": "von", + "got": "" + }, + "given_names": { + "expected": "Anna Karolína Eva Kristína", + "got": "Anna Karolína Eva Kristína von Mgr. Ing." + }, + "suffix": { + "expected": "", + "got": "CSc." + }, + "text": "Kant, Anna Karolína Eva Kristína von (Mgr. Ing., CSc.)" + }, + "sk/foreignFull/surnameFirst": { + "family": { + "expected": "von Kant Nováková Predná", + "got": "von" + }, + "family_base": { + "expected": "Kant Nováková Predná", + "got": "von" + }, + "family_particles": { + "expected": "von", + "got": "" + }, + "given_names": { + "expected": "Anna Karolína Eva Kristína", + "got": "Kant Nováková Predná Anna Karolína Eva Kristína" + }, + "surname2": { + "expected": "Nováková Predná", + "got": "" + }, + "text": "Mgr. Ing. von Kant Nováková Predná Anna Karolína Eva Kristína, CSc." + }, + "sk/foreignG/sorting": { + "family": { + "expected": "", + "got": "Jack" + }, + "given_names": { + "expected": "Jack", + "got": "" + }, + "text": "Jack" + }, + "sk/foreignG/surnameFirst": { + "family": { + "expected": "", + "got": "Jack" + }, + "given_names": { + "expected": "Jack", + "got": "" + }, + "text": "Jack" + }, + "sk/nativeFull/givenFirst": { + "given_names": { + "expected": "Alexandra Zuzana", + "got": "pani Alexandra Zuzana" + }, + "text": "pani Alexandra Zuzana Macáková PhD", + "title": { + "expected": "pani", + "got": "" + } + }, + "sk/nativeFull/surnameFirst": { + "family": { + "expected": "Macáková", + "got": "pani" + }, + "family_base": { + "expected": "Macáková", + "got": "pani" + }, + "given_names": { + "expected": "Alexandra Zuzana", + "got": "Macáková Alexandra Zuzana" + }, + "text": "pani Macáková Alexandra Zuzana, PhD", + "title": { + "expected": "pani", + "got": "" + } + }, + "sk/nativeG/sorting": { + "family": { + "expected": "", + "got": "Jozef" + }, + "given_names": { + "expected": "Jozef", + "got": "" + }, + "text": "Jozef" + }, + "sk/nativeG/surnameFirst": { + "family": { + "expected": "", + "got": "Jozef" + }, + "given_names": { + "expected": "Jozef", + "got": "" + }, + "text": "Jozef" + }, + "sl/foreignFull/givenFirst": { + "suffix": { + "expected": "ml. dr. med., dr. dent. med.", + "got": "ml., dent. med." + }, + "text": "prof. dr. Ada Cornelia César Martín von Brühl ml., dr. med., dr. dent. med.", + "title": { + "expected": "prof. dr.", + "got": "prof. dr. dr. med. dr." + } + }, + "sl/foreignFull/surnameFirst": { + "family": { + "expected": "von Brühl", + "got": "von" + }, + "family_base": { + "expected": "Brühl", + "got": "von" + }, + "family_particles": { + "expected": "von", + "got": "" + }, + "given_names": { + "expected": "Ada Cornelia César Martín", + "got": "Brühl Ada Cornelia César Martín" + }, + "suffix": { + "expected": "dr. med., dr. dent. med.", + "got": "dent. med." + }, + "text": "prof. dr. von Brühl Ada Cornelia César Martín, dr. med., dr. dent. med.", + "title": { + "expected": "prof. dr.", + "got": "prof. dr. dr. med. dr." + } + }, + "sl/foreignG/sorting": { + "family": { + "expected": "", + "got": "Sinbad" + }, + "given_names": { + "expected": "Sinbad", + "got": "" + }, + "text": "Sinbad" + }, + "sl/foreignG/surnameFirst": { + "family": { + "expected": "", + "got": "Sinbad" + }, + "given_names": { + "expected": "Sinbad", + "got": "" + }, + "text": "Sinbad" + }, + "sl/nativeFull/givenFirst": { + "given_names": { + "expected": "Janez Peter", + "got": "g. Janez Peter" + }, + "suffix": { + "expected": "ml. univ. dipl. inž.", + "got": "ml." + }, + "text": "g. Janez Peter Novak ml., univ. dipl. inž.", + "title": { + "expected": "g.", + "got": "univ. dipl. inž." + } + }, + "sl/nativeFull/surnameFirst": { + "family": { + "expected": "Novak", + "got": "g." + }, + "family_base": { + "expected": "Novak", + "got": "g." + }, + "given_names": { + "expected": "Janez Peter", + "got": "Novak Janez Peter" + }, + "suffix": { + "expected": "univ. dipl. inž.", + "got": "" + }, + "text": "g. Novak Janez Peter, univ. dipl. inž.", + "title": { + "expected": "g.", + "got": "univ. dipl. inž." + } + }, + "sl/nativeG/sorting": { + "family": { + "expected": "", + "got": "Svarun" + }, + "given_names": { + "expected": "Svarun", + "got": "" + }, + "text": "Svarun" + }, + "sl/nativeG/surnameFirst": { + "family": { + "expected": "", + "got": "Svarun" + }, + "given_names": { + "expected": "Svarun", + "got": "" + }, + "text": "Svarun" + }, "tr/foreignFull/givenFirst": { "given_names": { "expected": "Ada Cornelia César Martín", diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 87b5d8a8..807473b9 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -404,6 +404,7 @@ "Jean Pierre MA DUPONT" "Jeffrey (JD) Bricken" "Jiří Dienstbier ml." +"Jiří Dienstbier st." "John (Jack) Smith" "John . Smith" "John Dean Smith" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 2b4dfe14..5491a1bc 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -1624,6 +1624,7 @@ {"name": "Jiří", "given": "Jiří", "ambiguities": ["GIVEN_OR_FAMILY"], "alternatives": ["GIVEN_OR_FAMILY: name as family (position)"], "initials": "J."} {"name": "Jiří", "order": "FAMILY_FIRST", "family": "Jiří", "ambiguities": ["GIVEN_OR_FAMILY"], "alternatives": ["GIVEN_OR_FAMILY: name as given (position)"], "initials": "J."} {"name": "Jiří Dienstbier ml.", "given": "Jiří", "middle": "Dienstbier", "family": "ml.", "initials": "J. D. m.", "capitalized_force": "Jiří Dienstbier Ml."} +{"name": "Jiří Dienstbier st.", "title": "st.", "given": "Jiří", "family": "Dienstbier", "initials": "J. D.", "capitalized_force": "St. Jiří Dienstbier"} {"name": "Jo Ann", "given": "Jo", "family": "Ann", "initials": "J. A."} {"name": "Joan Antoni Desvalls i d’Ardena", "given": "Joan", "middle": "Antoni", "family": "Desvalls i d’Ardena", "initials": "J. A. D. d.", "capitalized_force": "Joan Antoni Desvalls i D’Ardena"} {"name": "Joan Canals i Pigem", "given": "Joan", "family": "Canals i Pigem", "initials": "J. C. P."}