diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 12517bce..fc376331 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -109,6 +109,16 @@ Open: [#380](https://github.com/derek73/python-nameparser/issues/380) covers "Be - 2026-07-03 (maiden-bucket design; #274 filed 2026-07-07, landed in the v2 core, PR #288) — the marker takes everything after it up to a trailing suffix, greedily: "née Jones Smith" is a two-word maiden name, matching how the marker is actually used in running text. The marker itself is dropped as structural, like a delimiter character. +- 2026-08-19 #399 (PR #409) — a maiden marker bounds a particle join reaching it from the left, where the marker is actually taken as a marker. rules.md carried the opposite as an Accepted limitation ("a particle chain swallows a marker in its path, the join (P2) running first"), but that was a consequence of stage order rather than a judgment: the chain merges pieces before the marker is consumed, and the consumer looks for a lone marker piece which by then no longer exists. So `Ursula von der Leyen geb. Albrecht` lost the maiden name that `Ursula Leyen geb. Albrecht` kept. + + The gate on the stop is the whole substance of the rule and was added in review, not in the first cut. Sharing one marker-piece predicate between the stop and the consumer settles what a marker piece IS; it does NOT settle whether the consumer takes, because the consumer also needs a non-suffix piece after the marker. Stopping without that left the marker as a piece of its own, and a lone trailing piece takes a role field: `Jane van der Berg née` read middle 'van der Berg', family 'née' — the marker became the surname and the real surname was demoted, which is the stranding the stop exists to prevent, one field over. Three consequences of the ungated form, each measured and each fixed by the gate: `Jane van der Nee` moved a Dutch bearer of the attested surname Nee out of the family (M1 already names Nee as a surname); `Ursula von der Leyen geb.`, an ordinary truncated record, reported the marker as the surname; and `St St née` stopped reporting its particle-or-given fork, because group's emitter is guarded on the chain having merged something and the stop made it merge nothing for a different reason than the guard assumes — an A1 violation of the shape #405 tracks, and worse than #405's, since it REMOVES a report callers already see. The gate tests the consumer's own condition rather than restating it, which is what makes the two halves agree. M2 states the gate ("a marker left as a word bounds nothing"); the trailing-marker rows pin it. + + Weighed against moving the maiden handler ahead of the chain, and the recorded reason for preferring the stop has been corrected twice. It is NOT that the reorder disturbs names whose maiden side carries its own particle chain: `Anna Müller geb. von der Berg` is identical before #399, after it, and under a reconstructed reorder, and mechanically cannot differ — no suffix and no conjunction on the maiden side means merged and unmerged pieces hand the consumer the same words. The reason is scope: the reorder changes what the consumer receives for every name rather than adding one condition, and two review sweeps found no test and no corpus name that separates the two options, so it would ship unmeasurable. A review sweep also found the reorder would reach maiden names the stop does not (the P3/P5 join-swallow below) at a cost on the given side, through P3's rootname count dropping when the maiden words leave early; neither figure is re-derived here, and the reorder stays available if that class is ever worth fixing. + + Two things fell out rather than being decided separately: the same issue's open question about the leftover given slot (the marker used to compete for it, giving `de la Cruz née Vega` given "née" under a family-first order; consumed and dropped, it never reaches the placement — though a marker the consumer declines still does, which is M2's "just a word" reading and not a leak), and the wholly-particle family a stop can leave, which #404's R2 reading already covers — "Jane de la née Jones" reports family "de la" with a non-empty base. A 305,364-parse sweep at review found no empty-base or partition violation either side of the change. + + Two limits stay, now recorded in M2 rather than left implicit: the bound reaches only a marker standing as a word of its own, so P3's connective join and P5's bound-given join each still absorb a marker before the bound can see it — the deleted Accepted clause covered a class of which this fixed one instance and left two. Tests carry the verification: of the marker-bearing differential corpus names none had a particle ahead of the marker, so the gate was blind to the class. Adding the cross-script case row put one such name in the CJK corpus, so it is gate-visible from here on, with its own ledger rule. + Open (M2): [#317](https://github.com/derek73/python-nameparser/issues/317) the fullwidth-colon marker (旧姓：佐藤 arrives as one word; the head-peel question). diff --git a/docs/design/rules.md b/docs/design/rules.md index de774bbb..d4f44e1d 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -156,8 +156,10 @@ P1. Rationale: a never-given particle standing alone cannot be P2. Rationale: a particle is written as part of the surname it precedes, and a title stands outside the name entirely. A particle joins the words after it into one name part, the - join running until the next particle starts a group of its own - or the name ends. The final group reads as the family name; + join running until the next particle starts a group of its own, + a trailing suffix begins, a maiden marker takes the words after + it (M2), or the name ends. The final group reads as the family + name; earlier groups read by position. The chain begins wherever the name begins, and a preceding title does not move that point. Where P1's fold has claimed the opening, the fold decides the @@ -169,11 +171,13 @@ P2. Rationale: a particle is written as part of the surname it "Dr. John van der Berg" → family="van der Berg" "Juan de" → family="de" · boundary "de la Cruz Juan Carlos" family-first → family="de la Cruz" + "John van der Berg PhD" → family="van der Berg" + "John van der Berg née Jones" → family="van der Berg" Accepted: a caller wanting the combined double-surname reading (#132's ask) has it as the surnames view rather than the family field. "Vincent van Gogh van Beethoven" → surnames="van Gogh van Beethoven" - history: decisions.md#P2 · interacts: P1, P4 · implemented: nameparser/_pipeline/_group.py, nameparser/_pipeline/_post_rules.py + history: decisions.md#P2 · interacts: P1, P4, M2 · implemented: nameparser/_pipeline/_group.py, nameparser/_pipeline/_post_rules.py P3. Rationale: connective words ("y", "of the") bind name words into one name part; but a single letter in a short name is more @@ -426,21 +430,33 @@ M2. Rationale: a maiden marker announces that what follows it is the word takes the words after it — up to any trailing suffix — as the maiden name, and the marker itself is dropped. A marker with nothing after it, or nothing before it, is just a word. + A marker taken this way also bounds a particle join arriving from + its left (P2), so the family name's particles stop at the marker + instead of absorbing it; a marker left as a word bounds nothing. + Where the bound leaves a family of nothing but particles, they + are not in particle position and read as ordinary words (R2). "Jane Smith née Jones" → maiden="Jones" "Jane née Jones Smith" → maiden="Jones Smith" "Jane Smith née Jones PhD" → suffix="PhD" "Jones née" → family="née" · boundary "née Jones" → family="Jones" · boundary + "Jane van der Berg née Jones" → maiden="Jones" + "Jane de la née Jones" → family="de la" + "Jane van der Berg née" → family="van der Berg née" + "Jane van der Berg née PhD" → family="van der Berg née" Accepted: the fullwidth-colon spelling arrives as one word, so the marker inside it goes unrecognized; #317 tracks whether it should peel. "山田 花子 旧姓：佐藤" → maiden="" Accepted: a marker straight after a comma is post-comma given - text, not a marker; and a particle chain swallows a marker in - its path, the join (P2) running first. - "Jane Smith, née Jones" → maiden="" - "Jane de la née Jones" → family="de la née Jones" - history: decisions.md#M2 · interacts: P2, M1 · implemented: nameparser/_pipeline/_group.py + text, not a marker; and the bound reaches only a marker standing + as a word of its own, so the connective join (P3) and the + bound-given join (P5) each still absorb a marker before the bound + can see it. + "Jane Smith, née Jones" → maiden="" + "Jane van der Berg née y Jones" → maiden="" + "van der Berg, abdul née Jones" → given="abdul née" + history: decisions.md#M2 · interacts: P2, P3, P5, R2, M1 · implemented: nameparser/_pipeline/_group.py ## Commas & structure (C) diff --git a/docs/release_log.rst b/docs/release_log.rst index 96d4210a..309bf664 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -42,6 +42,8 @@ Release Log - Change the ``detail`` text of a ``PARTICLE_OR_GIVEN`` ambiguity to name the role the leading particle was actually given. It said "read as a given name" under every ``name_order``, which is false under ``Policy(name_order=FAMILY_FIRST)`` -- there ``"Van Johnson"`` reads as family ``Van``, given ``Johnson``, and the report described the reading not taken. It now ends "read as a family name" in that case, reading the role off the assigned token the way ``SUFFIX_OR_NAME`` already did -- that kind names both parts (``read as a family name rather than a post-nominal``), while this one names only the part it took. The ``kind`` is unchanged and stays ``PARTICLE_OR_GIVEN``: the fork really is particle-or-given, and only the human-readable text moved. Default-order output is identical (#355) + - Fix a maiden name being lost when a particle stood in front of the marker. ``"Ursula Leyen geb. Albrecht"`` reported maiden ``Albrecht`` correctly, but ``"Ursula von der Leyen geb. Albrecht"`` -- the same words one particle chain apart -- gave family ``von der Leyen geb. Albrecht`` and no maiden name at all, and ``"Jane van der Berg née Jones"`` failed the same way. The particle chain absorbed everything up to the next particle or suffix, and the marker is consumed later in the same grouping stage, by which point the chain had merged the marker away and there was no marker word left standing to find. A suffix already stopped the chain; a marker now does too, so those read family ``von der Leyen`` maiden ``Albrecht`` and family ``van der Berg`` maiden ``Jones``. Only a particle that is not the name's own first word ever reached the marker, which is why a single leading particle always worked -- ``"von Müller geb. Schmidt"`` was already right -- while a leading run of two did not: in ``"von der Müller geb. Schmidt"`` the second particle starts a chain of its own. The bound applies only where the marker is actually taken as a marker. A marker needs a word after it to hand the maiden name to, so a trailing one is just a word and bounds nothing: ``"Jane van der Berg née"`` and ``"Jane van der Berg née PhD"`` both keep family ``van der Berg née``, which is the reading ``"Jones née"`` -> family ``née`` already had. That distinction is what keeps a name like ``"Jane van der Nee"`` intact -- ``Nee`` is an attested surname as well as a marker spelling -- and what keeps ``"St St née"`` reporting its ``PARTICLE_OR_GIVEN`` fork. Two consequences fall out. Under a family-first order the marker used to survive as an ordinary name word and take the leftover given slot, so ``"de la Cruz née Vega"`` read given ``née``, middle ``Vega`` under ``Policy(name_order=FAMILY_FIRST)`` and given ``Vega``, middle ``née`` under ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)``; consumed and dropped it never reaches that placement, and both orders now read family ``de la Cruz``, maiden ``Vega``. And stopping the chain can leave a family of nothing but particles, which the #385 rule above already covers: ``"Jane de la née Jones"`` reports family ``de la`` with ``family_base`` ``de la`` rather than an empty base. Two limits remain, and both are recorded in ``rules.md#M2``: the bound reaches only a marker standing as a word of its own, so a conjunction join and a bound given-name join each still absorb one first -- ``"Jane van der Berg née y Jones"`` and ``"van der Berg, abdul née Jones"`` are unchanged. The bracketed form was never affected, since ``"Ursula von der Leyen (geb. Albrecht)"`` is extracted before grouping. No differential corpus name carried a particle ahead of a marker, so this class was invisible to the harness at every baseline; a name of the shape was added to the CJK corpus with this change, and it now moves one name with its own ledger rule (closes #399) + **Deprecations** - Rename the four vocabularies whose 1.x names described the fields they feed in v1's words, so the data layer matches the ``Lexicon``: diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index 70da1d15..e1f20c62 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -66,8 +66,10 @@ def _is_title_piece(piece: Sequence[int], ptags: Set[str], # rules.md#P2: "a particle joins the words after it into one name # part, the join running until the next particle starts a group of -# its own or the name ends. The final group reads as the family -# name; earlier groups read by position." (history: decisions.md#P2) +# its own, a trailing suffix begins, a maiden marker takes the words +# after it (M2), or the name ends. The final group reads as the +# family name; earlier groups read by position." +# (history: decisions.md#P2) # rules.md#P4: "a particle in the name's leading position chains # nothing: the words stay separate" (history: decisions.md#P2) def _is_prefix_piece(piece: Sequence[int], ptags: Set[str], @@ -87,6 +89,22 @@ def _is_suffix_piece(piece: Sequence[int], ptags: Set[str], return "vocab:suffix" in tags and "initial" not in tags +# rules.md#M2: "a recognized maiden marker standing after at least +# one name word takes the words after it — up to any trailing +# suffix — as the maiden name, and the marker itself is dropped" +# (history: decisions.md#M2) +# +# Shared deliberately with the prefix chain's stop (#399): a chain that +# stopped at something the consumer below would not then take would +# leave the marker stranded inside the family name, which is the very +# defect the stop exists to fix. One definition, so the two cannot +# disagree about what a marker piece is. +def _is_maiden_marker_piece(piece: Sequence[int], + tokens: Sequence[WorkToken]) -> bool: + return (len(piece) == 1 + and "vocab:maiden-marker" in tokens[piece[0]].tags) + + # rules.md#P3: "a recognized connective joins its neighbors into one # name part, connective runs included — except a single-letter # connective in a three-word name, which stays a name word, and a @@ -133,6 +151,21 @@ def suffix(k: int) -> bool: def conj(k: int) -> bool: return _is_conj_piece(pieces[k], ptags[k], tokens) + def maiden_marker_stop(k: int) -> bool: + # A marker bounds the chain only where the consumer below will + # actually take it (#399). That consumer needs a non-suffix + # piece after the marker; stopping without one left the marker + # as a piece of its own, and a lone trailing piece takes a role + # field -- so the marker became the family name and the real + # surname was demoted to the middle ("Jane van der Berg née" + # read middle 'van der Berg', family 'née'). That is the defect + # this stop exists to prevent, one field over. Testing the + # consumer's own condition rather than restating it is what + # makes the two halves agree. + return (_is_maiden_marker_piece(pieces[k], tokens) + and any(not _is_suffix_piece(pieces[x], ptags[x], tokens) + for x in range(k + 1, len(pieces)))) + def merge(lo: int, hi: int, add: Set[str] = frozenset(), drop: Set[str] = frozenset()) -> None: # pieces/ptags are parallel arrays; every merge must update @@ -253,6 +286,18 @@ def merge(lo: int, hi: int, add: Set[str] = frozenset(), # classifies its leading piece as a TITLE and is already # covered here. # + # A maiden marker stops the scan for a different reason than a + # suffix does (#399): it is not a name word at all but the + # boundary between two names, and the piece that consumes it + # runs later in this same stage. Absorbing it took the maiden + # name into the family with it -- "Ursula von der Leyen geb. + # Albrecht" read family 'von der Leyen geb. Albrecht' where + # "Ursula Leyen geb. Albrecht" reported maiden 'Albrecht'. Only + # a NON-leading particle ever reached the marker, so a leading + # single particle always worked (P4 chains nothing) while a + # leading run of two did not, the second particle's own chain + # firing. + # # The `, 0` fallback is inert by construction rather than a # default worth testing: it is reached only when every piece is # a title and none is a prefix, and the loop below merges @@ -267,7 +312,8 @@ def merge(lo: int, hi: int, add: Set[str] = frozenset(), j = k + 1 while j < len(pieces) and prefix(j): j += 1 - while j < len(pieces) and not prefix(j) and not suffix(j): + while (j < len(pieces) and not prefix(j) and not suffix(j) + and not maiden_marker_stop(j)): j += 1 # The other half of PARTICLE_OR_GIVEN. _assign reports the # fork when an ambiguous particle stays a lone leading piece @@ -430,11 +476,9 @@ def group(state: ParseState) -> ParseState: # maiden markers: a non-leading marker piece consumes following # pieces until a suffix; consumed tokens become MAIDEN, the # marker is dropped (#274) - m = next( - (k for k in range(1, len(pieces)) - if len(pieces[k]) == 1 - and "vocab:maiden-marker" in tokens[pieces[k][0]].tags), - None) + m = next((k for k in range(1, len(pieces)) + if _is_maiden_marker_piece(pieces[k], tokens)), + None) if m is not None: j = m + 1 consumed: list[int] = [] diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 4d551546..7e0d49c4 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -72,7 +72,8 @@ def _retag(tokens: list[WorkToken], i: int, role: Role) -> None: # rules.md#P2: "a particle joins the words after it into one name # part, the join running until the next particle starts a group of -# its own or the name ends" +# its own, a trailing suffix begins, a maiden marker takes the words +# after it (M2), or the name ends" # rules.md#P3: "the joined part is ONE name word wherever another # rule counts them" # rules.md#P5: "a recognized bound given-name word joins the word diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 41e5dc6b..17eb6a4e 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -671,6 +671,220 @@ def __post_init__(self) -> None: {"given": "Jane", "family": "Smith", "maiden": "Jones"}, classification="fix(#274)", notes="v1 mangles to middle='Smith née'"), + Case("maiden_marker_after_particle_chain", + "Ursula von der Leyen geb. Albrecht", + {"given": "Ursula", "family": "von der Leyen", + "maiden": "Albrecht"}, + classification="fix(#399)", + notes="#399: the chain that joins 'von der' to Leyen used to " + "run on past the marker and take the maiden name with " + "it (family 'von der Leyen geb. Albrecht', maiden ''), " + "because the marker is consumed AFTER the chain merges " + "and by then there is no lone marker piece left to " + "find. A suffix already stopped the chain; a marker now " + "does too. The same words one chain apart -- " + "maiden_marker_no_particle below is the control"), + Case("maiden_marker_no_particle", "Ursula Leyen geb. Albrecht", + {"given": "Ursula", "family": "Leyen", "maiden": "Albrecht"}, + classification="fix(#274)", + notes="the control for maiden_marker_after_particle_chain: " + "identical but for the particles, and it always worked. " + "Pinned so a regression in the plain path cannot hide " + "behind the particle rows"), + Case("maiden_marker_after_one_particle", "Anna von Müller geb. Schmidt", + {"given": "Anna", "family": "von Müller", "maiden": "Schmidt"}, + classification="fix(#399)", + notes="one particle is enough to break it -- #399 is not " + "about the length of the run"), + Case("maiden_marker_after_leading_particle", "von Müller geb. Schmidt", + {"given": "von", "family": "Müller", "maiden": "Schmidt"}, + classification="fix(#274)", + ambiguities=("particle-or-given",), + notes="the second control for #399, and the one that shows " + "where the old boundary fell: a LEADING particle chains " + "nothing (P4), so it never reached the marker and this " + "shape always worked. given 'von' is P4 plus P1 " + "declining to fold ('von' is outside the never-given " + "particles), which is also why the row reports the " + "fork -- not R2, whose output here is the family " + "('Müller', base 'Müller', particles '')"), + Case("maiden_marker_after_leading_particle_run", + "von der Müller geb. Schmidt", + {"given": "von", "family": "der Müller", "maiden": "Schmidt"}, + classification="fix(#399)", + ambiguities=("particle-or-given",), + notes="a leading RUN of two broke where a leading single did " + "not: 'der' is not the leading piece, so its own chain " + "fired and swallowed the marker. What #399 left alone " + "is the given/family SPLIT -- given 'von', family " + "starting at 'der' (P4 + R2); the family itself shed " + "the marker and the maiden name it had swallowed " + "('der Müller geb. Schmidt' -> 'der Müller')"), + Case("maiden_marker_after_particle_chain_with_suffix", + "Jane van der Berg née Jones PhD", + {"given": "Jane", "family": "van der Berg", "maiden": "Jones", + "suffix": "PhD"}, + classification="fix(#399)", + notes="marker and suffix stops in the same name: the chain " + "stops at the marker, then M2's own walk stops the " + "maiden name at the suffix. Dutch spelling of the same " + "defect, and 'née' unaccented-vs-accented is not the " + "variable here"), + Case("maiden_marker_stops_the_leading_run_family_first", + "de la Cruz née Vega", + {"family": "de la Cruz", "maiden": "Vega"}, + policy=Policy(name_order=FAMILY_FIRST), + classification="fix(#399)", + notes="#399's open question, answered by the chain stop " + "rather than by a rule of its own: the marker used to " + "survive as an ordinary name word and compete for the " + "leftover given slot, so this read given 'née' / middle " + "'Vega'. Consumed and dropped, it never reaches the " + "placement. No given name at all is the right answer " + "for family-plus-maiden input"), + Case("maiden_marker_leaves_family_all_particles", + "Jane de la née Jones", + {"given": "Jane", "family": "de la", "maiden": "Jones"}, + classification="fix(#399)", + notes="stopping the chain can leave a family group that is " + "wholly particles, which is exactly the shape R2 " + "reserves: they are not in particle position, so they " + "report as ordinary words -- family_base 'de la', " + "family_particles ''. test_cases pins that split only " + "up to the partition invariant (non-empty base, words " + "conserved), so base 'la' / particles 'de' would also " + "satisfy it. Before #399 this read family 'de la née " + "Jones' with base 'née Jones'. Nonsense input either " + "way; pinned " + "because the two rules have to compose without " + "either producing an empty base"), + Case("maiden_marker_trailing_after_particles", + "Jane van der Berg née", + {"given": "Jane", "family": "van der Berg née"}, + notes="the boundary the #399 stop is GATED on. M2 says a " + "marker with nothing after it is just a word, so the " + "chain must not stop here: there is no maiden name to " + "hand it to, and a stop would leave the marker as a " + "piece of its own, which then takes the whole family " + "field and demotes the real surname to the middle " + "(family 'née', middle 'van der Berg'). The first cut " + "of #399 did exactly that -- the marker-less spelling " + "'Jones née' -> family 'née' is M2's own boundary " + "example, and the particle spelling has to agree with " + "the rest of the family name, not replace it"), + Case("maiden_marker_trailing_before_suffix", + "Jane van der Berg née PhD", + {"given": "Jane", "family": "van der Berg née", + "suffix": "PhD"}, + notes="the other half of the gate: the consumer walks only up " + "to a trailing suffix, so a marker with nothing but a " + "suffix after it is not consumed either. Pinned " + "separately from the row above because the two reach " + "the same decision through different conditions -- last " + "piece vs nothing-but-suffix-follows -- and a stop " + "keyed on only one of them looks correct on the other"), + Case("maiden_marker_surname_spelling_keeps_its_particles", + "Jane van der Nee", + {"given": "Jane", "family": "van der Nee"}, + notes="'Nee' is an attested surname as well as a marker " + "spelling, which M1 already says ('a one-word clause " + "keeps its word, which may itself be a surname'). " + "Because nothing follows it the gate declines, so a " + "Dutch bearer of the surname keeps the tussenvoegsel " + "in the family name. Ungated, #399 moved 'van der' out " + "to the middle and left family 'Nee'"), + Case("maiden_marker_trailing_keeps_the_fork_report", + "St St née", + {"title": "St", "family": "St née"}, + ambiguities=("particle-or-given",), + notes="'st' is both a title and an ambiguous particle (#367), " + "so this shape reaches group's PARTICLE_OR_GIVEN " + "emitter, which is guarded on the chain having merged " + "something. An ungated marker stop made the chain merge " + "nothing for a DIFFERENT reason than the guard assumes, " + "silencing the report while still deciding the fork -- " + "the shape A1 forbids and #405 tracks. Pinned because " + "removing a report a caller already sees is worse than " + "never emitting one"), + Case("maiden_marker_particles_on_both_sides", + "Anna von der Müller geb. von der Berg", + {"given": "Anna", "family": "von der Müller", + "maiden": "von der Berg"}, + classification="fix(#399)", + notes="a particle chain on each side of the marker. This is " + "the shape decisions.md#M2 leans on when it explains " + "why #399 stopped the chain instead of moving the " + "maiden handler ahead of it, so it is pinned where the " + "argument can be checked. Before #399 the marker rode " + "into the middle name (middle 'von der Müller geb.')"), + Case("maiden_marker_stops_the_leading_run", "de la Cruz née Vega", + {"family": "de la Cruz", "maiden": "Vega"}, + classification="fix(#399)", + notes="the default-order reading of the family-first row " + "below, added because that one is core-only and the " + "facade runner would otherwise never see this class. " + "Before #399: family 'de la Cruz née Vega'"), + Case("maiden_marker_stops_the_leading_run_family_first_given_last", + "de la Cruz née Vega", + {"family": "de la Cruz", "maiden": "Vega"}, + policy=Policy(name_order=FAMILY_FIRST_GIVEN_LAST), + classification="fix(#399)", + notes="the sibling of the family-first row, and the point is " + "that the two orders now AGREE: consuming the marker " + "leaves no leftover to distribute, so the reading that " + "distinguishes them has nothing to work on. Before " + "#399 they differed -- given 'Vega' middle 'née' here " + "against given 'née' middle 'Vega' under FAMILY_FIRST"), + Case("maiden_marker_swallowed_by_a_conjunction_join", + "Jane van der Berg née y Jones", + {"given": "Jane", "family": "van der Berg née y Jones"}, + notes="accepted, and the reason #399's stop does not reach " + "it: the stop tests for a LONE marker piece, and P3's " + "connective join runs earlier in the same stage and " + "merges the marker into a multi-word piece. #399 fixed " + "the P2 instance of the join-swallow; this is one of " + "the two that survive"), + Case("maiden_marker_swallowed_by_a_bound_given_join", + "van der Berg, abdul née Jones", + {"given": "abdul née", "middle": "Jones", + "family": "van der Berg"}, + notes="the other surviving join-swallow, through P5's " + "bound-given join rather than P3's -- the more " + "plausible of the two, a post-comma given side. Same " + "cause as the row above: the marker is inside a merged " + "piece before the stop can see it"), + Case("maiden_marker_inside_a_conjunction_piece", + "Jane née and Jones Smith", + {"given": "Jane", "middle": "née and Jones", + "family": "Smith"}, + notes="pins the lone-piece guard itself. Dropping " + "`len(piece) == 1` from the marker test leaves the " + "whole suite green otherwise, yet it is live: this " + "connective merge produces a marker-HEADED multi-word " + "piece, and without the guard the marker matches and " + "carries 'Smith' into the maiden field with the family " + "left empty. #399 gave that guard a second caller, so " + "it needed a pin"), + Case("maiden_marker_after_particles_in_a_comma_segment", + "Smith, Jane van der Berg née Jones", + {"given": "Jane", "middle": "van der Berg", + "family": "Smith", "maiden": "Jones"}, + classification="fix(#399)", + notes="the listing form, where the chain and the marker are " + "both on the given side of the comma. Before #399 the " + "marker and the maiden name stayed in the middle name " + "('van der Berg née Jones'). Distinct from M2's " + "remaining Accepted note, which is about a marker " + "standing straight AFTER the comma"), + Case("maiden_marker_after_particles_cross_script", + "Jane van der Berg 旧姓 Jones", + {"given": "Jane", "family": "van der Berg", + "maiden": "Jones"}, + classification="fix(#399)", + notes="the stop is one vocabulary lookup, so it reaches every " + "marker spelling; this is the only cross-script pair " + "of the set -- a native-script marker between a Latin " + "tussenvoegsel and a Latin maiden name"), Case("maiden_marker_kyusei", "山田花子 旧姓 佐藤", {"family": "山田花子", "maiden": "佐藤"}, classification="fix(#309)", diff --git a/tests/v2/pipeline/test_group.py b/tests/v2/pipeline/test_group.py index e892139c..5996b494 100644 --- a/tests/v2/pipeline/test_group.py +++ b/tests/v2/pipeline/test_group.py @@ -323,3 +323,65 @@ def test_the_reserve_still_declines_when_only_a_suffix_is_left() -> None: # name with no family at all. out = _grouped("dual Allah jr", lexicon=_DUAL_LEX) assert _piece_texts(out) == [["dual", "Allah", "jr"]] + + +def test_a_chain_never_leaves_a_marker_standing_alone() -> None: + """The #399 stop and the maiden consumer must agree at the PIECE + level, which is where they can disagree without any field looking + wrong enough to fail. + + Both halves ask "is this a marker piece?" through one predicate, so + they cannot disagree about that. What they can disagree about is + whether the consumer TAKES: it needs a non-suffix piece after the + marker. Stopping the chain without one left the marker standing as + its own piece, and a lone trailing piece takes a role field -- the + marker became the family name and the real surname was demoted to + the middle. Parsed fields alone do not catch it: misaligning the + two halves leaves the whole suite green, because every field still + holds something plausible. + + The invariant the gate buys, and it holds both ways round: a marker + standing as its own piece never follows a piece carrying a + particle. Either the consumer took the marker (and it is gone from + pieces altogether), or the chain never stopped and swallowed it. A + marker DOES stand alone where no chain reached it, which is M2's + own "Jones nee" -> family "nee" boundary, so the assertion is + keyed on the preceding piece rather than on markers as such. + """ + def particled(state: ParseState, piece: tuple[int, ...]) -> bool: + return any("particle" in state.tokens[i].tags for i in piece) + + for text in ("Jane van der Berg née Jones", # consumer takes + "Jane van der Berg née", # nothing follows + "Jane van der Berg née Jr", # only a suffix + "Jane van der née", # particles only + "Jane Smith née Jones", # no particle: takes + "Jane Smith née", # no particle: stands + "Jane née"): + out = _grouped(text) + for seg, texts in zip(out.pieces, _piece_texts(out)): + for k, (piece, shown) in enumerate(zip(seg, texts)): + if shown.lower() not in _LEX.maiden_markers or k == 0: + continue + assert not particled(out, seg[k - 1]), ( + f"{text!r}: marker {shown!r} left standing after " + f"the particle piece {texts[k - 1]!r}") + + +def test_where_the_marker_lands_when_the_consumer_declines() -> None: + """The concrete shapes behind the invariant above, so a change that + preserves it by restructuring the pieces still has to say so here. + """ + # consumer takes: marker and maiden name leave `pieces` entirely + assert _piece_texts(_grouped("Jane van der Berg née Jones")) == [ + ["Jane", "van der Berg"]] + assert _piece_texts(_grouped("Jane Smith née Jones")) == [ + ["Jane", "Smith"]] + # consumer declines, chain present: the marker rides inside it + assert _piece_texts(_grouped("Jane van der Berg née")) == [ + ["Jane", "van der Berg née"]] + assert _piece_texts(_grouped("Jane van der Berg née Jr")) == [ + ["Jane", "van der Berg née", "Jr"]] + # consumer declines, no chain: the marker stands as its own piece + assert _piece_texts(_grouped("Jane Smith née")) == [ + ["Jane", "Smith", "née"]] diff --git a/tests/v2/test_facade_cases.py b/tests/v2/test_facade_cases.py index b647ef46..1174bfce 100644 --- a/tests/v2/test_facade_cases.py +++ b/tests/v2/test_facade_cases.py @@ -74,6 +74,8 @@ "leading_never_given_particle_two_leftovers_family_first", "leading_never_given_particle_two_leftovers_family_first_given_last", "maiden_marker_delimited_beside_a_nickname_clause", + "maiden_marker_stops_the_leading_run_family_first", + "maiden_marker_stops_the_leading_run_family_first_given_last", "maiden_marker_kyusei_delimited", "ko_honorific_period_under_strict_comma_suffixes", "ja_honorific_glued_family_comma_strict_knob", diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index e610e803..4066e4bc 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -690,6 +690,18 @@ class _LatinCopy(NamedTuple): covers=frozenset({"de", "del", "den", "der", "di", "do", "dos", "du", "la", "le", "los", "mc", "van", "vd", "von", "zu"})), + # The same partial copy as fix(#379) above, and the same reason for + # being partial -- these are the words that actually precede a base + # in a Latin listing. Sharing the snapshot is deliberate: the two + # rules ask different questions of the same vocabulary (#379 wants + # the word that ENDS a comma listing, #399 the word that opens a + # chain), so a divergence between the two `covers` sets would be a + # fact about one of the rules, not a copy drifting. + "fix(#399)": _LatinCopy( + vocabulary=PARTICLES, + covers=frozenset({"de", "del", "den", "der", "di", "do", "dos", + "du", "la", "le", "los", "mc", "van", "vd", + "von", "zu"})), } #: Alternations that copy no vocabulary, so discovery must not demand a @@ -1022,11 +1034,11 @@ def _claim(rule: dict) -> _Claim: _CORPUS_CLAIMS: dict[str, dict[str, _Claim]] = { "expected_since_1.4.0.toml": { "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": - _Claim(97, ('family', 'given', 'middle'), "66e71d60a075"), + _Claim(98, ('family', 'given', 'middle'), "54f0cf4eb210"), "fix(#274) maiden markers consumed": _Claim(4, ('family', 'maiden', 'middle'), "b31dc2e2bbc4"), "fix(cjk-maiden-marker) maiden marker consumed, compounding with the CJK order flip": - _Claim(3, ('family', 'given', 'maiden', 'middle'), "cf5c9d671c14"), + _Claim(4, ('family', 'given', 'maiden', 'middle'), "ca6adbb3fa69"), "fix(#379) a tussenvoegsel after a family comma attaches to the family": _Claim(2, ('family', 'middle'), "f3a43bebfb91"), "fix(comma-family) lone post-comma piece routes to suffix/title, not first": @@ -1034,7 +1046,7 @@ def _claim(rule: dict) -> _Claim: "fix(comma-precomma-family) pre-comma run reads as family, not given": _Claim(215, ('family', 'given'), "f16a0e79cba3"), "fix(suffix-routing) two-token name with unambiguous trailing suffix stays suffix": - _Claim(751, ('family', 'given', 'suffix'), "231640fc7535"), + _Claim(752, ('family', 'given', 'suffix'), "92af19376071"), "fix(suffix-delimiter-rendering) no-space delimiter core token kept whole": _Claim(0, ('suffix',), "e3b0c44298fc"), "ambiguous-surname-acronym data change: parenthesized (MA)/(DO) now stays nickname": @@ -1066,13 +1078,13 @@ def _claim(rule: dict) -> _Claim: "fix(#379) a tussenvoegsel after a family comma attaches to the family": _Claim(2, ('family', 'middle'), "f3a43bebfb91"), "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": - _Claim(97, ('_ambiguities', 'family', 'given', 'middle'), "66e71d60a075"), + _Claim(98, ('_ambiguities', 'family', 'given', 'middle'), "54f0cf4eb210"), "fix(#308/#312/#319/#320) glued CJK honorific peeled off the name into suffix": _Claim(34, ('family', 'given', 'suffix'), "877ab3246d33"), "fix(#307/#308/#320) spaced CJK postnominal honorific routed to suffix": _Claim(16, ('family', 'given', 'middle', 'suffix'), "6d390e518bd2"), "fix(#309) 旧姓 maiden marker consumed, compounding with the CJK order flip": - _Claim(3, ('family', 'given', 'maiden', 'middle'), "cf5c9d671c14"), + _Claim(4, ('family', 'given', 'maiden', 'middle'), "ca6adbb3fa69"), "fix(#272) nakaguro inside delimited content renders as a space, compounding with the CJK order flip": _Claim(1, ('family', 'given', 'nickname'), "d4069d459f23"), "fix(#298) 间隔号 division changes the comma reading, sending the credential from title to suffix": @@ -1081,6 +1093,14 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('family', 'given', 'middle'), "dce0ae6df4be"), }, "expected_since_2.1.0.toml": { + # One name, and the rule is written to reach exactly that one: + # the particle-ahead-of-a-marker shape is what #399 fixed, and + # the four other marker-bearing corpus names have no particle + # in front of the marker. A count that grows here means either + # a new corpus name of the shape (record it) or the rule + # loosening (do not). + "fix(#399) a maiden marker bounds the particle chain that swallowed it": + _Claim(1, ('family', 'maiden'), "f016cc61ca43"), "fix(#379) a tussenvoegsel after a family comma attaches to the family": _Claim(2, ('family', 'middle'), "f3a43bebfb91"), "fix(#367) a title no longer displaces a leading particle out of the leading position": diff --git a/tools/differential/corpus_cjk.jsonl b/tools/differential/corpus_cjk.jsonl index 34f198e9..681403ac 100644 --- a/tools/differential/corpus_cjk.jsonl +++ b/tools/differential/corpus_cjk.jsonl @@ -3,6 +3,7 @@ "Dr 김민준, Jr." "Dr 김민준씨, Jr." "Dr 김민준씨, V." +"Jane van der Berg 旧姓 Jones" "John 王" "〆木 ひろ" "〆木 太郎" diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index 875c5f1d..339e7822 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -95,6 +95,47 @@ issue = "fix(#367) a title no longer displaces a leading particle out of the lea name_regex = "(?i)^mr\\.\\s+van\\b" fields = ["given", "middle", "family"] +[[change]] +issue = "fix(#399) a maiden marker bounds the particle chain that swallowed it" +# 'Jane van der Berg 旧姓 Jones': grouping's particle chain absorbed +# everything up to the next particle or suffix piece, and the maiden +# marker is consumed later in the SAME stage -- by then the chain had +# merged the marker away and there was no lone marker piece left to +# find, so the family kept both the marker and the maiden name. +# rules.md#M2 states the stop; decisions.md#M2 records it. +# +# Fields are {family, maiden}: the given name does not move, and no +# corpus name puts this shape under a non-default name_order. +# +# The rule requires a PARTICLE ahead of the marker because that IS the +# defect -- a marker with no particle in front of it was always consumed +# correctly, and the four other marker-bearing corpus names are all of +# that kind, so this claims exactly the one name it was read from. +# +# The trailing `\s+\S` is load-bearing, not padding: #399's stop is +# gated on the maiden consumer actually taking, and the consumer needs a +# non-suffix word after the marker. A marker with nothing after it is +# left alone as an ordinary name word (M2: "a marker with nothing after +# it ... is just a word"), so 'Smith nee' and 'Muller geb.' do NOT move +# and this rule must not claim them. Ungated, the stop stranded the +# marker as a piece of its own and it took the whole family field. +# +# Not Latin-anchored, unlike its #379 neighbour, and deliberately: this +# name is the one shape in the corpus that is Latin either side of a +# native-script marker, so an anchor over the whole name would fail to +# match the very diff the rule exists to explain. +# +# The marker is the single literal the corpus name carries, not an +# alternation over MAIDEN_MARKERS, for two reasons. The cause is +# script-independent and the case table proves it over every spelling, +# so breadth here would buy nothing but reach; and the ledger guards +# pin one hand-copied vocabulary per rule, which this rule has already +# spent on PARTICLES. A Latin-marker name of this shape entering the +# corpus should arrive UNEXPLAINED and be read once, the discipline +# fix(#379) states for its own vd-shaped gap. +name_regex = "(?i)\\b(de|del|den|der|di|do|dos|du|la|le|los|mc|van|vd|von|zu)\\b.*?\\s\u65e7\u59d3\\s+\\S" +fields = ["family", "maiden"] + [[change]] issue = "fix(#379) a tussenvoegsel after a family comma attaches to the family" # 'Vega, Juan de la': the Dutch alphabetized listing moves the