diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 4a3c768a..3cbaeb73 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -78,6 +78,8 @@ Open: [#360](https://github.com/derek73/python-nameparser/issues/360) which part - Group distribution is load-bearing, not incidental: the v1 Portuguese tests (test_portuguese_dos, test_portuguese_prefixes) require multiple particle groups to land across middle and family, while #132 wanted the combined double-surname reading — the same shape with opposing wants, which is why the combined reading lives in the surnames VIEW and the split in the fields. +- 2026-08-22 #424 — the chain stops where assign's trailing peel begins. The chain ran "until a trailing suffix begins" and asked with the suffix-piece test, which vetoes a bare `V` as an initial — the #401 question at a third site — so `John van der Berg V` read family 'van der Berg V' (1.4.0 read it so too: shipped since 1.x), and `John van der Berg Ma` read family 'van der Berg Ma' where 1.4.0 read suffix 'Ma' — a 2.0 regression on S2's other fork. Both now stop where `_trailing_start` says the run begins: the S2 peel (`_peel_trailing` over `_peel_walk`, shared with assign and P5's reserve since #425) read over the pieces as they stand from the first name piece, once per segment, kept as a length from the end that the chain's merges ahead of it do not move (the benchmark's `particles` shape is the guard). The chain takes both forks — its merges leave the acronym at least the three pieces the fork counted, so assign peels it after as the walk did before — where the maiden walk takes only the numeral (the M2 entry says why). The fork reads the piece before the numeral as it stands, so `John van der J. V` keeps family 'van der J. V', as `John J. V` reads the V as a name; and a numeral with a suffix behind it is not last in the walk, so `John van der Berg V Jr` keeps family 'van der Berg V'. No differential corpus name had either shape; the rules examples carry them — the numeral against every baseline, the acronym against 2.0.0 and 2.1.0. Two more sites were asking assign's question with a test of their own, both found by the reviews of this change. The walk's starting point and P4's leading-particle scan asked group's `title()`, which does not see H2's unlisted abbreviations — assign peels those as titles all the same — so `Xyz. van Johnson` chained where `Dr. van Johnson` did not (#367 had keyed the exception on the first piece of the name, by group's test), and `Xyz. van Berg MA` read family 'van Berg MA' on every 2.x tree — the chain had taken the acronym, as it took every acronym; the draft that stopped before it left assign two pieces where the fork had counted three, and MA became the family. Both ask assign's own test now — `_is_leading_title` and the period-abbreviation pattern moved down into group, with `_leading_titles` as the one definition of where the name begins: the chain's walk, the bound join's first name piece (the second docs review found that one still on group's test — `Xyz. abdul John Smith` joined nothing where `Dr. abdul John Smith` read given 'abdul John') and, through `_is_leading_title`, the scan and the PARTICLE_OR_GIVEN report — so `Xyz. van …` reads as `Dr. van …` does. `Esq. van Gogh`, a corpus name, moves with it (1.4.0 read family 'van Gogh' for both, as #367's rule records; its own rule at every baseline, and one for the P4 example). A particle of the unambiguous suffix vocabulary too (vd, mc) is a suffix piece to the peel: where it opens the trailing run the chain stops before it as before any suffix word and the peel takes it — `John Smith Mc V` reads suffix 'Mc, V' where master and 1.4.0 read family 'Mc V', a class the second docs review found unrecorded (2,554 of its 117,306 constructed names, an eighteen-word pool that puts the words in every position) — and where it continues a prefix run, the run takes it as a particle, so `John van Mc` keeps family 'van Mc', every baseline's reading and the one P6 chose for the shape after a comma. Both are Accepted under P2. The fork's count was the last thing the chain asked on its own authority, and the reviews found it wrong behind a title-and-particle word: #367 stops the leading-particle scan at `Freiherr`, `St`, `Do` — the name's own first piece — where assign's title peel steps over them, so the chain takes the name's first word, and the acronym the fork counted with three pieces met assign with two: `Freiherr von Berg MA` read given 'von Berg', family 'MA' (1.4.0's reading, as it happens; master read family 'von Berg MA'). So the chain asks the peel again over the pieces it leaves — `chain(tail)`, then `_trailing_start` over what it built, and a second run without the stop where the verdict changed, the snapshot being one copy per segment with a trailing run — and takes what assign will not peel: family 'von Berg MA'. The numeral cannot flip, a chain group never being initial-shaped, so `Freiherr von Richthofen V` keeps suffix 'V', and its chain, the one name piece left, reads as `Dr. Smith V` reads — given 'von Richthofen', no family, where 1.4.0 and master read family 'von Richthofen V'; the code review asked for that boundary to be on record, and P2 carries it. + ### P6 — the trailing orphan particle - 2026-08-18 #379/#380 IMPLEMENTED — the rule shipped as stated, both `deviates:` markers out. Keyed on the token's VOCABULARY rather than its assigned role, which is what gives the S2 precedence its effect: assign reads a trailing `vd`/`mc`/`do` as a post-nominal, and the attachment overrides that after a comma. The words-to-spare guard is a PIECE test, not a count — every trailing piece that is wholly particles attaches, and the run must leave a GIVEN word ahead of it, so "Nguyen, Van" keeps its only given word. Not "a name word": see the defect below, where that wider reading passed on family text P1 had just produced. MIDDLE was in the test briefly and came out — 0 hits over 740,552 instrumented guard sites, and structurally unreachable, since the only rule that leaves a MIDDLE with no GIVEN ahead of it is gated on state.order, which assign never records on the family-comma path P6 runs on. @@ -105,6 +107,8 @@ Open: [#360](https://github.com/derek73/python-nameparser/issues/360) which part Open: [#380](https://github.com/derek73/python-nameparser/issues/380) covers "Berg, Jan vd" under this rule, but the vd reading itself is decisions.md#vocabulary-collisions (C-ii); and the no-given-word case "Jong, vd" is deliberately unresolved — see the scope note. +- 2026-08-22 (late review of #425's docs, PR #426) — the bound join runs before P6 sees the name, and P6 now says so. P5's new title-word clause took `do`, `freiherr` and `st` — title AND particle vocabulary — away from the attachment: `Berg, abdul do` had read family 'do Berg' since #379 and reads given 'abdul do' now. That is 1.4.0's reading, and it is what every other particle already did: `Berg, abdul van` read given 'abdul van' at 1.4.0, on 2.0 and 2.1, and on master before #426 — the old reserve's title exclusion had held only the title-word trio back. The precedence is the stage order (P5 joins in group; P6 attaches in post_rules, over what is left) rather than a judgement about tussenvoegsels behind bound words, and both rules carry it as Accepted with `Berg, abdul van` as the example, parity at every baseline. The precedence has one exception, found by the review of #424's docs when the Accepted clause claimed the attachment "never sees" the particle: a trailing particle of the unambiguous suffix vocabulary too — `vd`, `mc`; `do` is ambiguous, and a name word to the join — is a suffix piece to the join, which never takes one (#421), so the attachment does see it, and `Berg, abdul vd` reads family 'vd Berg' where 1.4.0 read given 'abdul vd'. Both rules say so, with the example; at 1.4.0 the fix(comma-precomma-family) rule already reaches it (a rule written for it was shadowed, and dropped). + ### M2 — the maiden-marker rule - 2026-07-03 (maiden-bucket design; #274 filed 2026-07-07, landed in the v2 core, PR #288) — the marker takes everything after it up to a trailing suffix, greedily: "née Jones Smith" is a two-word maiden name, matching how the marker is actually used in running text. The marker itself is dropped as structural, like a delimiter character. @@ -135,6 +139,8 @@ the fullwidth-colon marker (旧姓：佐藤 arrives as one word; the head-peel q [#410](https://github.com/derek73/python-nameparser/issues/410) a maiden name makes H1 decline, so a title-plus-surname name reports no family — pre-existing, and widened by the stop. (#411, the same empty family from P5's reserve, closed 2026-08-21.) +- 2026-08-22 #424 — the walk stops before the trailing numeral assign reads as the suffix. "Up to any trailing suffix" was asked with the suffix-piece test, whose initial veto does not see a bare `V`: `John née Jones Smith V` read maiden 'Jones Smith V' where `John Smith V` reads suffix 'V'. The walk now also stops at `_trailing_start` — the S2 peel read from the marker on, over the pieces as they stand, stepping over a tail segment's delimiter cores as the walk does — but at the NUMERAL only. The bare-acronym fork counts pieces, and the walk removes the very pieces it counted: `John née Jones Smith Ma` peeled over the pieces as they stand reads the acronym as a credential with words to spare, and once 'Jones Smith' has left the name it is the family of a two-piece name — maiden 'Jones Smith', family 'Ma', which is worse than the maiden 'Jones Smith Ma' it had always read. The numeral fork reads the piece before the numeral, and the take removes that very piece: afterwards assign sees the piece before the MARKER there, and where that is initial-shaped the fork does not fire — the first draft stopped the walk all the same, and `J. née Jones Smith V` read maiden 'Jones Smith', family 'V' (both reviews found it: 27 of the design-docs review's 144 marker shapes, every one a head ending in an initial; the pool is unrecorded). The walk runs the peel again over the view the take would leave — the pieces before the marker and the numeral on, built as P5's reserve builds its view — so it stops only where the numeral reads as the suffix both as written and as left: `J. née Jones Smith V` keeps maiden 'Jones Smith V', as `J. V` reads the V as a name, and `Dr. née Jones Smith V` keeps maiden 'Jones Smith V', since assign peels the title first and a numeral alone is no fork (the first re-ask checked the preceding piece only and handed that V to the given name — the code review). As written too: `Jane née Jones J. V` keeps maiden 'Jones J. V' though the take would leave `Jane V`, the words as written being what the marker takes; the second docs review found the statement saying only the second half. The acronym is left to assign, recorded as Accepted. The peel is read from the marker rather than after it, so `Jane Smith née V` declines as `Jane Smith née PhD` does — nothing after the marker but a suffix, the marker stays a word, suffix 'V' — which is 1.4.0's reading (the first draft read maiden 'V', and the 1.4.0 gate is what showed it); and `Jane née Jones J. V` keeps maiden 'Jones J. V' (the piece before is initial-shaped). 1.4.0 had no maiden support, so the examples fall under the #274 rule at that baseline where its fields reach, and `J. née Jones Smith V` (v1 read suffix 'V') under an accepted rule of its own; against 2.0.0 and 2.1.0 the examples that move are classified — `John née Jones Smith V` in maiden and suffix, `Jane Smith née V` in the five fields the marker's survival touches — and `J. née Jones Smith V` is 2.x parity. + ### N3 — the lone-word nickname rule - 2026-07 (v2 core, PR #288; recorded plan deviation #2 of the core plan) — v1's rule counted pieces before grouping; the v2 port fires only when the nickname accompanies exactly ONE piece in total — a title counts against it, which is why "'Smitty' Dr. Jones" reads given="Jones" with the family empty (rules.md#N3's Accepted consequence) rather than family="Jones". The rule lives in assignment because that is where the piece count is settled. (An earlier wording here said "one non-title piece", predicting the opposite output; the coherence review measured the truth.) @@ -372,9 +378,9 @@ Excluded (SUFFIX_ACRONYMS / SUFFIX_WORDS — the esq dual membership, deliberate - 2026-08-22 #425 — the reserve runs assign's peel over the view the join would leave. #423 had the reserve re-derive assign's trailing peel by hand — the flagged-credential drop, the walk's start, the numeral fork's conditions, the preceding piece remapped to "as the join would leave it" — with each seam held by a pin, and the bare-ambiguous-acronym fork not mirrored at all: `abdul Smith Jr Ma` read given 'abdul Smith', family '', suffix 'Jr, Ma' where `John Smith Jr Ma` keeps family 'Smith', the whole residue of #401's shape on the main walk (every family-losing bound-word shape in the sweep below contains one). The altitude review's framing is taken: the peel is one function, `_peel_trailing` over the list `_peel_walk` builds (the walk's input defined once too, after the /simplify pass found it spelled at three sites), housed in group beside the piece predicates because assign imports those and group cannot import assign; assign calls it and reports what it returns (the numeral fork's piece, the acronym picks, in the order it always reported them — byte-identical over the suite and the three gates); and the reserve builds the VIEW merge() would leave — the pair one piece, its tags the union — and calls the same function. `BoundJoin`'s "the value IS the threshold" trick survives restated as post-join name pieces: STRICT 2, LENIENT 1. The remap and the `k == first_name_k` carve-out are gone, the joined piece being two tokens and never a suffix piece. - One principle came out of writing the ledger that the view alone did not state. `abdul Smith Ma` had read given 'abdul Smith', family 'Ma' since 2.0, and the #423 entry called that the lossless shape; the 1.4.0 gate's catch-all count said otherwise — 1.4.0 reads first 'abdul', last 'Smith', suffix 'Ma', as `John Smith Ma` reads, and the fields-only catch-all had absorbed the diff since the example went in (the count is the tell, per the #401 entry). Unjoined, the acronym has words to spare and peels as a credential; joined, the view is two pieces and the fork keeps it as the family — the join was CHANGING what assign reads. So the rule is stated as P5 now has it: the join joins two name words into one and changes nothing else, which the code asks directly: the pieces the peel takes are the same either way (a count proxy, "the view leaves exactly one name word fewer", was the first spelling; the altitude review pointed out it held only because the peel is a tail walk, and the comparison states the rule). That comparison subsumes the unjoined-piece check #423 added for `abdul V` (unjoined the peel takes the V, joined nothing) and restores `abdul Smith Ma` to 1.4.0's reading; `abdul Smith Berg Ma` keeps its join, the acronym peeling either way. Every #423 pin holds. Measured: the 1.4.0 gate is byte-identical at the code change, before the examples went in — the three `Ma`-final examples are v1 parity, the catch-all back at 15 — and `abdul Smith Jr Ma` and `abdul Smith Ma` move against 2.0.0 and 2.1.0, classified there; no hand corpus has the shapes. The population the corpus cannot see, from the design-docs review's sweep of 67,860 constructed main-walk shapes (three bound words × twelve words, none a title word, × tails of one to four; no comma shapes): before this change 2,160 lost the family and every one contained a bare ambiguous acronym (864 with further suffixes behind it, so "ending in" was an artefact of #423's shorter sweep); after it none does, and all 67,860 match the family and suffix their ordinary-given twin reads. 5,166 move against master in all — 3,006 where the acronym was the family and is now the suffix (`Sir abdul Smith Ma`), 2,160 where the family is restored. The title-word class below was outside that population and moves after a family comma too (`Berg, abdul Sir`); so was the licence class at the end of this entry. + One principle came out of writing the ledger that the view alone did not state. `abdul Smith Ma` had read given 'abdul Smith', family 'Ma' since 2.0, and the #423 entry called that the lossless shape; the 1.4.0 gate's catch-all count said otherwise — 1.4.0 reads first 'abdul', last 'Smith', suffix 'Ma', as `John Smith Ma` reads, and the fields-only catch-all had absorbed the diff since the example went in (the count is the tell, per the #401 entry). Unjoined, the acronym has words to spare and peels as a credential; joined, the view is two pieces and the fork keeps it as the family — the join was CHANGING what assign reads. So the rule is stated as P5 now has it: the join joins two name words into one and changes nothing else, which the code asks directly: the pieces the peel takes are the same either way (a count proxy, "the view leaves exactly one name word fewer", was the first spelling; the altitude review pointed out it held only because the peel is a tail walk, and the comparison states the rule). That comparison subsumes the unjoined-piece check #423 added for `abdul V` (unjoined the peel takes the V, joined nothing) and restores `abdul Smith Ma` to 1.4.0's reading; `abdul Smith Berg Ma` keeps its join, the acronym peeling either way. Every #423 pin holds. Measured: the 1.4.0 gate is byte-identical at the code change, before the examples went in — the three `Ma`-final examples are v1 parity, the catch-all back at 15 — and `abdul Smith Jr Ma` and `abdul Smith Ma` move against 2.0.0 and 2.1.0, classified there; no hand corpus has the shapes. The population the corpus cannot see, from the design-docs review's sweep of 67,860 constructed main-walk shapes (three bound words — abdul, abd, abu — × twelve words — Smith, Berg, Ed, J., Jr, M.A., Ma, PhD, V, X, 'Ph. D.', 'van der', none a title word — × tails of one to four; no comma shapes; a different twelve-word pool gives different counts, so the pool is the claim): before this change 2,160 lost the family and every one contained a bare ambiguous acronym (864 with further suffixes behind it, so "ending in" was an artefact of #423's shorter sweep); after it none does, and all 67,860 match the family and suffix their ordinary-given twin reads. 5,166 move against master in all — 3,006 where the acronym was the family and is now the suffix (`Sir abdul Smith Ma`), 2,160 where the family is restored. The title-word class below was outside that population and moves after a family comma too (`Berg, abdul Sir`); so was the licence class at the end of this entry. - Three things the principle does not reach, recorded so nobody applies it there. The post-comma LENIENT path is untouched, and NOT for the reason first written here ("its walk has no peel" — it reads a trailing numeral by the lenient last-of-two rule, as the #423 entry says): the family is fixed there and the pair is the given whatever follows, so the principle is not applied, and `Berg, abdul V` keeps given 'abdul V' though the V would be the suffix unjoined — 1.4.0 parity, and exactly the regression #423 measured when a mirror was applied there; P5 now carries it as Accepted. The vocabulary decline is the UNAMBIGUOUS vocabulary: a bare ambiguous acronym the peel does not take is a name word, so `abdul Ma Smith` reads given 'abdul Ma' (1.4.0 read middle 'Ma' — a 2.x difference no corpus witnesses, now an Accepted example). And the principle is about the suffix reading, not the report: `Sheik abdul Ma` reports no SUFFIX_OR_NAME where `Sheik John Ma` does, the #369 precedent — a bound word read as the bound word is not a fork. One more class moves, found by the code review and then measured against 1.4.0: the 2.x reserve had excluded TITLE pieces from its count — an unrecorded deviation from v1, whose join counted suffixes out and nothing else — and the shared peel, following assign, reads a mid-name title word as the name word it is. `Berg, abdul Sir` (given 'abdul', middle 'Sir' since 2.0) and `abdul Smith Sir` (declined since 2.0) read given 'abdul Sir' and given 'abdul Smith', family 'Sir' — 1.4.0's readings, both; `abdul Sir Smith Berg` never stopped reading given 'abdul Sir'. No corpus name has the shapes; P5 says it now and `abdul Sir Smith Berg` is an example. What the review actually caught was the tag artefact that exclusion had been hiding: the conjunction merge derives a `title` piece tag for a chain such as "Sheikh and Ahmad", merge()'s union handed it to the pair, and assign peeled the pair as a leading title — `abdul Sheikh and Ahmad Bakar Smith` read title 'abdul Sheikh and Ahmad' on 2.0 and 2.1 (v1 read it as the given), and `abdul Sheikh and Ahmad Bakar`, which the exclusion had kept unjoined, would have too once it went. The bound join drops the tag, and the view drops it the same way — the pair is a given name whatever tag the word carried. One more class the comparison moves, found by the code review's classification of every residual master-to-branch diff (12 of 17,640 constructed names, one shape; 360 of 360 in a dedicated sweep): behind a given-name title, a numeral after an initial-shaped piece. `Sir abdul J. V` read given 'abdul J.', suffix 'V' since #369 — the fork suppressed unjoined, fired joined, and the licence's threshold of one let the join through — and the comparison declines it: given 'abdul', middle 'J.', family 'V', exactly as `Sir John J. V`. Pinned at both levels; it was the only shape a looser comparison let through under mutation at the test review. #424 — P2's chain and M2's walk taking a trailing numeral — is the remaining asker of the same question and now has `_peel_trailing` to ask it of. + Three things the principle does not reach, recorded so nobody applies it there. The post-comma LENIENT path is untouched, and NOT for the reason first written here ("its walk has no peel" — it reads a trailing numeral by the lenient last-of-two rule, as the #423 entry says): the family is fixed there and the pair is the given whatever follows, so the principle is not applied, and `Berg, abdul V` keeps given 'abdul V' though the V would be the suffix unjoined — 1.4.0 parity, and exactly the regression #423 measured when a mirror was applied there; P5 now carries it as Accepted. The vocabulary decline is the UNAMBIGUOUS vocabulary: a bare ambiguous acronym the peel does not take is a name word, so `abdul Ma Smith` reads given 'abdul Ma' (1.4.0 read middle 'Ma' — a 2.x difference no corpus witnesses, now an Accepted example). And the principle is about the suffix reading, not the report: `Sheik abdul Ma` reports no SUFFIX_OR_NAME where `Sheik John Ma` does, the #369 precedent — a bound word read as the bound word is not a fork. One more class moves, found by the code review and then measured against 1.4.0: the 2.x reserve had excluded TITLE pieces from its count — an unrecorded deviation from v1, whose join counted suffixes out and nothing else — and the shared peel, following assign, reads a mid-name title word as the name word it is. `Berg, abdul Sir` (given 'abdul', middle 'Sir' since 2.0) and `abdul Smith Sir` (declined since 2.0) read given 'abdul Sir' and given 'abdul Smith', family 'Sir' — 1.4.0's readings, both; `abdul Sir Smith Berg` never stopped reading given 'abdul Sir'. No corpus name has the shapes that move, and no rules example carries one — `abdul Sir Smith Berg`, P5's example of the reading, is byte-identical at every baseline, so the class is pinned in tests alone (the release log says so; the late review of this entry found the sentence here implying otherwise). The same review found the clause had taken `do`, `freiherr` and `st` from P6 — recorded under P6. What the review actually caught was the tag artefact that exclusion had been hiding: the conjunction merge derives a `title` piece tag for a chain such as "Sheikh and Ahmad", merge()'s union handed it to the pair, and assign peeled the pair as a leading title — `abdul Sheikh and Ahmad Bakar Smith` read title 'abdul Sheikh and Ahmad' on 2.0 and 2.1 (v1 read it as the given), and `abdul Sheikh and Ahmad Bakar`, which the exclusion had kept unjoined, would have too once it went. The bound join drops the tag, and the view drops it the same way — the pair is a given name whatever tag the word carried. One more class the comparison moves, found by the code review's classification of every residual master-to-branch diff (12 of 17,640 constructed names — a fourteen-word pool, bare and behind `Sir`, tails of two and three, 6 × (14² + 14³); the review did not record the pool — one shape; 360 of 360 over {Sir, Sheikh, Dame, Sheik, King, Father} × {abdul, abd, abu, umm} × {J., V, X, A., I} × {V, X, I}): behind a given-name title, a numeral after an initial-shaped piece. `Sir abdul J. V` read given 'abdul J.', suffix 'V' since #369 — the fork suppressed unjoined, fired joined, and the licence's threshold of one let the join through — and the comparison declines it: given 'abdul', middle 'J.', family 'V', exactly as `Sir John J. V`. Pinned at both levels; it was the only shape a looser comparison let through under mutation at the test review. #424 — P2's chain and M2's walk taking a trailing numeral — is the remaining asker of the same question and now has `_peel_trailing` to ask it of. Excluded (BOUND_GIVEN_NAMES): diff --git a/docs/design/rules.md b/docs/design/rules.md index 8f6acd01..80b01799 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -69,7 +69,7 @@ H2. Rationale: before a name, an abbreviation is almost always a family name (C1), so no shape or vocabulary reading makes a title there. "Xyz. Smith, John" → family="Xyz. Smith" - history: decisions.md#H2 · interacts: C1 · implemented: nameparser/_pipeline/_assign.py + history: decisions.md#H2 · interacts: C1, P4 · implemented: nameparser/_pipeline/_assign.py, nameparser/_pipeline/_group.py H3. Rationale: compound titles are written as a run of title words, connectives included; a title word standing inside the name is @@ -157,9 +157,12 @@ P2. Rationale: a particle is written as part of the surname it precedes, and a title stands outside the name entirely. A particle joins the words after it into one name part, the join running until the next particle starts a group of its own, - a trailing suffix begins, a maiden marker takes the words after - it (M2), or the name ends. The final group reads as the family - name; + a trailing suffix begins — read as assign will read it (S2), + over the pieces the chain leaves: a trailing roman numeral, or a + bare acronym with words to spare, ends the chain as a suffix word + does — a maiden marker takes the + words after it (M2), or the name ends. The final group reads as + the family name; earlier groups read by position. The chain begins wherever the name begins, and a preceding title does not move that point. Where P1's fold has claimed the opening, the fold decides the @@ -172,12 +175,25 @@ P2. Rationale: a particle is written as part of the surname it "Juan de" → family="de" · boundary "de la Cruz Juan Carlos" family-first → family="de la Cruz" "John van der Berg PhD" → family="van der Berg" + "John van der Berg V" → family="van der Berg" + "John van der Berg V" → suffix="V" + "John van der Berg Ma" → suffix="Ma" + "John van der J. V" → family="van der J. V" · boundary + "Freiherr von Berg MA" → family="von Berg MA" + "Freiherr von Richthofen V" → suffix="V" · boundary "John van der Berg née Jones" → family="van der Berg" + Accepted: a particle of the unambiguous suffix vocabulary too + (vd, mc) is a suffix piece to the peel, so where it opens the + trailing run the chain stops before it as before any suffix + word, and the peel takes it; where it continues a prefix run, + the run takes it as a particle, as P6 reads it after a comma. + "John Smith Mc V" → suffix="Mc, V" + "John van Mc" → family="van Mc" Accepted: a caller wanting the combined double-surname reading (#132's ask) has it as the surnames view rather than the family field. "Vincent van Gogh van Beethoven" → surnames="van Gogh van Beethoven" - history: decisions.md#P2 · interacts: P1, P4, M2 · implemented: nameparser/_pipeline/_group.py, nameparser/_pipeline/_post_rules.py + history: decisions.md#P2 · interacts: P1, P4, M2, S2 · implemented: nameparser/_pipeline/_group.py, nameparser/_pipeline/_post_rules.py P3. Rationale: connective words ("y", "of the") bind name words into one name part; but a single letter in a short name is more @@ -226,11 +242,14 @@ P4. Rationale: a particle links forward from inside a name; at the comes from the fold (P1) or from position (O4), never from a join. This is why a title before a leading particle changes nothing (the title is not a name word), and why "Van Johnson" - is a given-name reading at all. + is a given-name reading at all. An unlisted abbreviation before + the particle is as transparent as a listed title, since assign + reads it as one (H2). "Van Johnson" → given="Van" "Sir de Mesnil" → pieces=[["Sir"], ["de"], ["Mesnil"]] + "Xyz. van Johnson" → given="van" "John van der Berg" → pieces=[["John"], ["van", "der", "Berg"]] · boundary - history: decisions.md#P2 · interacts: P1, P5 · implemented: nameparser/_pipeline/_group.py + history: decisions.md#P2 · interacts: P1, P5, H2 · implemented: nameparser/_pipeline/_group.py P5. Rationale: some given-name words are incomplete alone — "abdul" is a bound form that the next word completes. @@ -259,12 +278,18 @@ P5. Rationale: some given-name words are incomplete alone — "abdul" ambiguous acronym is a name word wherever the peel does not take it; a marker left as a word that a particle join (P2) has already taken travels with that join. A title word standing in the name - is a name word (H3) and joins like one: the pair is a given name - whatever tag the word carried. What there is to spare is what + is a name word (H3) and joins like one, and so is a particle the + chain has not taken (P2) — unless it is of the unambiguous + suffix vocabulary too (vd, mc), which the join declines as the + suffix piece it is: + the pair is a given name whatever tag the word carried, and + after a family comma the join runs before the trailing + particle's attachment (P6) sees the name. What + there is to spare is what assign will leave: the join is tried on the pieces as it would leave them, assign's trailing peel (S2) is read over that, and the name words it leaves are the words to spare — a trailing - roman numeral, or a bare acronym with words behind it, is no + roman numeral, or a bare acronym the peel takes, is no word to spare. The join joins two name words into one and changes no suffix reading: a word the peel reads as a suffix unjoined must read so joined, or the join declines. After a @@ -293,6 +318,8 @@ P5. Rationale: some given-name words are incomplete alone — "abdul" "abdul Smith Ma" → suffix="Ma" "abdul Smith Berg Ma" → family="Berg" · boundary "abdul Sir Smith Berg" → given="abdul Sir" + "Berg, abdul van" → given="abdul van" + "Berg, abdul vd" → family="vd Berg" "abdul Jr Smith Berg" → given="abdul" "abdul Jr Smith Berg" → middle="Jr Smith" "abdul Ph. D. Smith Berg" → suffix="Ph. D." @@ -316,7 +343,7 @@ P5. Rationale: some given-name words are incomplete alone — "abdul" "Sheik abdul salam" family-first → family="abdul salam" "Sheik abdul salam" family-first → given="" "Sheik abdul salam" family-first-given-last → family="abdul salam" - history: decisions.md#P5 · interacts: S2, M2, H1, P2, P4 · implemented: nameparser/_pipeline/_group.py, nameparser/_pipeline/_post_rules.py + history: decisions.md#P5 · interacts: S2, M2, H1, P2, P4, P6 · implemented: nameparser/_pipeline/_group.py, nameparser/_pipeline/_post_rules.py P6. Rationale: a particle ending the name has nothing to link forward to, so it is not doing a particle's work there. A @@ -383,7 +410,13 @@ P6. Rationale: a particle ending the name has nothing to link tracked with the other contested memberships. `do` sits in the AMBIGUOUS acronym half and was already read as a name word there, so the precedence decides nothing for it. - history: decisions.md#P6 · interacts: C1, P1, S2 · implemented: nameparser/_pipeline/_post_rules.py + Accepted: a bound given word ahead of the trailing particle takes + it as its pair first (P5), so the attachment never sees it — + unless the particle is of the unambiguous suffix vocabulary too + (vd, mc), which the join declines and the attachment then takes. + "Berg, abdul van" → given="abdul van" + "Berg, abdul vd" → family="vd Berg" + history: decisions.md#P6 · interacts: C1, P1, S2, P5 · implemented: nameparser/_pipeline/_post_rules.py ## Suffixes: generational & credentials (S) @@ -500,8 +533,12 @@ M1. Rationale: an enclosure the caller has declared to mean maiden M2. Rationale: a maiden marker announces that what follows it is the former family name; the marker is an announcement, not a name. A recognized maiden marker standing after at least one name - word takes the words after it — up to any trailing suffix — as - the maiden name, and the marker itself is dropped. A marker + word takes the words after it — up to any suffix word, or the + trailing roman numeral assign reads as the suffix (S2), both as + written and as the take would leave the name, the word before + the numeral being then the word before the marker — as the + maiden name, and + the marker itself is dropped. A marker with nothing after it, or nothing before it, is just a word. A marker taken this way also bounds a particle join arriving from its left (P2), so the family name's particles stop at the marker @@ -511,6 +548,11 @@ M2. Rationale: a maiden marker announces that what follows it is the "Jane Smith née Jones" → maiden="Jones" "Jane née Jones Smith" → maiden="Jones Smith" "Jane Smith née Jones PhD" → suffix="PhD" + "John née Jones Smith V" → maiden="Jones Smith" + "John née Jones Smith V" → suffix="V" + "Jane Smith née V" → suffix="V" + "J. née Jones Smith V" → maiden="Jones Smith V" · boundary + "Jane née Jones J. V" → maiden="Jones J. V" · boundary "Jones née" → family="née" · boundary "née Jones" → family="Jones" · boundary "Jane van der Berg née Jones" → maiden="Jones" @@ -531,7 +573,11 @@ M2. Rationale: a maiden marker announces that what follows it is the would have bound the two into one name word (P3); the connective then builds a family name out of what is left. "Jane née Jr y Jones" → maiden="" - history: decisions.md#M2 · interacts: P2, P3, P5, R2, M1 · implemented: nameparser/_pipeline/_group.py + Accepted: a bare acronym the peel would take with words to spare + is maiden text all the same — the count it needs includes the + very words the marker removes, so the reading is left to assign. + "John née Jones Smith Ma" → maiden="Jones Smith Ma" + history: decisions.md#M2 · interacts: P2, P3, P5, R2, M1, S2 · implemented: nameparser/_pipeline/_group.py ## Commas & structure (C) diff --git a/docs/release_log.rst b/docs/release_log.rst index 2abea1ec..36cab410 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -41,6 +41,8 @@ Release Log - Fix a bound given name joining past a credential that the suffix rule then takes, leaving no family: ``"abdul Smith Jr Ma"`` read given ``abdul Smith`` with no family, where ``"John Smith Jr Ma"`` reads family ``Smith`` with suffix ``Jr, Ma``. The join's reserve had re-derived the suffix rule's trailing walk by hand and left out one of its two forks -- a bare acronym such as ``Ma`` is a credential only with words to spare -- so it counted the acronym as a name word, joined, and the suffix rule then peeled it. The two stages now share one walk: the reserve tries the join, runs the suffix rule's own peel over the pieces as the join would leave them, and joins only when two name words became one and nothing else changed. ``"abdul Smith Jr Ma"`` reads family ``Smith``, suffix ``Jr, Ma``; ``"abdul Smith Ma"`` reads family ``Smith``, suffix ``Ma`` rather than joining the pair and keeping the credential as the family -- both as 1.4.0 read them and as ``John`` reads in their place; ``"abdul Smith Berg Ma"`` keeps its join, and a title word standing after the bound word joins again as it did in 1.4.0 (``"Berg, abdul Sir"`` reads given ``abdul Sir``) without the pair ever being read as a title (``"abdul Sheikh and Ahmad Bakar"`` reads given ``abdul Sheikh and Ahmad``). No name of the differential corpora has the shapes; the ``Ma`` rules examples carry the fix, byte-identical to 1.4.0 and moving against 2.0.0 and 2.1.0, and the title-word shapes are pinned in tests (closes #425) + - Fix a particle chain and a maiden name taking a trailing generational numeral as a name word: ``"John van der Berg V"`` read family ``van der Berg V`` and ``"John née Jones Smith V"`` read maiden ``Jones Smith V``, where ``"John Smith V"`` reads suffix ``V``. Both stops asked "is this a suffix?" with the test that vetoes a bare ``V`` as an initial -- the same question the bound given-name join asked until #401 -- and now ask the suffix rule's own walk, over the name as they would leave it: family ``van der Berg``, maiden ``Jones Smith``, suffix ``V``, for ``I`` and ``X`` alike; ``"John van der J. V"`` and ``"J. née Jones Smith V"`` keep their readings, the word before the numeral being an initial as written or as left. The chain stops before a bare credential with words to spare too, as ``"John Smith Ma"`` does, where the suffix rule will still find them once the chain has run: ``"John van der Berg Ma"`` reads suffix ``Ma``, which is how 1.4.0 read it, while ``"Freiherr von Berg MA"`` keeps family ``von Berg MA``; a maiden name keeps such a credential, since the words the marker takes are the very words that made it one. A particle that is also suffix vocabulary now ends the chain where it opens the trailing run: ``"John Smith Mc V"`` reads suffix ``Mc, V``. The chain also no longer swallows the given name behind an unlisted abbreviation: ``"Xyz. van Johnson"`` and ``"Esq. van Gogh"`` read given ``van``, as ``"Dr. van Johnson"`` has since #367, the chain and the bound given-name join now asking the same test that reads the abbreviation as a title (closes #424) + - Fix a name opening with a particle that is *never* a given name being split at the particle under a family-first name order -- ``Policy(name_order=FAMILY_FIRST)`` and ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)`` alike, and identically: ``"de Mesnil"`` read as family ``de``, given ``Mesnil``, and ``"de la Vega"`` as family ``de``, given ``la Vega``. Each is now the whole surname, as it has always been in the default order. The rule enforcing it asked for the particle by the ``GIVEN`` role, which under a family-first order belongs to the token *after* the particle, so the test read the wrong word and declined. It now also asks by position -- the piece that opens the name -- so both shapes of the same rule are caught: where such a particle stands alone as a piece, either opening the name or in the given position, the name is left with no given name at all, the given and the middles folding into the family. Standing *alone* is the whole of it, and the rule claims nothing wider: ``"Juan de la Vega"`` under ``FAMILY_FIRST`` still reports given ``de la Vega``, because there the particle chained onto the words after it rather than standing alone, and a bare ``"de"`` with nothing to fold into is still reported as the given name. The decision behind the fix: a word that can never be a given name leaves ``name_order`` nothing to decide, so declaring family-first is not a reason to make ``de`` a surname on its own. A leading particle that *may* be a given name is genuinely order-dependent and is untouched -- ``"van Gogh"`` still reads as family ``van``, given ``Gogh`` under both family-first orders. This is also what gives ``Lexicon.particles_ambiguous`` an effect outside the default order: taking a word out of it now changes the parsed fields under a family-first order, where before it moved only the ambiguity report. Seven of the 751 differential corpus names move, the same seven under each family-first order; default-order output is byte-identical over all 751, at the 1.4.0, 2.0.0 and 2.1.0 differential baselines alike (closes #359) - Fix a family name made only of particle words reporting no base on the 2.0 API, so the surname vanished from ``family_base`` and, on both APIs, from the initials: ``parse("Anh Do")`` gave family ``Do`` with ``family_base`` ``''`` and initials ``A.``, and under ``Policy(name_order=FAMILY_FIRST)`` ``"Del Toro"`` gave family ``Del`` the same way. The empty base was a 2.0 regression rather than a longstanding bug -- 1.4.0's own guard kept ``HumanName("Anh Do").last_base`` at ``Do``, and the facade has kept it right throughout; what 1.4.0 and 2.1 shared was the missing initial. A particle earns its name by joining forward to the word it modifies, so a particle standing alone in a name part is not doing a particle's work there and reads as an ordinary name word: it anchors the base, leaves the particles view, and contributes an initial. ``"Anh Do"`` is now base ``Do``, initials ``A. D.``; ``"Juan van der"`` is base ``van der``, initials ``J. v. d.``; ``"Nguyen, Van Le"`` initials ``V. L. N.`` where the middle name used to be dropped. Position decides this, not vocabulary -- whether the word is borne as a surname somewhere does not enter into it, which reverses the reasoning ``rules.md#R2`` carried before. The invariant it exists to hold: a non-empty family always has a non-empty base, because a particle needs a base to attach to. Where the particles DO join a name word nothing changes -- ``"Juan de la Vega"`` keeps base ``Vega``, particles ``de la`` and initials ``J. V.``. The parse fields themselves do not move: ``title``, ``given``, ``middle``, ``family``, ``suffix``, ``nickname`` and ``maiden`` are byte-identical over all 751 differential corpus names in all three name orders, and only the derived views and the initials change -- which is why the differential harness, which compares those seven roles, reports no diffs for this at any baseline. One consequence to know about: where the whole family is particles AND a tussenvoegsel was attached to it (#379), the two APIs order the initials differently -- ``parse("der, y van").initials()`` is ``y. d. v.`` against the facade's ``y. v. d.`` -- because ``initials()`` reads tokens in written order while the family FIELD renders folded words first. That affects 87 constructed inputs and no corpus name; it is a pre-existing ordering gap in ``initials()`` that this change makes visible, tracked at #408 (closes #385, closes #402) diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index a1b68cb9..c7509918 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -32,13 +32,12 @@ from __future__ import annotations import dataclasses -import re from nameparser._pipeline._vocab import ( effective_script, is_suffix_lenient, resolve_script_set, ) from nameparser._pipeline._group import ( - _is_suffix_piece, _is_title_piece, _peel_trailing, _peel_walk, + _is_suffix_piece, _leading_titles, _peel_trailing, _peel_walk, ) from nameparser._pipeline._state import ( ParseState, PendingAmbiguity, Structure, WorkToken, @@ -46,12 +45,6 @@ from nameparser._policy import Policy, Script from nameparser._types import AmbiguityKind, Role -# Ported verbatim from v1 (nameparser/config/regexes.py -# "period_abbreviation"; "roman_numeral" is _vocab._ROMAN since #401) -# -- layering forbids the config import; keep in sync by hand. -_PERIOD_ABBREV = re.compile(r'^[^\W\d_]{2,}\.$') - - def _set_roles(tokens: list[WorkToken], piece: tuple[int, ...], role: Role) -> None: for i in piece: @@ -60,30 +53,17 @@ def _set_roles(tokens: list[WorkToken], piece: tuple[int, ...], # rules.md#H2: "an abbreviation opening the part of the name that # carries the given name — the whole name, or the part after a -# family comma — reads as a title even when unlisted" -# (history: decisions.md#H2) -def _is_leading_title(piece: tuple[int, ...], ptags: frozenset[str], - tokens: list[WorkToken]) -> bool: - if _is_title_piece(piece, ptags, tokens): - return True - return (len(piece) == 1 - and bool(_PERIOD_ABBREV.match(tokens[piece[0]].text))) - - +# family comma — reads as a title even when unlisted" -- the count is +# group's _leading_titles since #424 (its test, _is_leading_title, is +# the leading-particle scan's too); the roles are set here. def _peel_leading_titles(pieces: tuple[tuple[int, ...], ...], ptags: tuple[frozenset[str], ...], tokens: list[WorkToken]) -> int: """Assign TITLE to the leading title pieces and return the first - non-title index. A title needs a following piece, unless the whole - segment is one title (v1 parity).""" - n = 0 - while n < len(pieces): - if ((n + 1 < len(pieces) or len(pieces) == 1) - and _is_leading_title(pieces[n], ptags[n], tokens)): - _set_roles(tokens, pieces[n], Role.TITLE) - n += 1 - continue - break + non-title index.""" + n = _leading_titles(pieces, ptags, tokens) + for k in range(n): + _set_roles(tokens, pieces[k], Role.TITLE) return n diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index c447f6be..24c123d7 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -13,20 +13,23 @@ registration becomes piece_tags entries -- per-parse state that dissolves with the state (v1 kept per-parse sets for the same reason). -Implements rules H3, P2, P3, P4 and M2 of docs/design/rules.md and the +Implements rules H3, P2, P3, P4 and M2, and houses H2's test of docs/design/rules.md and the group half of M1 (#329: the marker dropped inside EXTRACTED maiden content, which M2's pieces walk cannot reach because extract's content never enters pieces); each is cited at its code below. Also implements rule P5 (cited below at the bound-given join) and ports the "Ph. D."-split merge (v1 fix_phd; decisions.md#phd-merge). Houses -the S2 trailing peel (_peel_walk, _peel_trailing), a piece-level walk -that assign applies and P5's reserve reads over the view the join -would leave (#425). +the S2 trailing peel (_peel_walk, _peel_trailing, _trailing_start), a +piece-level walk that assign applies and that P5's reserve, P2's +chain and M2's walk read (#425, #424), and assign's leading-title +test (_is_leading_title, _leading_titles), which the chain's scan +shares since #424. """ from __future__ import annotations import bisect import dataclasses +import re from collections.abc import Sequence, Set from enum import IntEnum from typing import NamedTuple @@ -77,12 +80,51 @@ def _is_title_piece(piece: Sequence[int], ptags: Set[str], return len(piece) == 1 and "vocab:title" in tokens[piece[0]].tags +# Ported verbatim from v1 (nameparser/config/regexes.py +# "period_abbreviation") -- layering forbids the config import; keep +# in sync by hand (tests/v2/test_regex_sync.py). Here rather than in +# assign since #424: the leading-title test is assign's, and group's +# leading-particle scan and trailing-run walk must start where assign +# starts. +_PERIOD_ABBREV = re.compile(r'^[^\W\d_]{2,}\.$') + + +# rules.md#H2: "an abbreviation opening the part of the name that +# carries the given name — the whole name, or the part after a +# family comma — reads as a title even when unlisted" +# (history: decisions.md#H2) +def _is_leading_title(piece: Sequence[int], ptags: Set[str], + tokens: Sequence[WorkToken]) -> bool: + if _is_title_piece(piece, ptags, tokens): + return True + return (len(piece) == 1 + and bool(_PERIOD_ABBREV.match(tokens[piece[0]].text))) + + +def _leading_titles(pieces: Sequence[Sequence[int]], + ptags: Sequence[Set[str]], + tokens: Sequence[WorkToken]) -> int: + """How many leading pieces assign peels as titles: the first + non-title index. A title needs a following piece, unless the whole + segment is one title (v1 parity). One definition, read by assign + (which sets the roles) and by the chain's trailing-run walk; the + leading-particle scan shares the predicate, _is_leading_title, + but stops at a title-and-particle word (P4, #367, #424).""" + n = 0 + while n < len(pieces): + if ((n + 1 < len(pieces) or len(pieces) == 1) + and _is_leading_title(pieces[n], ptags[n], tokens)): + n += 1 + continue + break + return n + + # rules.md#P2: "a particle joins the words after it into one name # part, the join running until the next particle starts a group of -# its own, a trailing suffix begins, a maiden marker takes the words -# after it (M2), or the name ends. The final group reads as the -# family name; earlier groups read by position." -# (history: decisions.md#P2) +# its own, a trailing suffix begins" -- and on to the maiden marker +# (M2) or the name's end; the final group reads as the family name, +# earlier groups by position. (history: decisions.md#P2) # rules.md#P4: "a particle in the name's leading position chains # nothing: the words stay separate" (history: decisions.md#P2) def _is_prefix_piece(piece: Sequence[int], ptags: Set[str], @@ -122,14 +164,47 @@ class Peel(NamedTuple): # ambiguous acronym written with periods counts unambiguously. A BARE # ambiguous acronym is consumed only when the name has words to spare" # (v1's are_suffixes tail rule, with the roman-numeral special) -def _peel_walk(start: int, ptags: Sequence[Set[str]]) -> list[int]: +def _peel_walk(start: int, ptags: Sequence[Set[str]], + skip: Set[int] = frozenset()) -> list[int]: """The indices _peel_trailing walks: `start` to the segment's end, minus the group-flagged credential pieces (the Ph. D. merge), - which assign reads as suffixes at any position. Built here and - nowhere else, so the walk's input cannot drift between assign and - the reserve: the numeral fork is a last-piece test that reads the - piece before as rest[k - 2], which holds only over this list.""" - return [j for j in range(start, len(ptags)) if "suffix" not in ptags[j]] + which assign reads as suffixes at any position, and minus `skip` + -- a tail segment's delimiter cores, which are structure rather + than words (the maiden walk's case, #424). Built here and nowhere + else, so the walk's input cannot drift between assign and the + three group sites that read it: the numeral fork is a last-piece + test that reads the piece before as rest[k - 2], which holds only + over this list.""" + return [j for j in range(start, len(ptags)) + if j not in skip and "suffix" not in ptags[j]] + + +def _trailing_start(start: int, pieces: Sequence[Sequence[int]], + ptags: Sequence[Set[str]], tokens: Sequence[WorkToken], + skip: Set[int] = frozenset(), + numeral_only: bool = False) -> int: + """Where assign's trailing suffix run begins, read over the pieces + as they stand from `start`: the index of the first piece the S2 + peel takes, or len(pieces) when it takes none (#424). What P2's + chain and M2's walk stop before -- each had asked "is this a + suffix?" with the suffix-piece test, which vetoes a bare 'V' as + an initial (the #401 question), and so took a trailing numeral, + or a bare acronym with words to spare, into the family or the + maiden name. + + `numeral_only` is the maiden walk's reading: the bare-acronym + fork counts pieces, and the walk removes the very pieces it + counted, so an acronym peeled over the pieces as they stand may + be the family of what is left ('John née Jones Smith Ma' read + maiden 'Jones Smith', family 'Ma'). The numeral fork reads one + piece, the one before the numeral, and _maiden_take re-asks it + with the piece the take leaves there; the acronym is left to + assign.""" + rest = _peel_walk(start, ptags, skip) + peeled = _peel_trailing(rest, pieces, ptags, tokens) + if numeral_only: + return rest[-1] if peeled.numeral is not None else len(pieces) + return rest[peeled.names] if peeled.names < len(rest) else len(pieces) def _peel_trailing(rest: Sequence[int], pieces: Sequence[Sequence[int]], @@ -183,8 +258,9 @@ def _peel_trailing(rest: Sequence[int], pieces: Sequence[Sequence[int]], # rules.md#M2: "a recognized maiden marker standing after at least -# one name word takes the words after it — up to any trailing -# suffix — as the maiden name, and the marker itself is dropped" +# one name word takes the words after it" -- up to any suffix word, +# or the trailing numeral assign reads as the suffix, as the maiden +# name, the marker itself dropped # (history: decisions.md#M2) # # A marker piece is a LONE marker -- M2's own "standing as a word of @@ -243,9 +319,42 @@ def _maiden_take(pieces: Sequence[Sequence[int]], if _is_maiden_marker_piece(pieces[seen[v]], tokens)), None) if m is None: return None + # "up to any trailing suffix": a suffix WORD anywhere after the + # marker ends the maiden name, and so does the trailing numeral as + # assign will read it, which the suffix-piece test does not see + # (#424): 'John née Jones Smith V' took the V as maiden text. The + # numeral only -- _trailing_start says why the acronym fork is + # left to assign here. Read from the MARKER, not after it: a + # numeral straight after the marker then has the piece before it + # the fork wants, and 'Jane Smith née V' declines like 'Jane Smith + # née PhD' -- nothing after the marker but a suffix, so the marker + # stays a word -- as 1.4.0 read it. + skip = frozenset(range(len(pieces))) - frozenset(seen) + trailing = _trailing_start(seen[m], pieces, ptags, tokens, skip, + numeral_only=True) + # The fork reads the piece before the numeral, and the take + # REMOVES that piece: afterwards assign sees the piece before the + # marker there, and if that is initial-shaped the fork will not + # fire -- a walk that stopped anyway handed the V to the family + # ('J. née Jones Smith V'). So the numeral must read as the suffix + # as the take would leave the name too, and the question is asked + # the way P5's reserve asks it (#425): the peel is run over the + # VIEW the take would leave, not one condition of it -- the first + # re-ask checked the preceding piece alone, and a title before the + # marker ('Dr. née Jones Smith V') leaves the numeral as assign's + # whole rest, where no fork fires at all (the code review). + if trailing < len(pieces): + left = [i for i in seen if i < seen[m] or i >= trailing] + view = [pieces[i] for i in left] + view_tags = [ptags[i] for i in left] + if _trailing_start(_leading_titles(view, view_tags, tokens), + view, view_tags, tokens, + numeral_only=True) == len(view): + trailing = len(pieces) j = m + 1 - while j < len(seen) and not _is_suffix_piece( - pieces[seen[j]], ptags[seen[j]], tokens): + while (j < len(seen) and seen[j] < trailing + and not _is_suffix_piece(pieces[seen[j]], ptags[seen[j]], + tokens)): j += 1 # j == m + 1 means nothing followed the marker but a suffix, so the # pass declines and the marker stays an ordinary word (rules.md#M2). @@ -360,8 +469,9 @@ def merge(lo: int, hi: int, add: Set[str] = frozenset(), k += 1 # rules.md#M2: "a recognized maiden marker standing after at least - # one name word takes the words after it — up to any trailing - # suffix — as the maiden name, and the marker itself is dropped" + # one name word takes the words after it" -- up to any suffix + # word, or the trailing numeral assign reads as the suffix, as the + # maiden name, the marker itself dropped # (history: decisions.md#M2) -- the marker pass (#274), and it runs # BEFORE every join below. Each join rule asks a question about the # name -- how many words it has (P3's carve-out, P5's reserve), @@ -479,86 +589,148 @@ def merge(lo: int, hi: int, add: Set[str] = frozenset(), # default worth testing: it is reached only when every piece is # a title and none is a prefix, and the loop below merges # nothing unless some piece is a prefix. + # `title(k)` alone missed H2's unlisted abbreviations, which + # assign peels as titles all the same, so 'Xyz. van Johnson' + # chained where 'Dr. van Johnson' did not (#424 found it + # through the acronym fork: the chain had swallowed the given + # word and left assign two pieces where the fork counted + # three). The scan asks assign's own test. leading = next((k for k in range(len(pieces)) - if not title(k) or prefix(k)), 0) - k = 0 - while k < len(pieces): - if k == leading or not prefix(k): + if not _is_leading_title(pieces[k], ptags[k], + tokens) + or prefix(k)), 0) + # rules.md#P2: "a trailing suffix begins" -- where it begins + # is read by assign's peel over the pieces as they stand + # (#424), once per segment and kept as a length from the end, + # which the chain's merges ahead of it do not move -- except + # where a particle that is suffix vocabulary too (vd, mc, do) + # starts the run: the prefix run below takes it as a particle, + # as P6 reads it after a comma, and 'John van Mc' keeps family + # 'van Mc' (every baseline's reading). A suffix WORD stops the + # chain wherever it stands, and the trailing + # run -- the numeral, or the bare acronym with words to spare + # -- stops it where the suffix-piece test alone did not ('John + # van der Berg V' read family 'van der Berg V'). The chain + # takes both forks, and asks again after its merges whether + # the acronym still has the pieces the fork counted (below). + name_start = _leading_titles(pieces, ptags, tokens) + tail = len(pieces) - _trailing_start(name_start, pieces, ptags, + tokens) + def chain(tail: int) -> None: + k = 0 + while k < len(pieces): + if k == leading or not prefix(k): + k += 1 + continue + j = k + 1 + while j < len(pieces) and prefix(j): + j += 1 + while (j < len(pieces) - tail and not prefix(j) + and not suffix(j)): + j += 1 + # The other half of PARTICLE_OR_GIVEN. _assign reports the + # fork when an ambiguous particle stays a lone leading piece + # ("Van Johnson" -> given under the default order, family + # under FAMILY_FIRST); the chain here takes the opposite + # branch when the particle is not the name's leading piece. + # A fork whose two sides are decided in different stages + # needs an emitter in each. + # + # Narrow, and #367 is why. `all(_is_leading_title(...))` + # says every piece ahead of this one is a title, and the + # loop skipped k == leading, so `leading` is STRICTLY + # before k -- and being before k it is one of those titles, + # while being `leading` it satisfies `not title or prefix`. + # For both, it must be a prefix as well: a word in both + # vocabularies (`st`, `do`, `freiherr` by default, or any + # overlap a caller configures). A plain title alone can no + # longer put a particle off the name's leading piece; it is + # stepped over and _assign reports the fork instead. + # + # What that leaves is wider than one shape: any number of + # plain title pieces, then a piece in BOTH vocabularies, + # then any number of further titles, then the ambiguous + # particle whose chain claims something. "Freiherr von + # Richthofen" is the canonical spelling and the one + # tests/v2/cases.py and tests/v2/test_parser.py lead with, + # but "St Van Johnson", "Do St Johnson" (the chained + # particle itself in both vocabularies) and "Dr. Do van + # Johnson" (a plain title AHEAD of the both-vocabulary + # word) all reach here too. What none of them can do is + # dispense with the both-vocabulary WORD. The conjunction + # merge is the only other way a piece acquires `title` or + # `prefix`, and it cannot manufacture the pair: it derives + # from ONE neighbor, which is the left one whenever there + # is a left one, and its right operands are always fresh + # pieces (the loop runs left to right, so nothing to the + # right has been merged yet). Both tags therefore have to + # come from the piece it extends, which bottoms out at a + # lone token in both vocabularies. + # + # j > k + 1 is what makes this a DECISION rather than a + # shape: when the next piece is a suffix the inner scan + # never advances, merge(k, k+1) folds a piece into itself, + # and the particle stays a lone leading piece -- nothing + # was chained, and _assign reports that case instead. + # Without this the two emitters both fire on the same token. + # (Tag test first: it is a set lookup and almost no name has + # an ambiguous particle, while title() is a call per piece.) + if (j > k + 1 + and "vocab:particle-ambiguous" + in tokens[pieces[k][0]].tags + and all(_is_leading_title(pieces[x], ptags[x], + tokens) + for x in range(k))): + i = pieces[k][0] + ambiguities.append(PendingAmbiguity( + AmbiguityKind.PARTICLE_OR_GIVEN, + f"{tokens[i].text!r} was chained onto the following " + f"name piece; it is also a given name in other " + f"names", + (i,))) + merge(k, j, drop={"prefix"}) k += 1 - continue - j = k + 1 - while j < len(pieces) and prefix(j): - j += 1 - while j < len(pieces) and not prefix(j) and not suffix(j): - j += 1 - # The other half of PARTICLE_OR_GIVEN. _assign reports the - # fork when an ambiguous particle stays a lone leading piece - # ("Van Johnson" -> given under the default order, family - # under FAMILY_FIRST); the chain here takes the opposite - # branch when the particle is not the name's leading piece. - # A fork whose two sides are decided in different stages - # needs an emitter in each. - # - # Narrow, and #367 is why. `all(title(x) for x in range(k))` - # says every piece ahead of this one is a title, and the - # loop skipped k == leading, so `leading` is STRICTLY - # before k -- and being before k it is one of those titles, - # while being `leading` it satisfies `not title or prefix`. - # For both, it must be a prefix as well: a word in both - # vocabularies (`st`, `do`, `freiherr` by default, or any - # overlap a caller configures). A plain title alone can no - # longer put a particle off the name's leading piece; it is - # stepped over and _assign reports the fork instead. - # - # What that leaves is wider than one shape: any number of - # plain title pieces, then a piece in BOTH vocabularies, - # then any number of further titles, then the ambiguous - # particle whose chain claims something. "Freiherr von - # Richthofen" is the canonical spelling and the one - # tests/v2/cases.py and tests/v2/test_parser.py lead with, - # but "St Van Johnson", "Do St Johnson" (the chained - # particle itself in both vocabularies) and "Dr. Do van - # Johnson" (a plain title AHEAD of the both-vocabulary - # word) all reach here too. What none of them can do is - # dispense with the both-vocabulary WORD. The conjunction - # merge is the only other way a piece acquires `title` or - # `prefix`, and it cannot manufacture the pair: it derives - # from ONE neighbor, which is the left one whenever there - # is a left one, and its right operands are always fresh - # pieces (the loop runs left to right, so nothing to the - # right has been merged yet). Both tags therefore have to - # come from the piece it extends, which bottoms out at a - # lone token in both vocabularies. - # - # j > k + 1 is what makes this a DECISION rather than a - # shape: when the next piece is a suffix the inner scan - # never advances, merge(k, k+1) folds a piece into itself, - # and the particle stays a lone leading piece -- nothing - # was chained, and _assign reports that case instead. - # Without this the two emitters both fire on the same token. - # (Tag test first: it is a set lookup and almost no name has - # an ambiguous particle, while title() is a call per piece.) - if (j > k + 1 - and "vocab:particle-ambiguous" - in tokens[pieces[k][0]].tags - and all(title(x) for x in range(k))): - i = pieces[k][0] - ambiguities.append(PendingAmbiguity( - AmbiguityKind.PARTICLE_OR_GIVEN, - f"{tokens[i].text!r} was chained onto the following " - f"name piece; it is also a given name in other " - f"names", - (i,))) - merge(k, j, drop={"prefix"}) - k += 1 + + # The peel was read over the pieces as they stand, and the + # chain's own merges can change what it counts: behind a word + # in both the title and particle vocabularies the scan above + # stops where assign's title peel does not (P4, #367), so the + # chain takes the name's first word, and the acronym the fork + # counted with three pieces meets assign with two -- 'Freiherr + # von Berg Ma' read given 'von Berg', family 'Ma' (1.4.0's + # reading; the reviews found it behind the claim that the + # merges leave the count alone). So the peel is asked again + # over the pieces the chain leaves, and where it no longer + # takes what the chain stopped before, the chain runs again + # without that stop: what assign will not peel is a name word, + # and the chain takes it. The numeral cannot flip (a chain + # group is never initial-shaped), so the second run is the + # acronym's alone, and rare; the snapshot is one copy per + # segment with a trailing run, linear like the rest. + if tail: + kept = ([list(q) for q in pieces], [set(t) for t in ptags], + len(ambiguities)) + chain(tail) + left = len(pieces) - _trailing_start( + _leading_titles(pieces, ptags, tokens), pieces, ptags, + tokens) + if left < tail: + pieces[:], ptags[:] = kept[0], kept[1] + del ambiguities[kept[2]:] + chain(left) + else: + chain(0) # rules.md#P5: "a recognized bound given-name word joins the # word after it into one given name" (history: decisions.md#P5) # -- bound given names: the first non-title piece joins the next # ONCE (pairwise, v1 parity: 'Salem, Abdul Rahman Ahmed' keeps # Ahmed a middle name). BoundJoin encodes v1's reserve_last. - fk = next((k for k in range(len(pieces)) if not title(k)), None) + # "the first non-title piece" by assign's count (#424): group's + # title test does not see H2's unlisted abbreviations, and + # 'Xyz. abdul John Smith' joined nothing where 'Dr. abdul John + # Smith' read given 'abdul John'. + fk = _leading_titles(pieces, ptags, tokens) if (bound_join is not BoundJoin.DISABLED - and fk is not None and fk + 1 < len(pieces) and len(pieces[fk]) == 1 and "vocab:bound-given" in tokens[pieces[fk][0]].tags): diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 7e0d49c4..be910498 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -72,8 +72,7 @@ def _retag(tokens: list[WorkToken], i: int, role: Role) -> None: # rules.md#P2: "a particle joins the words after it into one name # part, the join running until the next particle starts a group of -# its own, a trailing suffix begins, a maiden marker takes the words -# after it (M2), or the name ends" +# its own, a trailing suffix begins" # rules.md#P3: "the joined part is ONE name word wherever another # rule counts them" # rules.md#P5: "a recognized bound given-name word joins the word diff --git a/tests/v2/pipeline/test_assign.py b/tests/v2/pipeline/test_assign.py index 7dd32231..9eebb82b 100644 --- a/tests/v2/pipeline/test_assign.py +++ b/tests/v2/pipeline/test_assign.py @@ -65,6 +65,14 @@ def test_title_only() -> None: assert not _by_role(out, Role.GIVEN) +def test_a_title_needs_a_following_piece() -> None: + # v1 parity, in the count group shares since #424: the last of two + # title words is the name, not a second title + out = _assigned("Dr. Mr.") + assert _by_role(out, Role.TITLE) == "Dr." + assert _by_role(out, Role.GIVEN) == "Mr." + + def test_leading_ambiguous_particle_reads_as_given_with_ambiguity() -> None: out = _assigned("Van Johnson") assert _by_role(out, Role.GIVEN) == "Van" diff --git a/tests/v2/pipeline/test_group.py b/tests/v2/pipeline/test_group.py index e1d7e0df..0b692383 100644 --- a/tests/v2/pipeline/test_group.py +++ b/tests/v2/pipeline/test_group.py @@ -448,6 +448,16 @@ def test_a_delimiter_core_in_a_suffix_tail_is_not_maiden_text() -> None: "Jones"] +def test_the_walk_peels_past_a_trailing_core() -> None: + # The walk's peel skips the cores a tail segment drops, so a core + # standing last does not make the numeral "not last": the V is the + # suffix and the marker takes 'Jones' alone (#424; the test + # review's surviving mutant). + out = _grouped("Smith, John, PhD née Jones V -", policy=_DASH) + assert [t.text for t in out.tokens if t.role is Role.MAIDEN] == [ + "Jones"] + + def test_a_core_is_screened_before_the_marker_looks_for_a_word_ahead() -> None: """M2 needs a name word BEFORE the marker. A core is not one, so a marker standing behind nothing but a core is a leading marker and @@ -788,3 +798,142 @@ def test_the_licence_does_not_lift_the_equality() -> None: # by the test review. out = _grouped("sir abdul J. V", lexicon=_GIVEN_NAME_TITLE_LEX) assert _piece_texts(out) == [["sir", "abdul", "J.", "V"]] + + +# -- #424: the chain and the maiden walk stop where assign's peel begins + + +def test_the_chain_stops_before_the_numeral_assign_reads_as_the_suffix() -> None: + # P2's chain ran "until a trailing suffix begins" and asked with + # the suffix-piece test, which vetoes a bare 'V' as an initial -- + # the #401 question at a third site: 'John van der Berg V' read + # family 'van der Berg V'. The chain now stops where the S2 peel, + # read over the pieces as they stand, begins the trailing run. + out = _grouped("John van der Berg V") + assert _piece_texts(out) == [["John", "van der Berg", "V"]] + # the stop is kept as a length from the end, so a second chain + # ahead of it stops there too (the test review's surviving mutant) + out = _grouped("John van der Berg de la Vega V") + assert _piece_texts(out) == [ + ["John", "van der Berg", "de la Vega", "V"]] + out = _grouped("John van der Berg jr") + assert _piece_texts(out) == [["John", "van der Berg", "jr"]] + + +def test_the_chain_keeps_an_acronym_assign_will_not_peel() -> None: + # Behind a word in both the title and particle vocabularies the + # leading-particle scan stops (P4, #367) before assign's title + # peel does, so the chain takes the name's first word: read over + # the pieces as they stand the acronym has three pieces to spare, + # read over the pieces the chain leaves it has two, and assign + # would make it the family ('Freiherr von Berg Ma' read given 'von + # Berg', family 'Ma' -- 1.4.0's reading, and the reviews' find). + # The chain asks the peel again over what it leaves, and takes the + # acronym assign will not peel. + lex = _AMBIGUOUS_LEX.add(titles={"st"}, particles={"st"}) + out = _grouped("St van Berg Ma", lexicon=lex) + assert _piece_texts(out) == [["St", "van Berg Ma"]] + # with a given word of its own the three pieces survive the chain + out = _grouped("St John van Berg Ma", lexicon=lex) + assert _piece_texts(out) == [["St", "John", "van Berg", "Ma"]] + + +def test_the_chain_keeps_a_numeral_the_peel_does_not_take() -> None: + # The fork is assign's: a numeral after an initial-shaped piece is + # a name word, so the chain takes it as before -- as 'John J. V' + # reads the V as a name. And a numeral with a suffix behind it is + # not last in the walk. + out = _grouped("John van der J. V") + assert _piece_texts(out) == [["John", "van der J. V"]] + out = _grouped("John van der Berg V jr") + assert _piece_texts(out) == [["John", "van der Berg V", "jr"]] + + +def test_the_chain_stops_before_a_bare_acronym_with_words_to_spare() -> None: + # S2's other fork, the same way: 'John Smith Ma' peels the acronym + # as a credential, so 'John van der Berg Ma' does too -- 1.4.0 read + # suffix 'Ma' there, and 2.0 had let the chain take it. + out = _grouped("John van der Berg Ma", lexicon=_AMBIGUOUS_LEX) + assert _piece_texts(out) == [["John", "van der Berg", "Ma"]] + + +def test_the_maiden_walk_stops_before_the_numeral_too() -> None: + # M2's walk takes the words after the marker "up to any trailing + # suffix", asked with the same test: 'John née Jones Smith V' took + # the V into the maiden name. Read by the peel from the marker on, + # the V is the suffix, and the walk stops before it. + out = _grouped("John née Jones Smith V") + assert [t.text for t in out.tokens if t.role is Role.MAIDEN] == \ + ["Jones", "Smith"] + assert _piece_texts(out) == [["John", "V"]] + + +def test_the_maiden_walk_leaves_the_acronym_fork_to_assign() -> None: + # The bare-acronym fork counts pieces, and the walk removes the + # pieces it counted: peeled over the pieces as they stand, 'Ma' + # would be a credential with words to spare, but once 'Jones + # Smith' has left the name it is the family of a two-piece name. + # So the walk takes it as maiden text, as it always did, and + # stops only at the numeral fork. + out = _grouped("John née Jones Smith Ma", lexicon=_AMBIGUOUS_LEX) + assert [t.text for t in out.tokens if t.role is Role.MAIDEN] == \ + ["Jones", "Smith", "Ma"] + # The numeral-only reading is what the acronym BETWEEN the maiden + # name and the numeral shows: the general peel would stop at the + # acronym, the re-ask would veto it, and the walk would take the + # numeral too (the test review's surviving mutant). + out = _grouped("Jane Smith née Jones Ma V", lexicon=_AMBIGUOUS_LEX) + assert [t.text for t in out.tokens if t.role is Role.MAIDEN] == \ + ["Jones", "Ma"] + + +def test_a_marker_followed_only_by_the_numeral_is_just_a_word() -> None: + # The peel is read from the marker, so 'née V' is two pieces and + # the fork fires on the V: nothing follows the marker but a + # suffix, the pass declines, and the marker stays a word -- as + # for 'Jane Smith née PhD', and as 1.4.0 read it (suffix 'V'). + out = _grouped("Jane Smith née V") + assert [t.text for t in out.tokens if t.role is Role.MAIDEN] == [] + assert _piece_texts(out) == [["Jane", "Smith", "née", "V"]] + + +def test_the_walk_stops_only_where_the_numeral_survives_the_take() -> None: + # The fork reads the piece before the numeral, and the walk + # REMOVES that piece: after the take, what assign sees before the + # V is the piece before the marker. Where that is initial-shaped + # the fork will not fire, and a walk that stopped anyway handed + # the V to the family ('J. née Jones Smith V' read family 'V'). + # So the walk stops only where the numeral reads as the suffix + # both as written and as the take would leave it. Found by both + # reviews. + out = _grouped("J. née Jones Smith V") + assert [t.text for t in out.tokens if t.role is Role.MAIDEN] == \ + ["Jones", "Smith", "V"] + # and it is the whole peel that is re-asked, not one condition of + # it: a title before the marker is peeled by assign first, leaving + # the numeral as the whole rest, where no fork fires (the code + # review found the first re-ask handing the V to the given name) + out = _grouped("Dr. née Jones Smith V") + assert [t.text for t in out.tokens if t.role is Role.MAIDEN] == \ + ["Jones", "Smith", "V"] + + +def test_an_unlisted_abbreviation_is_as_transparent_as_a_title() -> None: + # #367 keyed the leading-particle exception on the first piece of + # the NAME, stepping over titles; assign also peels an unlisted + # abbreviation as a title (H2), and the scan here did not, so + # 'Xyz. van Johnson' chained where 'Dr. van Johnson' did not -- + # and 'Xyz. van Berg Ma' chained the given word into the family, + # leaving assign two pieces where the acronym fork had counted + # three. The scan asks assign's leading-title test now. + out = _grouped("Xyz. van Johnson") + assert _piece_texts(out) == [["Xyz.", "van", "Johnson"]] + out = _grouped("Xyz. van Berg Ma", lexicon=_AMBIGUOUS_LEX) + assert _piece_texts(out) == [["Xyz.", "van", "Berg", "Ma"]] + # and the bound join's "first non-title piece" is the same count, + # so a bound word behind an unlisted abbreviation joins as it does + # behind a listed title (the design-docs review's find) + out = _grouped("Xyz. abdul John Smith") + assert _piece_texts(out) == [["Xyz.", "abdul John", "Smith"]] + out = _grouped("Berg, Xyz. abdul van") + assert _piece_texts(out)[1] == ["Xyz.", "abdul van"] diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 506f4b71..62980cad 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -1242,17 +1242,17 @@ def _claim(rule: dict) -> _Claim: "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": _Claim(107, ('family', 'given', 'middle'), "0191ac9143a6"), "fix(#274) maiden markers consumed": - _Claim(24, ('family', 'maiden', 'middle'), "3adf8b0a3b24"), + _Claim(29, ('family', 'maiden', 'middle'), "c0981c1c6557"), "fix(cjk-maiden-marker) maiden marker consumed, compounding with the CJK order flip": _Claim(5, ('family', 'given', 'maiden', 'middle'), "bc0e10dd7ec8"), "fix(#379) a tussenvoegsel after a family comma attaches to the family": - _Claim(11, ('family', 'middle'), "3acdb7d11969"), + _Claim(13, ('family', 'middle'), "973617235cda"), "fix(comma-family) lone post-comma piece routes to suffix/title, not first": - _Claim(262, ('given', 'suffix', 'title'), "d5e6ee237ff9"), + _Claim(264, ('given', 'suffix', 'title'), "a639a0ca7ce2"), "fix(comma-precomma-family) pre-comma run reads as family, not given": - _Claim(262, ('family', 'given'), "d5e6ee237ff9"), + _Claim(264, ('family', 'given'), "a639a0ca7ce2"), "fix(suffix-routing) two-token name with unambiguous trailing suffix stays suffix": - _Claim(1041, ('family', 'given', 'suffix'), "705c375e2778"), + _Claim(1056, ('family', 'given', 'suffix'), "bfb2e274a968"), "fix(suffix-delimiter-rendering) no-space delimiter core token kept whole": _Claim(0, ('suffix',), "e3b0c44298fc"), "ambiguous-surname-acronym data change: parenthesized (MA)/(DO) now stays nickname": @@ -1277,6 +1277,8 @@ def _claim(rule: dict) -> _Claim: _Claim(4, ('given', 'middle', 'title'), "e86eeb13eeb2"), "fix(leading-credential) a split 'Ph. D.' before the name stays one unit": _Claim(4, ('family', 'given', 'middle', 'suffix', 'title'), "1425d85a2d86"), + "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle": + _Claim(1, ('family', 'given'), "ca7b37af6cf8"), "fix(#367) a title no longer displaces a leading particle out of the leading position": _Claim(4, ('family', 'given', 'middle'), "ae299117dd60"), "fix(#400) abd joins the word after it as one given name": @@ -1305,12 +1307,26 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('given', 'middle'), "228abe0f32ef"), "fix(#425) accepted: a bare ambiguous acronym the peel does not take joins as a name word": _Claim(1, ('given', 'middle'), "2010cc79a34d"), + "fix(#424) the particle chain stops before the trailing numeral": + _Claim(1, ('family', 'suffix'), "2c99162bc9cf"), + "fix(#424) accepted: the maiden walk keeps a bare acronym": + _Claim(1, ('family', 'maiden', 'middle', 'suffix'), "f2c6cd2e3001"), + "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle, the P4 example": + _Claim(1, ('family', 'given'), "42b69cf1b320"), + "fix(#424) accepted: the maiden walk keeps the numeral an initial before the marker vetoes": + _Claim(1, ('family', 'maiden', 'middle', 'suffix'), "08c0158c8d3f"), + "fix(#424) accepted: the chain keeps an acronym assign will not peel behind a title-and-particle word": + _Claim(1, ('family', 'given'), "faa4bedda537"), + "fix(#424) a title-led chain before the numeral is the one name piece": + _Claim(1, ('family', 'given', 'suffix'), "5b3a743f9e35"), + "fix(#424) accepted: a particle of the suffix vocabulary opening the trailing run is a suffix piece": + _Claim(1, ('family', 'middle', 'suffix'), "a564b97f7162"), "fix(#360) ste moved into the never-given particles with mc": _Claim(1, ('family', 'given'), "e62caedec864"), }, "expected_since_2.0.0.toml": { "fix(#379) a tussenvoegsel after a family comma attaches to the family": - _Claim(11, ('family', 'middle'), "3acdb7d11969"), + _Claim(13, ('family', 'middle'), "973617235cda"), "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": _Claim(107, ('_ambiguities', 'family', 'given', 'middle'), "0191ac9143a6"), "fix(#308/#312/#319/#320) glued CJK honorific peeled off the name into suffix": @@ -1323,10 +1339,12 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('family', 'given', 'nickname'), "d4069d459f23"), "fix(#298) 间隔号 division changes the comma reading, sending the credential from title to suffix": _Claim(1, ('family', 'given', 'suffix', 'title'), "1d45596e6fdb"), + "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle": + _Claim(1, ('_ambiguities', 'family', 'given'), "ca7b37af6cf8"), "fix(#367) a title no longer displaces a leading particle out of the leading position": _Claim(4, ('family', 'given', 'middle'), "ae299117dd60"), "fix(#380) a trailing vd after a family comma is the tussenvoegsel, not a post-nominal": - _Claim(1, ('family', 'suffix'), "081ce07f927b"), + _Claim(2, ('family', 'suffix'), "ec0d45289dc1"), "fix(#399) a maiden marker bounds the particle chain that swallowed it": _Claim(5, ('family', 'maiden'), "15ec75a89f07"), "fix(#360) mc moved into the never-given particles, so it folds into the family": @@ -1357,6 +1375,20 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('given', 'middle', 'suffix'), "228abe0f32ef"), "fix(#425) the bound-given reserve runs assign's peel over the joined view": _Claim(2, ('family', 'given', 'suffix'), "ef1ab03b617e"), + "fix(#424) the particle chain stops before the trailing numeral": + _Claim(1, ('_ambiguities', 'family', 'suffix'), "2c99162bc9cf"), + "fix(#424) the particle chain stops before a bare acronym with words to spare": + _Claim(1, ('_ambiguities', 'family', 'suffix'), "3e3aae6a5b4b"), + "fix(#424) a title-led chain before the numeral is the one name piece": + _Claim(1, ('_ambiguities', 'family', 'given', 'suffix'), "5b3a743f9e35"), + "fix(#424) accepted: a particle of the suffix vocabulary opening the trailing run is a suffix piece": + _Claim(1, ('_ambiguities', 'family', 'middle', 'suffix'), "a564b97f7162"), + "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle, the P4 example": + _Claim(1, ('_ambiguities', 'family', 'given'), "42b69cf1b320"), + "fix(#424) the maiden walk stops before the trailing numeral": + _Claim(1, ('_ambiguities', 'maiden', 'suffix'), "cbe5bdd97317"), + "fix(#424) a marker followed only by the numeral is just a word": + _Claim(1, ('_ambiguities', 'family', 'maiden', 'middle', 'suffix'), "aaf53040b071"), "fix(#369) the bound given-name join takes a particle-and-bound word, so no fork is reported": _Claim(1, ('_ambiguities',), "81cf02ffdb33"), "fix(#360) ste moved into the never-given particles with mc": @@ -1366,11 +1398,13 @@ def _claim(rule: dict) -> _Claim: }, "expected_since_2.1.0.toml": { "fix(#379) a tussenvoegsel after a family comma attaches to the family": - _Claim(11, ('family', 'middle'), "3acdb7d11969"), + _Claim(13, ('family', 'middle'), "973617235cda"), + "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle": + _Claim(1, ('_ambiguities', 'family', 'given'), "ca7b37af6cf8"), "fix(#367) a title no longer displaces a leading particle out of the leading position": _Claim(4, ('family', 'given', 'middle'), "ae299117dd60"), "fix(#380) a trailing vd after a family comma is the tussenvoegsel, not a post-nominal": - _Claim(1, ('family', 'suffix'), "081ce07f927b"), + _Claim(2, ('family', 'suffix'), "ec0d45289dc1"), "fix(#399) a maiden marker bounds the particle chain that swallowed it": _Claim(5, ('family', 'maiden'), "15ec75a89f07"), "fix(#360) mc moved into the never-given particles, so it folds into the family": @@ -1399,6 +1433,20 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('given', 'middle', 'suffix'), "228abe0f32ef"), "fix(#425) the bound-given reserve runs assign's peel over the joined view": _Claim(2, ('family', 'given', 'suffix'), "ef1ab03b617e"), + "fix(#424) the particle chain stops before the trailing numeral": + _Claim(1, ('_ambiguities', 'family', 'suffix'), "2c99162bc9cf"), + "fix(#424) the particle chain stops before a bare acronym with words to spare": + _Claim(1, ('_ambiguities', 'family', 'suffix'), "3e3aae6a5b4b"), + "fix(#424) a title-led chain before the numeral is the one name piece": + _Claim(1, ('_ambiguities', 'family', 'given', 'suffix'), "5b3a743f9e35"), + "fix(#424) accepted: a particle of the suffix vocabulary opening the trailing run is a suffix piece": + _Claim(1, ('_ambiguities', 'family', 'middle', 'suffix'), "a564b97f7162"), + "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle, the P4 example": + _Claim(1, ('_ambiguities', 'family', 'given'), "42b69cf1b320"), + "fix(#424) the maiden walk stops before the trailing numeral": + _Claim(1, ('_ambiguities', 'maiden', 'suffix'), "cbe5bdd97317"), + "fix(#424) a marker followed only by the numeral is just a word": + _Claim(1, ('_ambiguities', 'family', 'maiden', 'middle', 'suffix'), "aaf53040b071"), "fix(#369) the bound given-name join takes a particle-and-bound word, so no fork is reported": _Claim(1, ('_ambiguities',), "81cf02ffdb33"), "fix(#360) ste moved into the never-given particles with mc": diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 9e18ca01..66117a9f 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -593,6 +593,53 @@ def test_the_licence_does_not_lift_the_equality() -> None: assert (n.middle, n.family, n.suffix) == ("J.", "V", "") +def test_the_chain_and_the_walk_stop_where_the_peel_begins() -> None: + # #424: the third and fourth sites that asked "is this a suffix?" + # with the initial-vetoed test. Each reads as its ordinary twin + # ('John Smith V', 'John Smith Ma') reads. + for text, family, suffix in ( + ("John van der Berg V", "van der Berg", "V"), + ("John van der Berg X", "van der Berg", "X"), + ("abdul van der Berg V", "van der Berg", "V"), + ("John van der Berg Ma", "van der Berg", "Ma")): + n = parse(text) + assert (n.family, n.suffix) == (family, suffix), text + n = parse("John née Jones Smith V") + assert (n.maiden, n.suffix) == ("Jones Smith", "V") + n = parse("Jane Smith née Jones V") + assert (n.family, n.maiden, n.suffix) == ("Smith", "Jones", "V") + # the numeral must read as the suffix as the take leaves the name + # too: an initial before the marker vetoes the fork, so the walk + # keeps the V as maiden text rather than hand it to the family + n = parse("J. née Jones Smith V") + assert (n.family, n.maiden, n.suffix) == ("", "Jones Smith V", "") + # an unlisted abbreviation is as transparent to the leading + # particle as a listed title (H2 meets #367), so the acronym fork + # counts the same pieces in group and in assign + for abbrev in ("Xyz.", "Dr."): + n = parse(f"{abbrev} van Berg MA") + assert (n.given, n.family, n.suffix) == ("van", "Berg", "MA"), abbrev + n = parse(f"{abbrev} van Johnson") + assert (n.given, n.family) == ("van", "Johnson"), abbrev + n = parse(f"{abbrev} abdul John Smith") + assert (n.given, n.middle) == ("abdul John", ""), abbrev + # behind a title-and-particle word the chain takes the name's first + # word (#367), and the acronym it leaves has no words to spare for + # assign: the chain keeps it rather than leave it as the family + n = parse("Freiherr von Berg MA") + assert (n.title, n.family, n.suffix) == ("Freiherr", "von Berg MA", "") + # the numeral keeps its three pieces behind the same word, and the + # chain, now the one name piece, reads as 'Dr. Smith V' reads + n = parse("Freiherr von Richthofen V") + assert (n.given, n.family, n.suffix) == ("von Richthofen", "", "V") + n = parse("Dr. Smith V") + assert (n.given, n.family, n.suffix) == ("Smith", "", "V") + # the walk takes the numeral only: an acronym between the maiden + # name and the numeral is maiden text + n = parse("Jane Smith née Jones Ma V") + assert (n.maiden, n.suffix) == ("Jones Ma", "V") + + def test_the_numeral_fork_fires_on_the_last_piece_only() -> None: # The shared peel's own contract, pinned at the stage that owns # it: a numeral with a suffix behind it is a name word, for an diff --git a/tests/v2/test_regex_sync.py b/tests/v2/test_regex_sync.py index 98096309..f547a717 100644 --- a/tests/v2/test_regex_sync.py +++ b/tests/v2/test_regex_sync.py @@ -23,7 +23,9 @@ import pytest from nameparser.config import regexes as _config -from nameparser._pipeline import _assign, _post_rules, _tokenize, _vocab +from nameparser._pipeline import ( + _assign, _group, _post_rules, _tokenize, _vocab, +) from nameparser import _render @@ -44,8 +46,8 @@ def test_period_not_at_end_matches_config() -> None: def test_period_abbreviation_matches_config() -> None: source = _config.REGEXES["period_abbreviation"] - assert _assign._PERIOD_ABBREV.pattern == source.pattern - assert _assign._PERIOD_ABBREV.flags == source.flags + assert _group._PERIOD_ABBREV.pattern == source.pattern + assert _group._PERIOD_ABBREV.flags == source.flags def test_roman_numeral_matches_config() -> None: @@ -97,7 +99,9 @@ def test_initial_copies_agree_with_each_other_and_config() -> None: # completeness check below: adding a pattern without declaring its # source now fails here instead of being silently unpinned. _SOURCES: dict[tuple[str, str], str | None] = { - ("_assign", "_PERIOD_ABBREV"): "period_abbreviation", + ("_group", "_PERIOD_ABBREV"): "period_abbreviation", + ("_group", "_D"): None, + ("_group", "_PH"): None, ("_vocab", "_ROMAN"): "roman_numeral", ("_post_rules", "_EAST_SLAVIC"): "east_slavic_patronymic", ("_post_rules", "_EAST_SLAVIC_CYR"): "east_slavic_patronymic_cyrillic", @@ -117,7 +121,8 @@ def test_initial_copies_agree_with_each_other_and_config() -> None: ("_render", "_COMMA_CHAR"): None, } -_MODULES = {"_assign": _assign, "_post_rules": _post_rules, +_MODULES = {"_assign": _assign, "_group": _group, + "_post_rules": _post_rules, "_render": _render, "_tokenize": _tokenize, "_vocab": _vocab} diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 14a06915..ea50a80a 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -20,18 +20,23 @@ "Berg, Jan vd" "Berg, abd née Jones" "Berg, abdul V" +"Berg, abdul van" +"Berg, abdul vd" "Del Toro" "Dr. John van der Berg" "Dr. Juan Q. Xavier de la Vega III" "Dr. Smith, John" "Dr. abdul salam" "Esq. Smith" +"Freiherr von Berg MA" +"Freiherr von Richthofen V" "Gal·la Serra" "Garcia Juan Carlos" "Hans „Erster“ und “Zweiter” Müller" "Hassan Mohamad Ali" "Hassan, Mohamad Ahmad Ali" "J. Smith" +"J. née Jones Smith V" "J.R. Smith" "Jack Wei Ma" "Jane Smith (Nee)" @@ -39,8 +44,10 @@ "Jane Smith (née Jones)" "Jane Smith née Jones" "Jane Smith née Jones PhD" +"Jane Smith née V" "Jane Smith, née Jones" "Jane de la née Jones" +"Jane née Jones J. V" "Jane née Jones Smith" "Jane née Jr y Jones" "Jane van der Berg née" @@ -56,6 +63,7 @@ "John Smith J.u.n.i.o.r." "John Smith Jr." "John Smith M.A." +"John Smith Mc V" "John Smith PhD" "John Smith Q.W.E.R.T." "John Smith, LEED AP" @@ -63,10 +71,16 @@ "John Smith, MD,, Jr." "John Smith, PhD" "John Smith, V." +"John née Jones Smith Ma" +"John née Jones Smith V" +"John van Mc" "John van der Berg" +"John van der Berg Ma" "John van der Berg PhD" "John van der Berg Smith" +"John van der Berg V" "John van der Berg née Jones" +"John van der J. V" "John😀Smith" "Jones née" "Jong Anke de" @@ -120,6 +134,7 @@ "Vincent van Gogh van Beethoven" "Xyz. John Smith" "Xyz. Smith, John" +"Xyz. van Johnson" "abd Allah" "abd Allah Smith" "abd Allah Smith née Jones" diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index b7834665..c5533985 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -835,6 +835,87 @@ issue = "fix(#425) accepted: a bare ambiguous acronym the peel does not take joi name_regex = "(?i)^abdul\\s+ma\\s+smith$" fields = ["given", "middle"] +[[change]] +issue = "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle, the P4 example" +# 'Xyz. van Johnson': the rules.md#P4 example of the 'Esq. van Gogh' +# rule of this change (beside the fix(#367) rules, further down), +# same reading and fields. Separate rather than an alternation, per +# the guards' one-alternation-per-rule roster. +name_regex = "(?i)^xyz\\.\\s+van\\s+johnson$" +fields = ["given", "family"] + +[[change]] +issue = "fix(#424) accepted: the maiden walk keeps the numeral an initial before the marker vetoes" +# 'J. née Jones Smith V': rules.md#M2 -- the numeral is taken "read as +# the take would leave the name, the word before the numeral being +# then the word before the marker". 1.4.0 has no maiden support and +# read middle 'née Jones', last 'Smith', suffix 'V'; 2.x reads maiden +# 'Jones Smith V' with the J. before the marker vetoing the fork, as +# 'J. V' reads the V as a name. The fix(#274) rule's fields omit +# the suffix v1 read, so this accepted reading names its own. +name_regex = "(?i)^j\\.\\s+n[ée]e\\s+jones\\s+smith\\s+v$" +fields = ["middle", "family", "suffix", "maiden"] + +[[change]] +issue = "fix(#424) accepted: the chain keeps an acronym assign will not peel behind a title-and-particle word" +# 'Freiherr von Berg MA': rules.md#P2 -- the trailing suffix is read +# "over the pieces the chain leaves". #367 stops the leading-particle +# scan at a title-and-particle word, so the chain takes 'von Berg', +# and the acronym the fork counted with three pieces meets assign +# with two; the chain asks the peel again and takes it. first 'von +# Berg', last 'MA' -> family 'von Berg MA' (master's reading since +# 2.0). A rules.md example. Without this rule the name lands on the +# fields-only fix(suffix-routing) catch-all below -- the count is the +# tell. +name_regex = "(?i)^freiherr\\s+von\\s+berg\\s+ma$" +fields = ["given", "family"] + +[[change]] +issue = "fix(#424) a title-led chain before the numeral is the one name piece" +# 'Freiherr von Richthofen V': rules.md#P2 boundary -- the chain stops +# before the numeral, and the chain is then the one name piece, which +# reads as 'Dr. Smith V' reads. last 'von Richthofen V' -> given 'von +# Richthofen', suffix 'V'. A rules.md example. +name_regex = "(?i)^freiherr\\s+von\\s+richthofen\\s+v$" +fields = ["given", "family", "suffix"] + +[[change]] +issue = "fix(#424) accepted: a particle of the suffix vocabulary opening the trailing run is a suffix piece" +# 'John Smith Mc V': rules.md#P2 Accepted -- "where it opens the +# trailing run the chain stops before it as before any suffix word, +# and the peel takes it". middle 'Smith', last 'Mc V' -> family +# 'Smith', suffix 'Mc, V'. A rules.md example; the class was found +# unrecorded by the second docs review of #424. +name_regex = "(?i)^john\\s+smith\\s+mc\\s+v$" +fields = ["middle", "family", "suffix"] + +[[change]] +issue = "fix(#424) the particle chain stops before the trailing numeral" +# 'John van der Berg V': rules.md#P2 -- "a trailing suffix begins -- +# read as assign will read it (S2): a trailing roman numeral, or a +# bare acronym with words to spare, ends the chain as a suffix word +# does". last 'van der Berg V' -> family 'van der Berg', suffix 'V', +# as 'John Smith V' reads. Shipped since 1.x: the chain's stop asked +# with the suffix-piece test, whose initial veto does not see a bare +# V. A rules.md example; no differential corpus name has the shape. +# Without this rule the name lands on the fields-only +# fix(suffix-routing) catch-all below -- the count is the tell. +name_regex = "(?i)^john\\s+van\\s+der\\s+berg\\s+v$" +fields = ["family", "suffix"] + +[[change]] +issue = "fix(#424) accepted: the maiden walk keeps a bare acronym" +# 'John née Jones Smith Ma': rules.md#M2 -- "a bare acronym the peel +# would take with words to spare is maiden text all the same". middle +# 'née Jones', family 'Smith', suffix 'Ma' -> maiden 'Jones Smith Ma': +# v1 had no maiden support; 2.0 has read the name so since #274, and +# the walk deliberately leaves the acronym fork to assign (the count +# it needs includes the words the marker removes). The fix(#274) rule +# cannot carry it: its fields omit the suffix v1 read. A rules.md +# Accepted example, first witnessed here. +name_regex = "(?i)^john\\s+n[ée]e\\s+jones\\s+smith\\s+ma$" +fields = ["middle", "family", "suffix", "maiden"] + [[change]] issue = "fix(#360) ste moved into the never-given particles with mc" # 'Ste Marie': the second word of the vocabulary move, which the @@ -845,6 +926,22 @@ issue = "fix(#360) ste moved into the never-given particles with mc" name_regex = "(?i)^ste\\s+\\S+$" fields = ["given", "family"] +[[change]] +issue = "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle" +# 'Esq. van Gogh' (corpus_issues): rules.md#P4 -- "An unlisted +# abbreviation before the particle is as transparent as a listed +# title, since assign reads it as one (H2)". title 'Esq.', last 'van +# Gogh' -> given 'van', family 'Gogh', as 'Dr. van Gogh' reads since +# #367 (whose rule below records v1's titled-particle reading). #367 keyed the leading-particle exception on the +# first piece of the name by group's title test, which does not see +# H2's abbreviations; the scan asks assign's test now. Literal for the +# reason the #367 rule gives: a regex cannot ask whether a word is a +# title, and this one cannot ask whether it is unlisted. +# Without this rule the name lands on the fields-only +# fix(suffix-routing) catch-all below -- the count is the tell. +name_regex = "(?i)^esq\\.\\s+van\\s+gogh$" +fields = ["given", "family"] + [[change]] issue = "fix(#367) a title no longer displaces a leading particle out of the leading position" # 'Mr. Van Nguyen': a leading particle deliberately does not chain -- diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index 56e1936f..01c323f4 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -425,6 +425,80 @@ issue = "fix(#425) the bound-given reserve runs assign's peel over the joined vi name_regex = "(?i)^abdul\\s+smith\\s+(jr\\s+)?ma$" fields = ["given", "family", "suffix"] +[[change]] +issue = "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle, the P4 example" +# 'Xyz. van Johnson': the rules.md#P4 example of the 'Esq. van Gogh' +# rule of this change (beside the fix(#367) rules, further down), +# same reading and fields. Separate rather than an alternation, per +# the guards' one-alternation-per-rule roster. +name_regex = "(?i)^xyz\\.\\s+van\\s+johnson$" +fields = ["given", "family", "_ambiguities"] + +[[change]] +issue = "fix(#424) a title-led chain before the numeral is the one name piece" +# 'Freiherr von Richthofen V': rules.md#P2 boundary -- the chain stops +# before the numeral, and the chain is then the one name piece, which +# reads as 'Dr. Smith V' reads. family 'von Richthofen V' -> given +# 'von Richthofen', suffix 'V'. A rules.md example. +name_regex = "(?i)^freiherr\\s+von\\s+richthofen\\s+v$" +fields = ["given", "family", "suffix", "_ambiguities"] + +[[change]] +issue = "fix(#424) accepted: a particle of the suffix vocabulary opening the trailing run is a suffix piece" +# 'John Smith Mc V': rules.md#P2 Accepted -- "where it opens the +# trailing run the chain stops before it as before any suffix word, +# and the peel takes it". middle 'Smith', family 'Mc V' -> family +# 'Smith', suffix 'Mc, V'. A rules.md example; the class was found +# unrecorded by the second docs review of #424. +name_regex = "(?i)^john\\s+smith\\s+mc\\s+v$" +fields = ["middle", "family", "suffix", "_ambiguities"] + +[[change]] +issue = "fix(#424) the particle chain stops before the trailing numeral" +# 'John van der Berg V': rules.md#P2 -- "a trailing suffix begins -- +# read as assign will read it (S2): a trailing roman numeral, or a +# bare acronym with words to spare, ends the chain as a suffix word +# does". last 'van der Berg V' -> family 'van der Berg', suffix 'V', +# as 'John Smith V' reads. Shipped since 1.x: the chain's stop asked +# with the suffix-piece test, whose initial veto does not see a bare +# V. A rules.md example; no differential corpus name has the shape. +# The fork's report arrives with the reading. +name_regex = "(?i)^john\\s+van\\s+der\\s+berg\\s+v$" +fields = ["family", "suffix", "_ambiguities"] + +[[change]] +issue = "fix(#424) the particle chain stops before a bare acronym with words to spare" +# 'John van der Berg Ma': the rule above, on S2's other fork. family +# 'van der Berg Ma' -> family 'van der Berg', suffix 'Ma', as 'John +# Smith Ma' reads -- and as 1.4.0 read it, so this rule has no 1.4.0 +# twin: a 2.0 regression. A rules.md example; no differential corpus +# name has the shape. +name_regex = "(?i)^john\\s+van\\s+der\\s+berg\\s+ma$" +fields = ["family", "suffix", "_ambiguities"] + +[[change]] +issue = "fix(#424) the maiden walk stops before the trailing numeral" +# 'John née Jones Smith V': rules.md#M2 -- "takes the words after it +# -- up to any suffix word, or the trailing roman numeral assign +# reads as the suffix (S2) -- as the maiden name". maiden 'Jones +# Smith V' -> maiden 'Jones Smith', suffix 'V'. The walk's stop asked +# with the suffix-piece test too. No 1.4.0 twin: v1 had no maiden +# support, and the fix(#274) rule carries the name there. A rules.md +# example; no differential corpus name has the shape. +name_regex = "(?i)^john\\s+n[ée]e\\s+jones\\s+smith\\s+v$" +fields = ["suffix", "maiden", "_ambiguities"] + +[[change]] +issue = "fix(#424) a marker followed only by the numeral is just a word" +# 'Jane Smith née V': the peel is read from the marker, so the V has +# the piece before it the fork wants and reads as the suffix; nothing +# follows the marker but a suffix, the pass declines, and the marker +# stays a word -- as for 'Jane Smith née PhD'. maiden 'V' -> middle +# 'Smith', family 'née', suffix 'V', which is 1.4.0's reading, so +# this rule has no 1.4.0 twin. A rules.md example. +name_regex = "(?i)^jane\\s+smith\\s+n[ée]e\\s+v$" +fields = ["middle", "family", "suffix", "maiden", "_ambiguities"] + [[change]] issue = "fix(#369) the bound given-name join takes a particle-and-bound word, so no fork is reported" # 'Sheik Abu Bakar': the fields are byte-identical to this baseline @@ -459,6 +533,19 @@ issue = "fix(#399) a maiden marker bounds the particle chain: the geb. spelling" name_regex = "(?i)^(?![^\\n]*,)[^\\n]*\\b(de|del|den|der|di|do|dos|du|la|le|los|mc|van|vd|von|zu)\\b.*?\\sgeb\\..?\\s+\\S+$" fields = ["family", "maiden"] +[[change]] +issue = "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle" +# 'Esq. van Gogh' (corpus_issues): rules.md#P4 -- "An unlisted +# abbreviation before the particle is as transparent as a listed +# title, since assign reads it as one (H2)". family 'van Gogh' -> +# given 'van', family 'Gogh', with the PARTICLE_OR_GIVEN report, as +# 'Dr. van Gogh' reads since #367. #367 keyed the leading-particle +# exception on the first piece of the name by group's title test, +# which does not see H2's abbreviations; the scan asks assign's test +# now. Literal for the reason the #367 rule gives. +name_regex = "(?i)^esq\\.\\s+van\\s+gogh$" +fields = ["given", "family", "_ambiguities"] + [[change]] issue = "fix(#367) a title no longer displaces a leading particle out of the leading position" # 'Mr. Van Nguyen': a leading particle deliberately does not chain -- diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index 996274d8..fdb0a7f1 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -123,6 +123,80 @@ issue = "fix(#425) the bound-given reserve runs assign's peel over the joined vi name_regex = "(?i)^abdul\\s+smith\\s+(jr\\s+)?ma$" fields = ["given", "family", "suffix"] +[[change]] +issue = "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle, the P4 example" +# 'Xyz. van Johnson': the rules.md#P4 example of the 'Esq. van Gogh' +# rule of this change (beside the fix(#367) rules, further down), +# same reading and fields. Separate rather than an alternation, per +# the guards' one-alternation-per-rule roster. +name_regex = "(?i)^xyz\\.\\s+van\\s+johnson$" +fields = ["given", "family", "_ambiguities"] + +[[change]] +issue = "fix(#424) a title-led chain before the numeral is the one name piece" +# 'Freiherr von Richthofen V': rules.md#P2 boundary -- the chain stops +# before the numeral, and the chain is then the one name piece, which +# reads as 'Dr. Smith V' reads. family 'von Richthofen V' -> given +# 'von Richthofen', suffix 'V'. A rules.md example. +name_regex = "(?i)^freiherr\\s+von\\s+richthofen\\s+v$" +fields = ["given", "family", "suffix", "_ambiguities"] + +[[change]] +issue = "fix(#424) accepted: a particle of the suffix vocabulary opening the trailing run is a suffix piece" +# 'John Smith Mc V': rules.md#P2 Accepted -- "where it opens the +# trailing run the chain stops before it as before any suffix word, +# and the peel takes it". middle 'Smith', family 'Mc V' -> family +# 'Smith', suffix 'Mc, V'. A rules.md example; the class was found +# unrecorded by the second docs review of #424. +name_regex = "(?i)^john\\s+smith\\s+mc\\s+v$" +fields = ["middle", "family", "suffix", "_ambiguities"] + +[[change]] +issue = "fix(#424) the particle chain stops before the trailing numeral" +# 'John van der Berg V': rules.md#P2 -- "a trailing suffix begins -- +# read as assign will read it (S2): a trailing roman numeral, or a +# bare acronym with words to spare, ends the chain as a suffix word +# does". last 'van der Berg V' -> family 'van der Berg', suffix 'V', +# as 'John Smith V' reads. Shipped since 1.x: the chain's stop asked +# with the suffix-piece test, whose initial veto does not see a bare +# V. A rules.md example; no differential corpus name has the shape. +# The fork's report arrives with the reading. +name_regex = "(?i)^john\\s+van\\s+der\\s+berg\\s+v$" +fields = ["family", "suffix", "_ambiguities"] + +[[change]] +issue = "fix(#424) the particle chain stops before a bare acronym with words to spare" +# 'John van der Berg Ma': the rule above, on S2's other fork. family +# 'van der Berg Ma' -> family 'van der Berg', suffix 'Ma', as 'John +# Smith Ma' reads -- and as 1.4.0 read it, so this rule has no 1.4.0 +# twin: a 2.0 regression. A rules.md example; no differential corpus +# name has the shape. +name_regex = "(?i)^john\\s+van\\s+der\\s+berg\\s+ma$" +fields = ["family", "suffix", "_ambiguities"] + +[[change]] +issue = "fix(#424) the maiden walk stops before the trailing numeral" +# 'John née Jones Smith V': rules.md#M2 -- "takes the words after it +# -- up to any suffix word, or the trailing roman numeral assign +# reads as the suffix (S2) -- as the maiden name". maiden 'Jones +# Smith V' -> maiden 'Jones Smith', suffix 'V'. The walk's stop asked +# with the suffix-piece test too. No 1.4.0 twin: v1 had no maiden +# support, and the fix(#274) rule carries the name there. A rules.md +# example; no differential corpus name has the shape. +name_regex = "(?i)^john\\s+n[ée]e\\s+jones\\s+smith\\s+v$" +fields = ["suffix", "maiden", "_ambiguities"] + +[[change]] +issue = "fix(#424) a marker followed only by the numeral is just a word" +# 'Jane Smith née V': the peel is read from the marker, so the V has +# the piece before it the fork wants and reads as the suffix; nothing +# follows the marker but a suffix, the pass declines, and the marker +# stays a word -- as for 'Jane Smith née PhD'. maiden 'V' -> middle +# 'Smith', family 'née', suffix 'V', which is 1.4.0's reading, so +# this rule has no 1.4.0 twin. A rules.md example. +name_regex = "(?i)^jane\\s+smith\\s+n[ée]e\\s+v$" +fields = ["middle", "family", "suffix", "maiden", "_ambiguities"] + [[change]] issue = "fix(#369) the bound given-name join takes a particle-and-bound word, so no fork is reported" # 'Sheik Abu Bakar': the fields are byte-identical to this baseline @@ -177,6 +251,19 @@ issue = "fix(#399) a maiden marker bounds the particle chain: a native-script ma name_regex = "(?i)\\b(de|del|den|der|di|do|dos|du|la|le|los|mc|van|vd|von|zu)\\b.*?\\s\u65e7\u59d3\\s+\\S+$" fields = ["family", "maiden"] +[[change]] +issue = "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle" +# 'Esq. van Gogh' (corpus_issues): rules.md#P4 -- "An unlisted +# abbreviation before the particle is as transparent as a listed +# title, since assign reads it as one (H2)". family 'van Gogh' -> +# given 'van', family 'Gogh', with the PARTICLE_OR_GIVEN report, as +# 'Dr. van Gogh' reads since #367. #367 keyed the leading-particle +# exception on the first piece of the name by group's title test, +# which does not see H2's abbreviations; the scan asks assign's test +# now. Literal for the reason the #367 rule gives. +name_regex = "(?i)^esq\\.\\s+van\\s+gogh$" +fields = ["given", "family", "_ambiguities"] + [[change]] issue = "fix(#367) a title no longer displaces a leading particle out of the leading position" # 'Mr. Van Nguyen': a leading particle deliberately does not chain --