diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 718975f5..4a3c768a 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -370,6 +370,12 @@ Excluded (SUFFIX_ACRONYMS / SUFFIX_WORDS — the esq dual membership, deliberate Fixed with one predicate, `reads_as_suffix`: a suffix piece, or the trailing roman numeral assign's fork takes — and that fork is now `_vocab.is_trailing_numeral_suffix`, lifted out of assign so the two stages share its definition instead of the reserve carrying a copy (the `_ROMAN` pattern moved with it; the regex-sync pin follows it). The reserve counts a piece out when it reads as a suffix, and the join declines the piece it would take when it does — the same decline it already made for a marker. The mirror is assign's fork condition for condition: the numeral is the LAST piece and not the first name piece — not "the piece before it is not a title", which the first draft wrote and the design-docs review falsified, `jr` being title vocabulary as well as a suffix, so `abdul Smith Jr V` lost its family exactly as before — and the piece before it is read as the join would leave it, the joined pair's first token being the bound word (`abdul J. V`, the design-docs review's other find, otherwise suppressed the fork for the reserve alone and read given 'abdul J.', family ''; the reserve now declines, and assign, seeing the unjoined pieces, reads given 'abdul', middle 'J.', family 'V', exactly as it reads `John J. V`). Two more the code review found: assign drops the group-flagged credential pieces (the Ph. D. merge) from its walk at any position before the trailing peel, so a numeral can be last in that walk without being the last piece — `abdul Smith V Ph. D.` lost its family until "last" and "the piece before" were read over the pieces assign keeps — pinned at the piece level only, because a trailing split credential is the shape the 1.4.0 ledger's `[[never]]` entry refuses to explain any diff on, and v1's reading of that name carried the #401 family loss; the example stayed out of the rules corpus rather than the never-entry being narrowed for one bound-word spelling; and the post-comma walk has no numeral fork at all (a trailing numeral there is read by a lenient last-of-two rule, or as a middle initial with a third part), so a mirror applied under LENIENT declined `Berg, abdul V` and `Berg, abdul V, jr` against every baseline with nothing classifying it — the numeral branch is scoped to STRICT, and after a family comma the reserve counts suffix pieces alone, as it always did. So `abdul Smith V Jr` still joins and assign reads family 'V', and `abdul V Smith` still joins an initial, which is a name word. What is NOT mirrored: S2's other positional fork, the bare ambiguous acronym. Where the join leaves two pieces no family is lost — `abdul Smith Ma` reads family 'Ma' where `John Smith Ma` reads suffix 'Ma', assign keeping the acronym as the family of a two-piece name — but with a suffix word between, the join leaves three, assign peels the acronym and then the suffix, and the family is gone: `abdul Smith Jr Ma` reads given 'abdul Smith', family '', suffix 'Jr, Ma' where `John Smith Jr Ma` keeps family 'Smith'. Pre-existing and byte-identical before this fix, and the whole of what remains of #401's shape on the main walk: a review sweep of the STRICT path found 132 family-losing shapes, every one ending in a bare ambiguous acronym. Left for #425 rather than widened into here, and framed there as sharing assign's peel over a post-join VIEW rather than mirroring a second fork by hand: with a view the count is the view's length, the remap and the first-piece carve-out fall away, and the other two sites that stop at a suffix with the same initial veto — P2's chain and M2's walk, which take a trailing numeral the same way (#424) — would ask the question once. The walk was also computed once rather than per piece after review measured the per-piece rebuild as quadratic (13x growth for 4x the input against the benchmark's 6x bound, on a shape no benchmark unit led with); `bound_given` is that unit now. Readings: `abdul Smith V` → given 'abdul', family 'Smith', suffix 'V', the SUFFIX_OR_NAME report unchanged; `abdul Jr Smith Berg` → given 'abdul', middle 'Jr Smith', where `John Jr Smith Berg` puts it; `abdul Ph. D. Smith Berg` → given 'abdul', middle 'Smith', suffix 'Ph. D.' — the SUFFIX field is 1.4.0's again, the given is not: v1's fix_phd extracted the credential before parsing and the join then took 'Smith', reading given 'abdul Smith' — which the issue reports correctly, and then mislabels its own proposed reading as "1.4.0's reading restored". The decline is chosen over reaching past the credential for the next name word, because P5 joins "the word after it": a join that skipped a suffix would read a shape no one writes as if the suffix were not there, and the marker decline already sets the precedent that the join takes the word after it or nothing. A judgement, recorded as one; `Berg, abdul Jr Smith` → suffix 'Jr'; and `Sir abdul V`, the shape #369's licence had widened #401 onto, reads suffix 'V'. #369's licence-local suffix clause is retired, subsumed by the general decline. `bound_given_join_sees_only_the_surviving_name` re-pins to given 'abd', middle 'Jr Smith', as the #418 entry anticipated. No differential corpus name has either shape — measured: the gate was byte-identical at all three baselines before the rules.md examples went in — so the examples carry the shapes into the rules corpus, where they move against every baseline and are classified there. The 1.4.0 `fix(#401)` rule lists three fields where the 2.x rules list two, because 1.4.0 did not even read the V as a suffix; a draft with two fields handed the name to the fields-only suffix-routing catch-all, whose capped printout hid it, and this entry briefly called #401 a 2.0 regression on that inert measurement — the design-docs review caught it. EXPLAINED NOTHING is not byte-identical. +- 2026-08-22 #425 — the reserve runs assign's peel over the view the join would leave. #423 had the reserve re-derive assign's trailing peel by hand — the flagged-credential drop, the walk's start, the numeral fork's conditions, the preceding piece remapped to "as the join would leave it" — with each seam held by a pin, and the bare-ambiguous-acronym fork not mirrored at all: `abdul Smith Jr Ma` read given 'abdul Smith', family '', suffix 'Jr, Ma' where `John Smith Jr Ma` keeps family 'Smith', the whole residue of #401's shape on the main walk (every family-losing bound-word shape in the sweep below contains one). The altitude review's framing is taken: the peel is one function, `_peel_trailing` over the list `_peel_walk` builds (the walk's input defined once too, after the /simplify pass found it spelled at three sites), housed in group beside the piece predicates because assign imports those and group cannot import assign; assign calls it and reports what it returns (the numeral fork's piece, the acronym picks, in the order it always reported them — byte-identical over the suite and the three gates); and the reserve builds the VIEW merge() would leave — the pair one piece, its tags the union — and calls the same function. `BoundJoin`'s "the value IS the threshold" trick survives restated as post-join name pieces: STRICT 2, LENIENT 1. The remap and the `k == first_name_k` carve-out are gone, the joined piece being two tokens and never a suffix piece. + + One principle came out of writing the ledger that the view alone did not state. `abdul Smith Ma` had read given 'abdul Smith', family 'Ma' since 2.0, and the #423 entry called that the lossless shape; the 1.4.0 gate's catch-all count said otherwise — 1.4.0 reads first 'abdul', last 'Smith', suffix 'Ma', as `John Smith Ma` reads, and the fields-only catch-all had absorbed the diff since the example went in (the count is the tell, per the #401 entry). Unjoined, the acronym has words to spare and peels as a credential; joined, the view is two pieces and the fork keeps it as the family — the join was CHANGING what assign reads. So the rule is stated as P5 now has it: the join joins two name words into one and changes nothing else, which the code asks directly: the pieces the peel takes are the same either way (a count proxy, "the view leaves exactly one name word fewer", was the first spelling; the altitude review pointed out it held only because the peel is a tail walk, and the comparison states the rule). That comparison subsumes the unjoined-piece check #423 added for `abdul V` (unjoined the peel takes the V, joined nothing) and restores `abdul Smith Ma` to 1.4.0's reading; `abdul Smith Berg Ma` keeps its join, the acronym peeling either way. Every #423 pin holds. Measured: the 1.4.0 gate is byte-identical at the code change, before the examples went in — the three `Ma`-final examples are v1 parity, the catch-all back at 15 — and `abdul Smith Jr Ma` and `abdul Smith Ma` move against 2.0.0 and 2.1.0, classified there; no hand corpus has the shapes. The population the corpus cannot see, from the design-docs review's sweep of 67,860 constructed main-walk shapes (three bound words × twelve words, none a title word, × tails of one to four; no comma shapes): before this change 2,160 lost the family and every one contained a bare ambiguous acronym (864 with further suffixes behind it, so "ending in" was an artefact of #423's shorter sweep); after it none does, and all 67,860 match the family and suffix their ordinary-given twin reads. 5,166 move against master in all — 3,006 where the acronym was the family and is now the suffix (`Sir abdul Smith Ma`), 2,160 where the family is restored. The title-word class below was outside that population and moves after a family comma too (`Berg, abdul Sir`); so was the licence class at the end of this entry. + + Three things the principle does not reach, recorded so nobody applies it there. The post-comma LENIENT path is untouched, and NOT for the reason first written here ("its walk has no peel" — it reads a trailing numeral by the lenient last-of-two rule, as the #423 entry says): the family is fixed there and the pair is the given whatever follows, so the principle is not applied, and `Berg, abdul V` keeps given 'abdul V' though the V would be the suffix unjoined — 1.4.0 parity, and exactly the regression #423 measured when a mirror was applied there; P5 now carries it as Accepted. The vocabulary decline is the UNAMBIGUOUS vocabulary: a bare ambiguous acronym the peel does not take is a name word, so `abdul Ma Smith` reads given 'abdul Ma' (1.4.0 read middle 'Ma' — a 2.x difference no corpus witnesses, now an Accepted example). And the principle is about the suffix reading, not the report: `Sheik abdul Ma` reports no SUFFIX_OR_NAME where `Sheik John Ma` does, the #369 precedent — a bound word read as the bound word is not a fork. One more class moves, found by the code review and then measured against 1.4.0: the 2.x reserve had excluded TITLE pieces from its count — an unrecorded deviation from v1, whose join counted suffixes out and nothing else — and the shared peel, following assign, reads a mid-name title word as the name word it is. `Berg, abdul Sir` (given 'abdul', middle 'Sir' since 2.0) and `abdul Smith Sir` (declined since 2.0) read given 'abdul Sir' and given 'abdul Smith', family 'Sir' — 1.4.0's readings, both; `abdul Sir Smith Berg` never stopped reading given 'abdul Sir'. No corpus name has the shapes; P5 says it now and `abdul Sir Smith Berg` is an example. What the review actually caught was the tag artefact that exclusion had been hiding: the conjunction merge derives a `title` piece tag for a chain such as "Sheikh and Ahmad", merge()'s union handed it to the pair, and assign peeled the pair as a leading title — `abdul Sheikh and Ahmad Bakar Smith` read title 'abdul Sheikh and Ahmad' on 2.0 and 2.1 (v1 read it as the given), and `abdul Sheikh and Ahmad Bakar`, which the exclusion had kept unjoined, would have too once it went. The bound join drops the tag, and the view drops it the same way — the pair is a given name whatever tag the word carried. One more class the comparison moves, found by the code review's classification of every residual master-to-branch diff (12 of 17,640 constructed names, one shape; 360 of 360 in a dedicated sweep): behind a given-name title, a numeral after an initial-shaped piece. `Sir abdul J. V` read given 'abdul J.', suffix 'V' since #369 — the fork suppressed unjoined, fired joined, and the licence's threshold of one let the join through — and the comparison declines it: given 'abdul', middle 'J.', family 'V', exactly as `Sir John J. V`. Pinned at both levels; it was the only shape a looser comparison let through under mutation at the test review. #424 — P2's chain and M2's walk taking a trailing numeral — is the remaining asker of the same question and now has `_peel_trailing` to ask it of. + Excluded (BOUND_GIVEN_NAMES): - mohamad — a standalone given name in its own right; binding it would eat the middle name. diff --git a/docs/design/rules.md b/docs/design/rules.md index 3242b376..8f6acd01 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -254,15 +254,23 @@ P5. Rationale: some given-name words are incomplete alone — "abdul" not among the words to spare: they leave the name, so counting them asks the question about a name that will not exist. The join never absorbs a marker standing as a word of its own — a marker - is not a name word (M2) — nor a word of the suffix vocabulary - (S2), wherever position will then place it; a marker left as a - word that a particle join (P2) has already taken travels with - that join. The reserve counts the same words out, and one more - that position decides: a trailing roman numeral that assign - reads as the suffix (S2) is no word to spare, and is not joined - — read as assign will read it, after the join, with the bound - word first. After a family comma there is no such fork to read: - the reserve there counts suffix words alone. + is not a name word (M2) — nor a word of the unambiguous suffix + vocabulary (S2), wherever position will then place it; a bare + ambiguous acronym is a name word wherever the peel does not take + it; a marker left as a word that a particle join (P2) has already + taken travels with that join. A title word standing in the name + is a name word (H3) and joins like one: the pair is a given name + whatever tag the word carried. What there is to spare is what + assign will leave: the join is tried on the pieces as it would + leave them, assign's trailing peel (S2) is read over that, and + the name words it leaves are the words to spare — a trailing + roman numeral, or a bare acronym with words behind it, is no + word to spare. The join joins two name words into one and + changes no suffix reading: a word the peel reads as a suffix + unjoined must read so joined, or the join declines. After a + family comma the family is fixed and the joined pair is the given + whatever follows, so the reserve there reads no peel: the join + stands whenever the word after the bound word is a name word. "abdul salam ahmed salem" → given="abdul salam" "abd Allah Smith" → given="abd Allah" "Salam, abd Allah" → given="abd Allah" @@ -280,7 +288,11 @@ P5. Rationale: some given-name words are incomplete alone — "abdul" "abdul Smith V" → family="Smith" "abdul Smith V" → suffix="V" "abdul Smith Jr V" → family="Smith" - "Berg, abdul V" → given="abdul V" · boundary + "abdul Smith Jr Ma" → family="Smith" + "abdul Smith Jr Ma" → suffix="Jr, Ma" + "abdul Smith Ma" → suffix="Ma" + "abdul Smith Berg Ma" → family="Berg" · boundary + "abdul Sir Smith Berg" → given="abdul Sir" "abdul Jr Smith Berg" → given="abdul" "abdul Jr Smith Berg" → middle="Jr Smith" "abdul Ph. D. Smith Berg" → suffix="Ph. D." @@ -289,6 +301,12 @@ P5. Rationale: some given-name words are incomplete alone — "abdul" "abd Allah Smith née Jones" → given="abd Allah" "abd née Jones" → given="abd" "Berg, abd née Jones" → suffix="abd" + Accepted: after a family comma the join stands though, unjoined, + assign would read the word it takes as the suffix — the family + is fixed there and the pair is the given; and a bare ambiguous + acronym the peel does not take joins as any name word does. + "Berg, abdul V" → given="abdul V" + "abdul Ma Smith" → given="abdul Ma" Accepted: a given-name title plus a bound pair leaves the family empty, as H1 accepts for "Sir John" — the input names no family. Under a family-first order the joined pair is the family and the @@ -401,7 +419,7 @@ S2. Rationale: generational suffixes and credentials are recognized "Jack Wei Ma" → suffix="Ma" "Jack Wei Ma" → ambiguities=("suffix-or-name",) "Smith Jr." → family="" - implemented: nameparser/_pipeline/_classify.py, nameparser/_pipeline/_vocab.py + implemented: nameparser/_pipeline/_classify.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_vocab.py S3. Rationale: credentials are often written run together with periods; the chunks between the periods are what carry the diff --git a/docs/release_log.rst b/docs/release_log.rst index 5ef0a617..2abea1ec 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -39,6 +39,8 @@ Release Log - Fix a bound given name joining a suffix as "the word after it": ``"abdul Jr Smith Berg"`` read given ``abdul Jr``, and where the suffix was the split credential the bound word joined *into* it -- ``"abdul Ph. D. Smith Berg"`` read suffix ``abdul Ph. D.``, a 2.0 regression -- 1.4.0 extracted the credential before parsing and read suffix ``Ph. D.`` with given ``abdul Smith``. The join now declines a suffix piece the way it already declines a maiden marker: ``"abdul Jr Smith Berg"`` reads given ``abdul``, middle ``Jr Smith`` (where ``"John Jr Smith Berg"`` puts it), ``"abdul Ph. D. Smith Berg"`` reads given ``abdul``, middle ``Smith``, suffix ``Ph. D.`` -- the suffix field restored, the given deliberately not, since the join takes the word after it or nothing rather than reaching past a credential -- and ``"Berg, abdul Jr Smith"`` reads suffix ``Jr``. Both shapes move against every baseline once the rules examples carry them; no name of the differential corpora had either (closes #421) + - Fix a bound given name joining past a credential that the suffix rule then takes, leaving no family: ``"abdul Smith Jr Ma"`` read given ``abdul Smith`` with no family, where ``"John Smith Jr Ma"`` reads family ``Smith`` with suffix ``Jr, Ma``. The join's reserve had re-derived the suffix rule's trailing walk by hand and left out one of its two forks -- a bare acronym such as ``Ma`` is a credential only with words to spare -- so it counted the acronym as a name word, joined, and the suffix rule then peeled it. The two stages now share one walk: the reserve tries the join, runs the suffix rule's own peel over the pieces as the join would leave them, and joins only when two name words became one and nothing else changed. ``"abdul Smith Jr Ma"`` reads family ``Smith``, suffix ``Jr, Ma``; ``"abdul Smith Ma"`` reads family ``Smith``, suffix ``Ma`` rather than joining the pair and keeping the credential as the family -- both as 1.4.0 read them and as ``John`` reads in their place; ``"abdul Smith Berg Ma"`` keeps its join, and a title word standing after the bound word joins again as it did in 1.4.0 (``"Berg, abdul Sir"`` reads given ``abdul Sir``) without the pair ever being read as a title (``"abdul Sheikh and Ahmad Bakar"`` reads given ``abdul Sheikh and Ahmad``). No name of the differential corpora has the shapes; the ``Ma`` rules examples carry the fix, byte-identical to 1.4.0 and moving against 2.0.0 and 2.1.0, and the title-word shapes are pinned in tests (closes #425) + - Fix a name opening with a particle that is *never* a given name being split at the particle under a family-first name order -- ``Policy(name_order=FAMILY_FIRST)`` and ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)`` alike, and identically: ``"de Mesnil"`` read as family ``de``, given ``Mesnil``, and ``"de la Vega"`` as family ``de``, given ``la Vega``. Each is now the whole surname, as it has always been in the default order. The rule enforcing it asked for the particle by the ``GIVEN`` role, which under a family-first order belongs to the token *after* the particle, so the test read the wrong word and declined. It now also asks by position -- the piece that opens the name -- so both shapes of the same rule are caught: where such a particle stands alone as a piece, either opening the name or in the given position, the name is left with no given name at all, the given and the middles folding into the family. Standing *alone* is the whole of it, and the rule claims nothing wider: ``"Juan de la Vega"`` under ``FAMILY_FIRST`` still reports given ``de la Vega``, because there the particle chained onto the words after it rather than standing alone, and a bare ``"de"`` with nothing to fold into is still reported as the given name. The decision behind the fix: a word that can never be a given name leaves ``name_order`` nothing to decide, so declaring family-first is not a reason to make ``de`` a surname on its own. A leading particle that *may* be a given name is genuinely order-dependent and is untouched -- ``"van Gogh"`` still reads as family ``van``, given ``Gogh`` under both family-first orders. This is also what gives ``Lexicon.particles_ambiguous`` an effect outside the default order: taking a word out of it now changes the parsed fields under a family-first order, where before it moved only the ambiguity report. Seven of the 751 differential corpus names move, the same seven under each family-first order; default-order output is byte-identical over all 751, at the 1.4.0, 2.0.0 and 2.1.0 differential baselines alike (closes #359) - Fix a family name made only of particle words reporting no base on the 2.0 API, so the surname vanished from ``family_base`` and, on both APIs, from the initials: ``parse("Anh Do")`` gave family ``Do`` with ``family_base`` ``''`` and initials ``A.``, and under ``Policy(name_order=FAMILY_FIRST)`` ``"Del Toro"`` gave family ``Del`` the same way. The empty base was a 2.0 regression rather than a longstanding bug -- 1.4.0's own guard kept ``HumanName("Anh Do").last_base`` at ``Do``, and the facade has kept it right throughout; what 1.4.0 and 2.1 shared was the missing initial. A particle earns its name by joining forward to the word it modifies, so a particle standing alone in a name part is not doing a particle's work there and reads as an ordinary name word: it anchors the base, leaves the particles view, and contributes an initial. ``"Anh Do"`` is now base ``Do``, initials ``A. D.``; ``"Juan van der"`` is base ``van der``, initials ``J. v. d.``; ``"Nguyen, Van Le"`` initials ``V. L. N.`` where the middle name used to be dropped. Position decides this, not vocabulary -- whether the word is borne as a surname somewhere does not enter into it, which reverses the reasoning ``rules.md#R2`` carried before. The invariant it exists to hold: a non-empty family always has a non-empty base, because a particle needs a base to attach to. Where the particles DO join a name word nothing changes -- ``"Juan de la Vega"`` keeps base ``Vega``, particles ``de la`` and initials ``J. V.``. The parse fields themselves do not move: ``title``, ``given``, ``middle``, ``family``, ``suffix``, ``nickname`` and ``maiden`` are byte-identical over all 751 differential corpus names in all three name orders, and only the derived views and the initials change -- which is why the differential harness, which compares those seven roles, reports no diffs for this at any baseline. One consequence to know about: where the whole family is particles AND a tussenvoegsel was attached to it (#379), the two APIs order the initials differently -- ``parse("der, y van").initials()`` is ``y. d. v.`` against the facade's ``y. v. d.`` -- because ``initials()`` reads tokens in written order while the family FIELD renders folded words first. That affects 87 constructed inputs and no corpus name; it is a pre-existing ordering gap in ``initials()`` that this change makes visible, tracked at #408 (closes #385, closes #402) diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index 316dc2d3..a1b68cb9 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -35,11 +35,10 @@ import re from nameparser._pipeline._vocab import ( - effective_script, is_suffix_lenient, is_trailing_numeral_suffix, - resolve_script_set, + effective_script, is_suffix_lenient, resolve_script_set, ) from nameparser._pipeline._group import ( - _is_suffix_piece, _is_title_piece, + _is_suffix_piece, _is_title_piece, _peel_trailing, _peel_walk, ) from nameparser._pipeline._state import ( ParseState, PendingAmbiguity, Structure, WorkToken, @@ -190,7 +189,7 @@ def _assign_main(seg_idx: int, state: ParseState, flagged = [k for k in rest if "suffix" in ptags[k]] for k in flagged: _set_roles(tokens, pieces[k], Role.SUFFIX) - rest = [k for k in rest if "suffix" not in ptags[k]] + rest = _peel_walk(n, ptags) if not rest: return None # rules.md#N3: "a name that is only a nickname and one name word @@ -203,60 +202,26 @@ def _assign_main(seg_idx: int, state: ParseState, _set_roles(tokens, pieces[rest[0]], Role.FAMILY) return None # peel the trailing suffix run: k = first index in rest from which - # every piece is a strict suffix (v1's are_suffixes tail rule, with - # the roman-numeral special: a final roman numeral after a - # non-initial piece is a suffix) - # every bare ambiguous acronym the peel had to resolve -- one - # coin-flip each, in either direction, so this collects rather than - # overwrites. Deferred to after assignment because the wording reads - # the role back, and which role "not peeled" means depends on - # name_order. (The roman-numeral fork needs no such deferral and is - # reported at its trigger below.) - ambiguous_picks: list[tuple[int, ...]] = [] - k = len(rest) - while k > 0: - piece = pieces[rest[k - 1]] - tags = ptags[rest[k - 1]] - if _is_suffix_piece(piece, tags, tokens): - k -= 1 - continue - # Shared with group's bound-given reserve (#401); the - # docstring carries the is_initial_shaped reasoning (#320). - if (k == len(rest) and k >= 2 and len(piece) == 1 - and is_trailing_numeral_suffix( - tokens[piece[0]].text, - tokens[pieces[rest[k - 2]][0]].text)): - # a trailing single letter is a name part unless it happens - # to be a roman numeral -- and V/X/I are ordinary middle - # initials, so taking it as a suffix is a call, not a fact - ambiguities.append(PendingAmbiguity( - AmbiguityKind.SUFFIX_OR_NAME, - f"{tokens[piece[0]].text!r} is a roman numeral, so it " - f"reads as a generational suffix; any other single " - f"letter there would be a middle initial", - tuple(piece))) - k -= 1 - continue - # A bare ambiguous acronym ("MA", not "M.A.") is a credential - # only when peeling it still leaves a given AND a family name. - # With two pieces, "one of them is a credential" is the less - # likely reading, so it stays the family name -- "Jack MA" is a - # person, "John Smith MA" is a person with a degree. This is - # v1's reserve_last narrowed to the ambiguous set: 2.0 - # deliberately peels UNambiguous suffixes even when nothing is - # left ("Smith PhD" -> suffix, a classified fix), because there - # the vocabulary is not in doubt. - bare_ambiguous = (len(piece) == 1 - and "vocab:suffix-ambiguous" in tokens[piece[0]].tags) - # k < 2 means it is the only piece left, which is not the fork - # this reports. - if bare_ambiguous and k >= 2: - ambiguous_picks.append(tuple(piece)) - if k >= 3: # peeling still leaves given + family - k -= 1 - continue - break - name_pieces, suffix_pieces = rest[:k], rest[k:] + # every piece is a suffix. The walk is group's _peel_trailing since + # #425 -- one walk, shared with the bound-given reserve, and + # documented there. Every bare ambiguous acronym it had to resolve + # is one coin-flip each, in either direction, so the report + # collects rather than overwrites. Deferred to after assignment + # because the wording reads the role back, and which role "not + # peeled" means depends on name_order. (The roman-numeral fork + # needs no such deferral and is reported here.) + peeled = _peel_trailing(rest, pieces, ptags, tokens) + if peeled.numeral is not None: + # a trailing single letter is a name part unless it happens + # to be a roman numeral -- and V/X/I are ordinary middle + # initials, so taking it as a suffix is a call, not a fact + ambiguities.append(PendingAmbiguity( + AmbiguityKind.SUFFIX_OR_NAME, + f"{tokens[peeled.numeral[0]].text!r} is a roman numeral, so " + f"it reads as a generational suffix; any other single " + f"letter there would be a middle initial", + peeled.numeral)) + name_pieces, suffix_pieces = rest[:peeled.names], rest[peeled.names:] if not name_pieces and suffix_pieces: # everything suffix-shaped after titles: first one is the name name_pieces, suffix_pieces = suffix_pieces[:1], suffix_pieces[1:] @@ -271,7 +236,7 @@ def _assign_main(seg_idx: int, state: ParseState, _set_roles(tokens, pieces[piece_idx], roles[pos]) for piece_idx in suffix_pieces: _set_roles(tokens, pieces[piece_idx], Role.SUFFIX) - for piece in ambiguous_picks: + for piece in peeled.picks: # every pick is in rest, so the loops above just gave it a role token = tokens[piece[0]] assert token.role is not None diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index db28201a..c447f6be 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -18,7 +18,10 @@ content, which M2's pieces walk cannot reach because extract's content never enters pieces); each is cited at its code below. Also implements rule P5 (cited below at the bound-given join) and ports -the "Ph. D."-split merge (v1 fix_phd; decisions.md#phd-merge). +the "Ph. D."-split merge (v1 fix_phd; decisions.md#phd-merge). Houses +the S2 trailing peel (_peel_walk, _peel_trailing), a piece-level walk +that assign applies and P5's reserve reads over the view the join +would leave (#425). """ from __future__ import annotations @@ -26,6 +29,7 @@ import dataclasses from collections.abc import Sequence, Set from enum import IntEnum +from typing import NamedTuple from nameparser._lexicon import _title_key from nameparser._pipeline._state import ( @@ -51,12 +55,16 @@ class BoundJoin(IntEnum): """v1 _join_bound_first_name's reserve_last, as the three states it - actually has. IntEnum: the value IS the non_suffix threshold, so - the >= comparison below reads unchanged.""" + actually has. IntEnum: the value IS the number of name pieces + assign's peel must leave in the JOINED view for the join to stand + (#425), so the >= comparison below reads unchanged. Post-comma no + peel is run -- the pair alone is that one piece -- and DISABLED + is a mode, never compared: as a threshold 0 would join everything, + which is why the block is entered on identity first.""" DISABLED = 0 # the FAMILY_COMMA family segment (v1 never joined it) - LENIENT = 2 # FAMILY_COMMA's post-comma segment (reserve_last=False) - STRICT = 3 # main segments (reserve_last=True: keep a family piece) + LENIENT = 1 # FAMILY_COMMA's post-comma segment (reserve_last=False) + STRICT = 2 # main segments (reserve_last=True: keep a family piece) # rules.md#H3: "successive title words at the start of the part @@ -94,6 +102,86 @@ def _is_suffix_piece(piece: Sequence[int], ptags: Set[str], return "vocab:suffix" in tags and "initial" not in tags +class Peel(NamedTuple): + """What assign's trailing peel made of a walk. `names` is a count + of positions in the caller's `rest`: rest[:names] are the name + pieces and rest[names:] the suffixes. The other two are pieces -- + token-index tuples, as PendingAmbiguity wants them -- and each is + one token long: `numeral` is the piece the roman-numeral fork + took (None when it did not fire; always the walk's last piece), + `picks` the bare ambiguous acronyms the peel had to resolve, in + peel order, either way (the last may sit at rest[names - 1]).""" + + names: int + numeral: tuple[int, ...] | None + picks: tuple[tuple[int, ...], ...] + + +# rules.md#S2: "a trailing word of the suffix vocabulary reads as a +# suffix — generational forms and credential acronyms alike, and an +# ambiguous acronym written with periods counts unambiguously. A BARE +# ambiguous acronym is consumed only when the name has words to spare" +# (v1's are_suffixes tail rule, with the roman-numeral special) +def _peel_walk(start: int, ptags: Sequence[Set[str]]) -> list[int]: + """The indices _peel_trailing walks: `start` to the segment's end, + minus the group-flagged credential pieces (the Ph. D. merge), + which assign reads as suffixes at any position. Built here and + nowhere else, so the walk's input cannot drift between assign and + the reserve: the numeral fork is a last-piece test that reads the + piece before as rest[k - 2], which holds only over this list.""" + return [j for j in range(start, len(ptags)) if "suffix" not in ptags[j]] + + +def _peel_trailing(rest: Sequence[int], pieces: Sequence[Sequence[int]], + ptags: Sequence[Set[str]], + tokens: Sequence[WorkToken]) -> Peel: + """The S2 trailing peel over `rest`, a _peel_walk list. Housed + here rather than in assign because assign imports group's piece + predicates, and group's bound-given reserve asks the same question + of the view the join would leave (#425): one walk, so the reserve + and the assignment cannot drift. Pure -- the ambiguities are + returned for assign to report, in the order it always reported + them.""" + picks: list[tuple[int, ...]] = [] + numeral: tuple[int, ...] | None = None + k = len(rest) + while k > 0: + piece = pieces[rest[k - 1]] + if _is_suffix_piece(piece, ptags[rest[k - 1]], tokens): + k -= 1 + continue + # a final single letter that is a roman numeral, after a piece + # that is not initial-shaped; the predicate's docstring carries + # the is_initial_shaped reasoning (#320) + if (k == len(rest) and k >= 2 and len(piece) == 1 + and is_trailing_numeral_suffix( + tokens[piece[0]].text, + tokens[pieces[rest[k - 2]][0]].text)): + numeral = tuple(piece) + k -= 1 + continue + # A bare ambiguous acronym ("MA", not "M.A.") is a credential + # only when peeling it still leaves a given AND a family name. + # With two pieces, "one of them is a credential" is the less + # likely reading, so it stays the family name -- "Jack MA" is a + # person, "John Smith MA" is a person with a degree. This is + # v1's reserve_last narrowed to the ambiguous set: 2.0 + # deliberately peels UNambiguous suffixes even when nothing is + # left ("Smith PhD" -> suffix, a classified fix), because there + # the vocabulary is not in doubt. + bare_ambiguous = (len(piece) == 1 + and "vocab:suffix-ambiguous" in tokens[piece[0]].tags) + # k < 2 means it is the only piece left, which is not the fork + # this reports. + if bare_ambiguous and k >= 2: + picks.append(tuple(piece)) + if k >= 3: # peeling still leaves given + family + k -= 1 + continue + break + return Peel(k, numeral, tuple(picks)) + + # rules.md#M2: "a recognized maiden marker standing after at least # one name word takes the words after it — up to any trailing # suffix — as the maiden name, and the marker itself is dropped" @@ -109,9 +197,10 @@ def _is_suffix_piece(piece: Sequence[int], ptags: Set[str], # # With the pass ahead of the joins no default-vocabulary input reaches # the lone-piece half through the one caller left that sees joined -# pieces (P5's absorbs_non_name): a marker-headed wider piece needs a -# connective right after a declined marker, and a connective after a -# marker is a word the consumer takes. Measured at #420's review -- +# pieces (P5's marker decline, marker(fk + 1)): a marker-headed wider +# piece needs a connective right after a declined marker, and a +# connective after a marker is a word the consumer takes. Measured at +# #420's review -- # dropping `len(piece) == 1` leaves the suite and a 337k-name sweep # identical -- so it stays as the rule's definition, not as a guard a # pin holds. @@ -214,6 +303,18 @@ def conj(k: int) -> bool: def marker(k: int) -> bool: return _is_maiden_marker_piece(pieces[k], tokens) + def joined_tags(lo: int, hi: int, add: Set[str] = frozenset(), + drop: Set[str] = frozenset()) -> set[str]: + # the ONE definition of a merged piece's tags: merge() applies + # it, and P5's reserve reads it to model the join it is + # weighing (#425) -- so the view cannot drift from the merge. + # A merged piece inherits every part's tags, so a site whose + # product is not what its parts were drops what no longer + # applies: the particle chain drops `prefix`, the bound join + # `title` (a derived title tag on the pair would have assign + # peel the given name as a leading title). + return (set().union(*ptags[lo:hi]) | add) - drop + def merge(lo: int, hi: int, add: Set[str] = frozenset(), drop: Set[str] = frozenset()) -> None: # pieces/ptags are parallel arrays; every merge must update @@ -243,7 +344,7 @@ def merge(lo: int, hi: int, add: Set[str] = frozenset(), for piece in pieces[lo + 1:hi]: combined.extend(piece) pieces[lo:hi] = [combined] - ptags[lo:hi] = [(set().union(*ptags[lo:hi]) | add) - drop] + ptags[lo:hi] = [joined_tags(lo, hi, add, drop)] # ph-d merge first: "Ph." "D." adjacent -> one suffix piece # (decisions.md#phd-merge; v1 fix_phd did this by regex on the @@ -455,65 +556,12 @@ def merge(lo: int, hi: int, add: Set[str] = frozenset(), # -- bound given names: the first non-title piece joins the next # ONCE (pairwise, v1 parity: 'Salem, Abdul Rahman Ahmed' keeps # Ahmed a middle name). BoundJoin encodes v1's reserve_last. - first_name_k = next( - (k for k in range(len(pieces)) if not title(k)), None) + fk = next((k for k in range(len(pieces)) if not title(k)), None) if (bound_join is not BoundJoin.DISABLED - and first_name_k is not None - and first_name_k + 1 < len(pieces) - and len(pieces[first_name_k]) == 1 - and "vocab:bound-given" - in tokens[pieces[first_name_k][0]].tags): - # rules.md#P5: "a trailing roman numeral that assign reads as the - # suffix (S2) is no word to spare, and is not joined" (history: - # decisions.md#P5) -- numeral_k is that piece, or None. Mirrors - # assign's fork condition for condition, over the walk assign - # makes (#401): - # - flagged credential pieces are out of the walk, so "last" - # and "the piece before" are read over the pieces assign - # keeps ('abdul Smith V Ph. D.'); - # - the numeral is last and not the first name piece (rest[0] - # is first_name_k, so len(rest) >= 2) -- NOT "the piece - # before it is not a title": jr is title vocabulary too - # ('abdul Smith Jr V'); - # - the piece before it is read as the join would leave it, - # bound word first -- the remap below ('abdul J. V'); - # - STRICT only: the post-comma walk has no numeral fork, so - # under LENIENT the reserve counts suffix pieces alone - # ('Berg, abdul V'). - # Computed once: a per-piece rebuild was quadratic, and the - # benchmark's bound_given shape guards it. - numeral_k = None - if bound_join is BoundJoin.STRICT: - rest = [j for j in range(first_name_k, len(pieces)) - if "suffix" not in ptags[j]] - if len(rest) >= 2 and len(pieces[rest[-1]]) == 1: - prev = rest[-2] - if prev == first_name_k + 1: - prev = first_name_k - if is_trailing_numeral_suffix( - tokens[pieces[rest[-1]][0]].text, - tokens[pieces[prev][0]].text): - numeral_k = rest[-1] - - def reads_as_suffix(k: int) -> bool: - return suffix(k) or k == numeral_k - - # first_name_k counts as a name piece even when it is - # ALSO suffix vocabulary. The reserve asks whether enough - # OTHER words are left to spare, and this piece is the one - # the rule has already claimed as a name -- excluding it - # made a dual-membership word silently un-joinable -- - # found while adding 'abd' ("All But Dissertation" as well - # as عبد), which this had to be fixed for, though it is - # not why the word was excluded. Same shape as the count - # #397 describes. - # - # The words the maiden name took are already gone (#411 - # excluded them by hand while the pass still ran later). - non_suffix = sum(1 for k in range(len(pieces)) - if not title(k) - and (k == first_name_k - or not reads_as_suffix(k))) + and fk is not None + and fk + 1 < len(pieces) + and len(pieces[fk]) == 1 + and "vocab:bound-given" in tokens[pieces[fk][0]].tags): # P5 joins the bound word to "the word after it", and a # marker is not a name word -- it is the announcement that # another name follows. The only marker left by now is one @@ -522,46 +570,68 @@ def reads_as_suffix(k: int) -> bool: # abdul nee PhD' read given 'abdul nee', and 'Berg, abdul # nee' clears the LENIENT reserve the same way. Measured # rather than assumed -- dropping this undid #411 on - # exactly that row. A suffix piece is not a name word - # either (#421) -- rules.md#P5: "nor a word of the suffix + # exactly that row. Nor is a suffix piece (#421) -- + # rules.md#P5: "nor a word of the unambiguous suffix # vocabulary (S2), wherever position will then place it" # -- and declining it is also what keeps merge()'s tag - # union from making the joined piece a suffix piece. Same - # predicate as the reserve, so the two agree about what a - # name word is. - absorbs_non_name = (marker(first_name_k + 1) - or reads_as_suffix(first_name_k + 1)) - # A given-name title ahead of the bound word asserts that a - # given name follows -- the assertion H1 reads when it - # keeps "Sir John" a given name -- so behind one there is - # no family to spare and the post-comma reserve applies - # (#369). Keyed on the WHOLE title run exactly as - # post_rules keys H1, so the two rules cannot disagree - # about what one run asserts: a join licensed here that H1 - # then read as title-plus-family would hand the joined - # piece to the family. (post_rules' run also takes H2's - # unlisted abbreviations, which no given-name title key - # can contain, so the runs match whenever the key does.) - # The STRICT test is documentary: DISABLED never reaches - # this block, and LENIENT is already the floor, so only - # STRICT can move. And the licence lifts the reserve for - # two name WORDS: the piece the join would take must be - # one word -- a particle chain is the family name P2 built - # ("Sir abdul van der Berg" keeps family 'van der Berg' as - # the untitled name does). Found in review; the rules.md - # example carries it into the rules corpus, so the gate - # witnesses it. (That the piece is not a suffix is the - # join's own decline above, general since #421.) - reserve = bound_join - if (bound_join is BoundJoin.STRICT and first_name_k > 0 - and len(pieces[first_name_k + 1]) == 1 - and _title_key(tokens[i].text - for k in range(first_name_k) - for i in pieces[k]) - in given_name_titles): - reserve = BoundJoin.LENIENT - if non_suffix >= reserve and not absorbs_non_name: - merge(first_name_k, first_name_k + 2) + # union from making the joined piece a suffix piece. + if marker(fk + 1) or suffix(fk + 1): + pass + elif bound_join is BoundJoin.LENIENT: + # post-comma the family is fixed and the pair is the + # given whatever follows, so no peel is read + # (decisions.md#P5, #423) + merge(fk, fk + 2, drop={"title"}) + else: + # rules.md#P5: "the join is tried on the pieces as it + # would leave them, assign's trailing peel (S2) is read + # over that, and the name words it leaves are the words + # to spare" (history: decisions.md#P5). The view is what + # merge() builds -- the same slice assignment, the same + # joined_tags -- and the peel is assign's own, so the + # reserve and the assignment cannot drift. And the join + # changes no suffix reading -- rules.md#P5: "a word the + # peel reads as a suffix unjoined must read so joined, + # or the join declines" -- compared as the peeled + # pieces themselves: 'abdul V' peels the V unjoined and + # nothing joined, 'abdul Smith Ma' peels the acronym + # unjoined and keeps it joined. Shapes pinned in + # test_group.py. + rest = _peel_walk(fk, ptags) + before = _peel_trailing(rest, pieces, ptags, tokens) + view, view_tags = list(pieces), list(ptags) + view[fk:fk + 2] = [pieces[fk] + pieces[fk + 1]] + view_tags[fk:fk + 2] = [joined_tags(fk, fk + 2, + drop={"title"})] + view_rest = _peel_walk(fk, view_tags) + after = _peel_trailing(view_rest, view, view_tags, tokens) + same_suffixes = ( + [tuple(view[j]) for j in view_rest[after.names:]] + == [tuple(pieces[j]) for j in rest[before.names:]]) + # A given-name title ahead of the bound word asserts + # that a given name follows -- the assertion H1 reads + # when it keeps "Sir John" a given name -- so behind + # one there is no family to spare (#369). Keyed on the + # WHOLE title run exactly as post_rules keys H1, so the + # two rules cannot disagree about what one run asserts + # (post_rules' run also takes H2's unlisted + # abbreviations, which no given-name title key can + # contain, so the runs match whenever the key does). + # The licence lifts the reserve for two name WORDS: the + # piece the join would take must be one word -- a + # particle chain is the family name P2 built ('Sir + # abdul van der Berg' keeps family 'van der Berg'). + licensed = (fk > 0 and len(pieces[fk + 1]) == 1 + and _title_key(tokens[i].text + for k in range(fk) + for i in pieces[k]) + in given_name_titles) + reserve = BoundJoin.LENIENT if licensed else BoundJoin.STRICT + if same_suffixes and after.names >= reserve: + # the pair is a given name whatever tag the word + # carried (rules.md#P5); joined_tags says why the + # title tag is dropped. Pinned in test_group.py. + merge(fk, fk + 2, drop={"title"}) return pieces, ptags, taken diff --git a/tests/v2/pipeline/test_group.py b/tests/v2/pipeline/test_group.py index aae41eb5..e1d7e0df 100644 --- a/tests/v2/pipeline/test_group.py +++ b/tests/v2/pipeline/test_group.py @@ -706,3 +706,85 @@ def test_the_join_declines_a_suffix_after_a_family_comma_too() -> None: # under every reserve. out = _grouped("Berg, abdul jr Smith") assert _piece_texts(out) == [["Berg"], ["abdul", "jr", "Smith"]] + + +# -- #425: the reserve runs assign's peel over the post-join view + +_AMBIGUOUS_LEX = _LEX.add(suffix_acronyms={"ma"}, + suffix_acronyms_ambiguous={"ma"}) + + +def test_the_reserve_mirrors_the_bare_acronym_fork() -> None: + # S2's other positional fork: a bare ambiguous acronym is peeled + # only with a given AND a family left. Read over the joined view + # 'abdul Smith Jr Ma' is three pieces, so assign peels the acronym, + # then the suffix, and one piece remains -- no family. The reserve + # now runs that same peel over the view and declines (#425); it + # used to count 'Ma' as a name word and join. + out = _grouped("abdul Smith jr Ma", lexicon=_AMBIGUOUS_LEX) + assert _piece_texts(out) == [["abdul", "Smith", "jr", "Ma"]] + + +def test_the_join_never_turns_a_suffix_into_a_name() -> None: + # Unjoined, 'abdul Smith Ma' has words to spare and the peel reads + # the acronym as a credential; joined, the view is two pieces and + # the fork would keep it as the family. The join joins two name + # words and changes nothing else, so it declines -- 1.4.0's + # reading, and 'John Smith Ma's. With a family behind it the + # acronym peels either way, and the join stands. + out = _grouped("abdul Smith Ma", lexicon=_AMBIGUOUS_LEX) + assert _piece_texts(out) == [["abdul", "Smith", "Ma"]] + out = _grouped("abdul Smith Berg Ma", lexicon=_AMBIGUOUS_LEX) + assert _piece_texts(out) == [["abdul Smith", "Berg", "Ma"]] + + +def test_the_join_never_turns_a_name_into_a_suffix_either() -> None: + # The same rule from the other side: 'abdul V' is two pieces + # whose V the peel reads as the suffix; joined it would be one + # piece the fork cannot fire on, so the V would become a name + # word. The peel takes the V unjoined and nothing joined -- the + # join declines. (Of the numeral pins above, 'abdul J. V' + # declines by this comparison; 'abdul Smith V' by the threshold, + # one name word being no family to spare.) + out = _grouped("abdul V") + assert _piece_texts(out) == [["abdul", "V"]] + + +def test_the_joined_pair_is_a_given_name_whatever_tag_the_word_carried() -> None: + # A title word standing in the name is a name word (H3) and the + # join takes it as v1 did -- but the conjunction merge derives a + # `title` piece tag for "mr and mrs", and merge()'s tag union + # would hand that tag to the joined pair, which assign then peels + # as a leading title: 'abdul Sheikh and Ahmad Bakar Smith' read + # title 'abdul Sheikh and Ahmad' on 2.0 and 2.1, and the shorter + # 'abdul Sheikh and Ahmad Bakar' would have too once the count's + # title exclusion went. The bound join drops the tag: the pair is + # a given name. Found by the code review. + out = _grouped("abdul mr and mrs Smith Berg") + assert _piece_texts(out) == [["abdul mr and mrs", "Smith", "Berg"]] + assert "title" not in out.piece_tags[0][0] + + +def test_a_title_word_in_the_name_is_a_name_word_to_the_join() -> None: + # The 2.x reserve had excluded title pieces from its count, an + # unrecorded deviation from v1, which joined them; assign reads a + # mid-name title word as a name word, and the shared peel follows + # assign. 1.4.0 parity on every shape here. + out = _grouped("abdul mr Smith Berg") + assert _piece_texts(out) == [["abdul mr", "Smith", "Berg"]] + out = _grouped("abdul Smith mr") + assert _piece_texts(out) == [["abdul Smith", "mr"]] + out = _grouped("Berg, abdul mr") + assert _piece_texts(out) == [["Berg"], ["abdul mr"]] + + +def test_the_licence_does_not_lift_the_equality() -> None: + # The one shape where the join would move the numeral fork AND + # the licence's threshold of one would let it through: 'sir abdul + # J. V' -- unjoined the V is a name word (the fork is suppressed + # by the initial-shaped 'J.'), joined it is the suffix: the peel + # takes nothing unjoined and the V joined. Declines, as 'Sir John + # J. V' reads. A looser comparison passed every other test; found + # by the test review. + out = _grouped("sir abdul J. V", lexicon=_GIVEN_NAME_TITLE_LEX) + assert _piece_texts(out) == [["sir", "abdul", "J.", "V"]] diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 084668a4..506f4b71 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -1252,7 +1252,7 @@ def _claim(rule: dict) -> _Claim: "fix(comma-precomma-family) pre-comma run reads as family, not given": _Claim(262, ('family', 'given'), "d5e6ee237ff9"), "fix(suffix-routing) two-token name with unambiguous trailing suffix stays suffix": - _Claim(1036, ('family', 'given', 'suffix'), "43399188180d"), + _Claim(1041, ('family', 'given', 'suffix'), "705c375e2778"), "fix(suffix-delimiter-rendering) no-space delimiter core token kept whole": _Claim(0, ('suffix',), "e3b0c44298fc"), "ambiguous-surname-acronym data change: parenthesized (MA)/(DO) now stays nickname": @@ -1303,6 +1303,8 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('given', 'middle'), "9523e518e6ec"), "fix(#421) the bound-given join never absorbs a split credential": _Claim(1, ('given', 'middle'), "228abe0f32ef"), + "fix(#425) accepted: a bare ambiguous acronym the peel does not take joins as a name word": + _Claim(1, ('given', 'middle'), "2010cc79a34d"), "fix(#360) ste moved into the never-given particles with mc": _Claim(1, ('family', 'given'), "e62caedec864"), }, @@ -1353,6 +1355,8 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('given', 'middle'), "9523e518e6ec"), "fix(#421) the bound-given join never absorbs a split credential": _Claim(1, ('given', 'middle', 'suffix'), "228abe0f32ef"), + "fix(#425) the bound-given reserve runs assign's peel over the joined view": + _Claim(2, ('family', 'given', 'suffix'), "ef1ab03b617e"), "fix(#369) the bound given-name join takes a particle-and-bound word, so no fork is reported": _Claim(1, ('_ambiguities',), "81cf02ffdb33"), "fix(#360) ste moved into the never-given particles with mc": @@ -1393,6 +1397,8 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('given', 'middle'), "9523e518e6ec"), "fix(#421) the bound-given join never absorbs a split credential": _Claim(1, ('given', 'middle', 'suffix'), "228abe0f32ef"), + "fix(#425) the bound-given reserve runs assign's peel over the joined view": + _Claim(2, ('family', 'given', 'suffix'), "ef1ab03b617e"), "fix(#369) the bound given-name join takes a particle-and-bound word, so no fork is reported": _Claim(1, ('_ambiguities',), "81cf02ffdb33"), "fix(#360) ste moved into the never-given particles with mc": diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index b25e6fd6..9e18ca01 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -493,6 +493,12 @@ def test_a_given_name_title_licenses_the_bound_given_join() -> None: plain = parse("Dr. abdul salam") assert (plain.title, plain.given, plain.family) == \ ("Dr.", "abdul", "salam") + # and the pair's report: a bound word read as the bound word is + # not a fork, so the pick 'Sheik John Ma' reports is not reported + # for 'Sheik abdul Ma' (decisions.md#P5, the #369 precedent) + assert parse("Sheik abdul Ma").ambiguities == () + assert [a.kind for a in parse("Sheik John Ma").ambiguities] == \ + [AmbiguityKind.SUFFIX_OR_NAME] def test_the_bound_given_reserve_spares_the_family_assign_will_keep() -> None: @@ -539,6 +545,64 @@ def test_the_reserve_declines_and_assign_reads_the_unjoined_pieces() -> None: assert (n.given, n.family, n.suffix) == ("abdul", "Smith", "V, Ph. D.") +def test_the_reserve_spares_the_family_the_acronym_fork_would_take() -> None: + # #425: with a suffix word between the pair and a bare ambiguous + # acronym, assign peels the acronym (three pieces, words to spare) + # and then the suffix, and the family the join left was never + # there. The reserve now runs assign's peel over the joined view + # and declines, so these read as their ordinary-given twins. + for bound, plain in (("abdul Smith Jr Ma", "John Smith Jr Ma"), + ("abdul Rahman PhD MA", "John Rahman PhD MA")): + n, m = parse(bound), parse(plain) + assert (n.family, n.suffix) == (m.family, m.suffix) + assert n.family != "" + n = parse("abu Bakar Jr Ed") + assert (n.family, n.suffix) == ("Bakar", "Jr, Ed") + # and the join never turns a suffix into a name: unjoined, the + # acronym is a credential with words to spare, so 'abdul Smith + # Ma' reads as 'John Smith Ma' does (1.4.0 parity restored) + n, m = parse("abdul Smith Ma"), parse("John Smith Ma") + assert (n.family, n.suffix) == (m.family, m.suffix) == ("Smith", "Ma") + + +def test_a_joined_pair_is_never_peeled_as_a_title() -> None: + # The conjunction merge derives a title tag for 'Sheikh and Ahmad'; + # the bound join takes the piece (a mid-name title word is a name + # word, as 1.4.0 read it) and the pair must stay the given name, + # not inherit the tag and be peeled as a leading title. + n = parse("abdul Sheikh and Ahmad Bakar") + assert (n.title, n.given, n.family) == \ + ("", "abdul Sheikh and Ahmad", "Bakar") + # the title-word class: name words to P5, as v1 read them -- + # parity restored for the post-comma shape, never lost for the + # main-walk one + n = parse("abdul Sir Smith Berg") + assert (n.given, n.middle, n.family) == ("abdul Sir", "Smith", "Berg") + n = parse("Berg, abdul Sir") + assert (n.given, n.family) == ("abdul Sir", "Berg") + + +def test_the_licence_does_not_lift_the_equality() -> None: + # Behind a given-name title the reserve needs one name piece, so + # "changes no suffix reading" is the only thing between 'Sir abdul + # J. V' and a join that turns the V from a name word into the + # suffix (#369 had joined it). It reads exactly as 'Sir John J. V' + # does. + for text in ("Sir abdul J. V", "Sir John J. V"): + n = parse(text) + assert (n.middle, n.family, n.suffix) == ("J.", "V", "") + + +def test_the_numeral_fork_fires_on_the_last_piece_only() -> None: + # The shared peel's own contract, pinned at the stage that owns + # it: a numeral with a suffix behind it is a name word, for an + # ordinary given name and the bound pair alike. + for text, family in (("John Smith V Jr", "V"), + ("abdul Smith V Jr", "V")): + n = parse(text) + assert (n.family, n.suffix) == (family, "Jr") + + @pytest.mark.parametrize("bound", ["abd", "abu"]) @pytest.mark.parametrize("numeral", ["I", "X"]) def test_every_bound_word_spares_the_family_before_every_numeral( diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 75868a58..14a06915 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -126,8 +126,13 @@ "abd Berg née Jones" "abd née Jones" "abdul Jr Smith Berg" +"abdul Ma Smith" "abdul Ph. D. Smith Berg" +"abdul Sir Smith Berg" +"abdul Smith Berg Ma" +"abdul Smith Jr Ma" "abdul Smith Jr V" +"abdul Smith Ma" "abdul Smith V" "abdul V Smith" "abdul salam ahmed salem" diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 3867ba9c..b7834665 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -802,9 +802,10 @@ fields = ["given", "family", "suffix"] issue = "fix(#421) the bound-given join never absorbs a suffix piece" # 'abdul Jr Smith Berg': rules.md#P5 -- "the join never absorbs a # marker standing as a word of its own -- a marker is not a name word -# (M2) -- nor a word of the suffix vocabulary (S2), wherever position -# will then place it". given 'abdul Jr', middle 'Smith' -> given -# 'abdul', middle 'Jr Smith'. Parity with this baseline until now; a +# (M2) -- nor a word of the unambiguous suffix vocabulary (S2), +# wherever position will then place it". given 'abdul Jr', middle +# 'Smith' -> given 'abdul', middle 'Jr Smith'. Parity with this +# baseline until now; a # rules.md example, since no differential corpus name has the shape. name_regex = "(?i)^abdul\\s+jr\\s+smith\\s+berg$" fields = ["given", "middle"] @@ -821,6 +822,19 @@ issue = "fix(#421) the bound-given join never absorbs a split credential" name_regex = "(?i)^abdul\\s+ph\\.\\s+d\\.\\s+smith\\s+berg$" fields = ["given", "middle"] +[[change]] +issue = "fix(#425) accepted: a bare ambiguous acronym the peel does not take joins as a name word" +# 'abdul Ma Smith': rules.md#P5 -- "a bare ambiguous acronym is a name +# word wherever the peel does not take it". first 'abdul', middle +# 'Ma' -> given 'abdul Ma': v1's join counted the acronym out as +# suffix vocabulary and declined; since 2.0 the join takes it, as +# assign reads it (a middle name, not a credential, with a family +# behind it). A 2.0 reading, byte-identical at 2.0.0 and 2.1.0, first +# witnessed here -- P5 records it as Accepted, and the example exists +# to carry the shape into this corpus. +name_regex = "(?i)^abdul\\s+ma\\s+smith$" +fields = ["given", "middle"] + [[change]] issue = "fix(#360) ste moved into the never-given particles with mc" # 'Ste Marie': the second word of the vocabulary move, which the diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index 217e7cc1..56e1936f 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -391,9 +391,10 @@ fields = ["given", "family"] issue = "fix(#421) the bound-given join never absorbs a suffix piece" # 'abdul Jr Smith Berg': rules.md#P5 -- "the join never absorbs a # marker standing as a word of its own -- a marker is not a name word -# (M2) -- nor a word of the suffix vocabulary (S2), wherever position -# will then place it". given 'abdul Jr', middle 'Smith' -> given -# 'abdul', middle 'Jr Smith'. Parity with this baseline until now; a +# (M2) -- nor a word of the unambiguous suffix vocabulary (S2), +# wherever position will then place it". given 'abdul Jr', middle +# 'Smith' -> given 'abdul', middle 'Jr Smith'. Parity with this +# baseline until now; a # rules.md example, since no differential corpus name has the shape. name_regex = "(?i)^abdul\\s+jr\\s+smith\\s+berg$" fields = ["given", "middle"] @@ -408,6 +409,22 @@ issue = "fix(#421) the bound-given join never absorbs a split credential" name_regex = "(?i)^abdul\\s+ph\\.\\s+d\\.\\s+smith\\s+berg$" fields = ["given", "middle", "suffix"] +[[change]] +issue = "fix(#425) the bound-given reserve runs assign's peel over the joined view" +# 'abdul Smith Jr Ma', 'abdul Smith Ma': rules.md#P5 -- "the join is +# tried on the pieces as it would leave them, assign's trailing peel +# (S2) is read over that, and the name words it leaves are the words +# to spare" and "a word the peel reads as a suffix unjoined must read +# so joined, or the join declines". given 'abdul Smith', family '' -> +# given 'abdul', family 'Smith' (the acronym and the suffix peel as +# they do for 'John Smith Jr Ma'); given 'abdul Smith', family 'Ma' +# -> given 'abdul', family 'Smith', suffix 'Ma' (the join would have +# turned a credential into the family). Both 1.4.0 parity, so the +# 1.4.0 ledger has no twin. Rules.md examples; no differential corpus +# name has either shape. +name_regex = "(?i)^abdul\\s+smith\\s+(jr\\s+)?ma$" +fields = ["given", "family", "suffix"] + [[change]] issue = "fix(#369) the bound given-name join takes a particle-and-bound word, so no fork is reported" # 'Sheik Abu Bakar': the fields are byte-identical to this baseline diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index 03b01b56..996274d8 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -89,9 +89,10 @@ fields = ["given", "family"] issue = "fix(#421) the bound-given join never absorbs a suffix piece" # 'abdul Jr Smith Berg': rules.md#P5 -- "the join never absorbs a # marker standing as a word of its own -- a marker is not a name word -# (M2) -- nor a word of the suffix vocabulary (S2), wherever position -# will then place it". given 'abdul Jr', middle 'Smith' -> given -# 'abdul', middle 'Jr Smith'. Parity with this baseline until now; a +# (M2) -- nor a word of the unambiguous suffix vocabulary (S2), +# wherever position will then place it". given 'abdul Jr', middle +# 'Smith' -> given 'abdul', middle 'Jr Smith'. Parity with this +# baseline until now; a # rules.md example, since no differential corpus name has the shape. name_regex = "(?i)^abdul\\s+jr\\s+smith\\s+berg$" fields = ["given", "middle"] @@ -106,6 +107,22 @@ issue = "fix(#421) the bound-given join never absorbs a split credential" name_regex = "(?i)^abdul\\s+ph\\.\\s+d\\.\\s+smith\\s+berg$" fields = ["given", "middle", "suffix"] +[[change]] +issue = "fix(#425) the bound-given reserve runs assign's peel over the joined view" +# 'abdul Smith Jr Ma', 'abdul Smith Ma': rules.md#P5 -- "the join is +# tried on the pieces as it would leave them, assign's trailing peel +# (S2) is read over that, and the name words it leaves are the words +# to spare" and "a word the peel reads as a suffix unjoined must read +# so joined, or the join declines". given 'abdul Smith', family '' -> +# given 'abdul', family 'Smith' (the acronym and the suffix peel as +# they do for 'John Smith Jr Ma'); given 'abdul Smith', family 'Ma' +# -> given 'abdul', family 'Smith', suffix 'Ma' (the join would have +# turned a credential into the family). Both 1.4.0 parity, so the +# 1.4.0 ledger has no twin. Rules.md examples; no differential corpus +# name has either shape. +name_regex = "(?i)^abdul\\s+smith\\s+(jr\\s+)?ma$" +fields = ["given", "family", "suffix"] + [[change]] issue = "fix(#369) the bound given-name join takes a particle-and-bound word, so no fork is reported" # 'Sheik Abu Bakar': the fields are byte-identical to this baseline