diff --git a/AGENTS.md b/AGENTS.md index 26474e58..4d798328 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -340,7 +340,7 @@ Don't use the bare `python3 -m doctest .rst` CLI (no `optionflags`) to che **Prefix-join uses value-based `list.index()`** in `join_on_conjunctions` — fragile when a token value repeats (e.g. a trailing title that's also a suffix acronym, or two `van`s); constrain such lookups to start at `i + 1`. See #100. -**Title vs suffix is positional for BARE words, and the leading period-abbreviation rule overrides even that** — a word matching `TITLES` at the front of a name becomes `title`; the same word matching `SUFFIX_ACRONYMS`/`SUFFIX_WORDS` at the end becomes `suffix` (never both, regardless of the word's real-world meaning). External test sources (old issue gists, etc.) sometimes assert `suffix` for a leading professional abbreviation like `RA`/`PD`/`Dipl.-Ing.` — that's the source data being wrong, not a parser bug. Verify position before "fixing" it. Two qualifications the older "purely positional" wording papered over, both measured 2026-08-01: a PERIOD-marked leading word is claimed by the shape rule before any vocabulary is read (`"Esq. Smith"` → `title`, though `esq` is suffix-only), and trailing position has no such rule at all, so a title word there is neither title nor suffix but a NAME part (`"John Smith Prof."` → `family='Prof.'`) — which is what the comma path already disagrees with. Why it is not simply inverted to "vocabulary decides": `TITLES` holds 692 words that are in no suffix set, and many are ordinary surnames (`king`, `bishop`, `prince`, `pope`, `judge`, `sheriff`, `baron`, `master`, ...), so a vocabulary-first trailing rule would read `"Mary Jane King"` as `title='King'`, `family='Jane'`. The period is what separates the safe case from that one — `King` is a surname, `King.` is not. +**Title vs suffix is positional for BARE words, and the leading period-abbreviation rule overrides even that** — a word matching `TITLES` at the front of a name becomes `title`; the same word matching `SUFFIX_ACRONYMS`/`SUFFIX_WORDS` at the end becomes `suffix` (never both, regardless of the word's real-world meaning). The `TITLES`/suffix overlap was audited in #296 (2026-08-23): the pure postnominals (`jr`, `junior`, `phd`, `do`, `se`) left `TITLES`, the v1-residue `dr`/`sra` left the suffix sets, and the twelve words still in both (`md`, `ms`, `sa`, `sr`, `lt`, `ra`, `vc`, and the ranks `cpl`, `cpo`, `cpt`, `csm`, `sgm`) are deliberate duals that position decides. External test sources (old issue gists, etc.) sometimes assert `suffix` for a leading professional abbreviation like `RA`/`PD`/`Dipl.-Ing.` — that's the source data being wrong, not a parser bug. Verify position before "fixing" it. Two qualifications the older "purely positional" wording papered over, both measured 2026-08-01: a PERIOD-marked leading word is claimed by the shape rule before any vocabulary is read (`"Esq. Smith"` → `title`, though `esq` is suffix-only), and trailing position has no such rule at all, so a title word there is neither title nor suffix but a NAME part (`"John Smith Prof."` → `family='Prof.'`) — which is what the comma path already disagrees with. Why it is not simply inverted to "vocabulary decides": `TITLES` holds 694 words that are in no suffix set, and many are ordinary surnames (`king`, `bishop`, `prince`, `pope`, `judge`, `sheriff`, `baron`, `master`, ...), so a vocabulary-first trailing rule would read `"Mary Jane King"` as `title='King'`, `family='Jane'`. The period is what separates the safe case from that one — `King` is a surname, `King.` is not. ### Tests (`tests/`) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 3cbaeb73..99568344 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -82,6 +82,7 @@ Open: [#360](https://github.com/derek73/python-nameparser/issues/360) which part ### P6 — the trailing orphan particle +- 2026-08-23 — `Jong, vd` is C1's now, not P6's: C1's listing-form clause (#296's bundle) reads a post-comma part that is nothing but suffix words as the credential run, and `vd` is suffix vocabulary, so the row reads family 'Jong', suffix 'vd' by statement (it read so before, by the lone-piece deviation). The #380 scope note below calling it deliberately unresolved is superseded for that one row; the P6 question — a tussenvoegsel with no given word left — is unchanged. - 2026-08-18 #379/#380 IMPLEMENTED — the rule shipped as stated, both `deviates:` markers out. Keyed on the token's VOCABULARY rather than its assigned role, which is what gives the S2 precedence its effect: assign reads a trailing `vd`/`mc`/`do` as a post-nominal, and the attachment overrides that after a comma. The words-to-spare guard is a PIECE test, not a count — every trailing piece that is wholly particles attaches, and the run must leave a GIVEN word ahead of it, so "Nguyen, Van" keeps its only given word. Not "a name word": see the defect below, where that wider reading passed on family text P1 had just produced. MIDDLE was in the test briefly and came out — 0 hits over 740,552 instrumented guard sites, and structurally unreachable, since the only rule that leaves a MIDDLE with no GIVEN ahead of it is gated on state.order, which assign never records on the family-comma path P6 runs on. Measured: ONE of 751 corpus names moves ("Vega, Juan de la"), at all three baselines, and it needed a new ledger rule in each. The LEDGER rule's regex — not the parse rule, which has no regex and fires on the whole particle vocabulary through the tag — is a PARTIAL copy of PARTICLES, deliberately: the words that actually end a Dutch, Iberian or German listing, not all 70. The ledger guards forced that narrowing: the first draft used the neighbours' bare comma regex, which claimed 215 names and reached two protected exclusions. A member `des` had to come out as well, and for a reason worth keeping straight: it is not in PARTICLES at all, so it could copy nothing, and the text it reached was the opening of "Designated Judge David A. Ezra" rather than a person's name. `_reaches_non_vocabulary("des", PARTICLES)` prints the FRAGMENT it matched, `['Des']`, not the name containing it — which is how the retracted wording arose in the first place. Two defects found in review, both from the rule's neighbours rather than from its own logic. P1's fold runs EARLIER in the same function and retags all of segment 1 to FAMILY, so a words-to-spare test written as "some name role remains" passed on family text P1 had just produced — and the attachment then hoisted the particle in front of a base it never preceded ("Smith, de Mesnil van" → 'van Smith de Mesnil'). 1.4 gives 'de Mesnil van Smith' for that input and 2.1 gives 'Smith de Mesnil van', so neither the defect nor the fix restores 1.4 — what the fix restores is 2.1, the pre-comma-family reading having moved back in 2.0. The test has to name GIVEN, which is what the rule says. And a post-nominal sits BEHIND the tussenvoegsel in this listing, so the backward walk stopped on it and the attachment never fired: "Berg, Jan van Jr." and "Berg, Jan van, Jr." parsed two ways on nothing but a comma. The walk now steps over a trailing piece holding no name, unless it is itself particle vocabulary — `vd` arrives suffix-roled and IS the run. @@ -107,7 +108,7 @@ Open: [#360](https://github.com/derek73/python-nameparser/issues/360) which part Open: [#380](https://github.com/derek73/python-nameparser/issues/380) covers "Berg, Jan vd" under this rule, but the vd reading itself is decisions.md#vocabulary-collisions (C-ii); and the no-given-word case "Jong, vd" is deliberately unresolved — see the scope note. -- 2026-08-22 (late review of #425's docs, PR #426) — the bound join runs before P6 sees the name, and P6 now says so. P5's new title-word clause took `do`, `freiherr` and `st` — title AND particle vocabulary — away from the attachment: `Berg, abdul do` had read family 'do Berg' since #379 and reads given 'abdul do' now. That is 1.4.0's reading, and it is what every other particle already did: `Berg, abdul van` read given 'abdul van' at 1.4.0, on 2.0 and 2.1, and on master before #426 — the old reserve's title exclusion had held only the title-word trio back. The precedence is the stage order (P5 joins in group; P6 attaches in post_rules, over what is left) rather than a judgement about tussenvoegsels behind bound words, and both rules carry it as Accepted with `Berg, abdul van` as the example, parity at every baseline. The precedence has one exception, found by the review of #424's docs when the Accepted clause claimed the attachment "never sees" the particle: a trailing particle of the unambiguous suffix vocabulary too — `vd`, `mc`; `do` is ambiguous, and a name word to the join — is a suffix piece to the join, which never takes one (#421), so the attachment does see it, and `Berg, abdul vd` reads family 'vd Berg' where 1.4.0 read given 'abdul vd'. Both rules say so, with the example; at 1.4.0 the fix(comma-precomma-family) rule already reaches it (a rule written for it was shadowed, and dropped). +- 2026-08-22 (late review of #425's docs, PR #426) — the bound join runs before P6 sees the name, and P6 now says so. P5's new title-word clause took `do`, `freiherr` and `st` — title AND particle vocabulary, until #296's audit took `do` out of TITLES, since when the join takes `do` as the particle it is — away from the attachment: `Berg, abdul do` had read family 'do Berg' since #379 and reads given 'abdul do' now. That is 1.4.0's reading, and it is what every other particle already did: `Berg, abdul van` read given 'abdul van' at 1.4.0, on 2.0 and 2.1, and on master before #426 — the old reserve's title exclusion had held only the title-word trio back. The precedence is the stage order (P5 joins in group; P6 attaches in post_rules, over what is left) rather than a judgement about tussenvoegsels behind bound words, and both rules carry it as Accepted with `Berg, abdul van` as the example, parity at every baseline. The precedence has one exception, found by the review of #424's docs when the Accepted clause claimed the attachment "never sees" the particle: a trailing particle of the unambiguous suffix vocabulary too — `vd`, `mc`; `do` is ambiguous, and a name word to the join — is a suffix piece to the join, which never takes one (#421), so the attachment does see it, and `Berg, abdul vd` reads family 'vd Berg' where 1.4.0 read given 'abdul vd'. Both rules say so, with the example; at 1.4.0 the fix(comma-precomma-family) rule already reaches it (a rule written for it was shadowed, and dropped). ### M2 — the maiden-marker rule @@ -193,10 +194,10 @@ The mechanism shipped twice before anyone wrote down its criterion. Sizes as of axis; `Lo` does lead in romanized Chinese ("Lo Wei"), so it may genuinely collide where `das` does not. Corrected on #360, which carried the wrong table publicly. -- The concrete demonstration is "do", which three vocabularies claim — titles, particles/ambiguous, suffix_acronyms/ambiguous. Two mark it ambiguous; the third, TITLES, is the one that actually decides "Do Quang Minh" (title="Do", given="Quang") and reports nothing. Same word behind #385's "Anh Do". +- The concrete demonstration is "do", which three vocabularies claimed — titles, particles/ambiguous, suffix_acronyms/ambiguous. Two mark it ambiguous; the third, TITLES, was the one that actually decided "Do Quang Minh" (title="Do", given="Quang") and reported nothing — until #296's audit (2026-08-23) took it out of TITLES, since when the name reads given "Do" and reports the fork. Same word behind #385's "Anh Do". - Applications, each still its own work: #360 (mc, ste — neither is a borne given name, so both leave the ambiguous half); #342 (rai — Rai IS a borne surname, so it earns the marking rather than moving); #385 (do — resolved at decisions.md#R2). - C-ii's per-word framing versus a rule stated for a SHAPE: measured 2026-08-16, the words that are both particle and suffix vocabulary are vd, do and mc — three, not the one this criterion adjudicated. rules.md#P6 states its precedence for the shape, so do and mc inherit vd's answer without being weighed. Recorded rather than papered over: stating a per-word judgement as a general clause is how an unexamined word acquires a decision, and the two are named here so the next reader knows which one was actually argued. -- Caution when applying C-i to the particle set: TITLES ∩ ambiguous == {do, freiherr, st} is load-bearing, per the Excluded note in the W2 section. Emptying it makes the particle-or-given emitter dead code. +- Caution when applying C-i to the particle set: TITLES ∩ ambiguous — {do, freiherr, st} until #296's audit took 'do' out of TITLES (2026-08-23; `Do Quang Minh` reads given 'Do' with the fork now), {freiherr, st} since — is load-bearing, per the Excluded note in the W2 section. Emptying it makes the particle-or-given emitter dead code. Open: [#348](https://github.com/derek73/python-nameparser/issues/348) applying C-i to the 711 title entries, then titles_ambiguous plus a TITLE_OR_GIVEN kind. Blocked on data, not on judgement — the census needs a given-name frequency corpus this repo does not have, which is why the criterion is recorded here and the census is not attempted. @@ -326,7 +327,7 @@ Excluded (the never-given / ambiguous particle line, nameparser/config/particles - Only 9 of the 39 ambiguous members were ever individually justified; the rest sit there by the conservative default (ambiguous unless argued never-given). - mc, ste — measured misparses ("Mc Donald" → given "Mc"), tracked in #360; st is inert at the head because TITLES claims it first; mac must stay ambiguous because Mac is a real given name. - Encoding rationale (#293, predating #360's membership questions): the data layer stores the NEVER-GIVEN set and derives the ambiguous one, because that is safe-by-default for new particles — a one-place addition — and the v1 shim translates by one-directional complement. And the constants are FROZEN specifically to kill the cached-Lexicon.default()-vs-fresh- Constants desync that runtime module-constant mutation caused. -- Load-bearing dependency: TITLES ∩ ambiguous == {do, freiherr, st} is what keeps the particle-or-given ambiguity emitter reachable at all; moving all three would make it dead code, which is why test_the_chained_emitter_is_still_reachable distinguishes "pick another word" from "delete the emitter". +- Load-bearing dependency: TITLES ∩ ambiguous — {do, freiherr, st} until #296's audit, {freiherr, st} since — is what keeps the particle-or-given ambiguity emitter reachable at all; moving them all would leave the shipped vocabulary unable to reach it, which is why test_the_chained_emitter_is_reachable_by_construction (tests/v2/test_parser.py) pins reachability by construction rather than by shipped vocabulary — an empty intersection fails no test and does not mean the emitter is unreachable. Open (contested vocabulary memberships — the rule is right, the word's set is questioned; the issue is canonical): [#342](https://github.com/derek73/python-nameparser/issues/342) @@ -435,6 +436,8 @@ Excluded (MAIDEN_MARKERS, per nameparser/config/maiden_markers.py): - 2026-07 (plan deviation #3, recorded) — the decision is definitionally vocabulary-dependent: there is no way to recognize a credential run without consulting the suffix word lists, so the structural stage reads vocabulary through one predicate. - 2026-07-12/13 (v2 Policy work, PR #288) — the lenient token test is the default and `lenient_comma_suffixes=False` restores the strict one. (An earlier entry here credited #291/#296 — git author dates place both the field and its wiring in the Policy commits, and the #291/#296 design doc never mentions the knob.) - 2026-08 #319 — the wholly-suffix predicate was lifted into the vocabulary layer so the comma decision and the honorific peel's segment test cannot drift apart. +- 2026-08-23 #296/#325 — the listing form's post-comma part is read for what it is. Three commits of the approved #291/#296 bundle (spec 2026-07-30, plan 2026-08-01; see the comma-suffix arc below), in the plan's order. (1) A comma followed only by titles fixed no family boundary, so the pre-comma name keeps its positional read: `John Smith, Mr.` reads given 'John', family 'Smith', title 'Mr.' where 2.0 had merged the family ('Bob Jones, author' moves with it at every baseline). "Nothing but titles" is title-shaped AND not suffix vocabulary — the plan's helper said only the first, and would have read `Smith, PhD Jr.` as the title run 'PhD Jr.' and `Smith, Mr. Jr.` as 'Mr. Jr.' (both master's suffix 'Jr.'; the gate and a measurement against master caught them). (2) The TITLES/suffix overlap audit and the ordering, one commit because each alone is wrong (the plan's F1): reading the credential run first would make `Smith, Dr.` a suffix while 'dr' was SUFFIX_WORDS residue, and the audit alone leaves `Smith, Jr.` a title by the period-abbreviation inference. The audit table's dispositions ship as approved — 'do', 'jr', 'junior', 'phd', 'se' out of TITLES; 'dr' and 'sra' out of the suffix sets; 'ms' and 'sa' into the ambiguous set — with ONE deviation on evidence the table did not have: 'md' keeps dual membership, because bare 'Md' before a name is the Bengali and South Asian Muslim abbreviation of Muhammad (#343/#345's corpus rows `Md Abdul Karim`, `Md. Abdul Karim`; the drop read the first as given 'Md', middle 'Abdul') while 'MD' after the name is the degree — position decides, as for 'sr'. The ordering is wider than the plan's lone piece: a post-comma part that is nothing but suffix pieces is the credential run, whole, which is what #325 needed (`Smith, Ph. D. Jr.` had put the split credential in the given name once a second suffix made it no longer lone — a 1.4.0 regression; v1 read suffix 'Ph. D.', title 'Jr.'). Vocabulary decides which words qualify and position breaks the tie for the genuine duals: `Smith, Sr.` is Senior where `Sr. Garcia` is Señor, `Smith, Esq.` is the postnominal where H2 still reads `Esq. Smith` as a title (H2 carries that as Accepted). `Smith, Dr. Jr.` keeps the 2.0 deviation's other case — a title, then the last piece as the suffix (v1 read first 'Jr.'; 2.0 through master suffix 'Jr.') — and `Smith, John Jr.` its given name. Consequences the audit carries, all measured and pinned: a leading 'PhD' or 'Jr' is a name word; `Do Quang Minh` reads given 'Do' with the particle-or-given fork, and `Dr. Do Van Johnson, MD` given 'Do', family 'Van Johnson' (the #367 rule for that name is replaced; TITLES ∩ ambiguous is {freiherr, st} now, which keeps the chained emitter reachable); a trailing bare 'Dr' is a name word, joining every other title-only word there (#316 is the open question); `田中さん, PhD` reads suffix 'さん, PhD'; the #100 prefix-join tests use 'lt', a genuine dual, as their token. #291 (multi-word credentials) is the bundle's fourth commit and its own PR. +- 2026-08-23 (review of PR #428, four reviewers) — the repair's condition is "no name word", not "all titles": `John Smith, Mr. Jr.` (a title and a postnominal) had kept the merge the rationale argues against, and reads given 'John', family 'Smith', suffix 'Jr.' now. Its guard counts NAME pieces: the positional read peels a trailing suffix first, so `Smith Jr., Mr.` — two pieces, one name — read positionally lost its family (the code review); it keeps family 'Smith', suffix 'Jr.' at every baseline. The positional read records its ORDER, and post_rules' leading-piece scan keys on "assign records no order after a family comma" rather than on the structure, so P1's family-first fold reaches the pre-comma name as it does without the comma — `de Mesnil Juan, Dr.` under a family-first order had read family 'de' (the test review). The credential run does NOT step over a delimiter core the policy names (`Smith, RN - CRNA` reads given 'RN' under `extra_suffix_delimiters` as without it): the code review asked for it, a draft did it, and tests/test_suffixes.py's "the delimiter must not affect parsing at all" outside the suffix-comma form is v1 parity pinned on purpose — C1 carries the limit as Accepted. A second deviation from the audit table: `ms` and `sa` do NOT join the ambiguous set — the gate is position-blind and the collision is not; gated, `John Smith, MS` lost its suffix-comma route and read title 'MS', and `Smith, Ms.` passed the gate on its one period and read as a credential anyway. Both are genuine duals, and position decides: `Smith, Ms.` reads suffix 'Ms.' (title at every baseline; C1 carries it as Accepted, with `Smith, Ms. Jane` keeping the title) and `John Smith, MS` keeps suffix 'MS'. And the gate itself now means what S2 says: an ambiguous acronym counts when written with ITS periods, one after each letter (`M.A.`, `J.D.`), not on any period — `Smith, Ed.` had become a credential on a name's trailing period, and `Jack Ma.` had read suffix 'Ma.' since 2.0 (S2 carries the boundary; 1.4.0 read family 'Ma.'). Two limits recorded rather than fixed: `田中さん, Dr.` keeps the honorific glued, joining master's `田中さん, Mr.` — the honorific peel runs in script_segment on the suffix-comma and no-comma structures, before group or assign can say the comma fixed nothing (C1 Accepted); and group's chain emitter still reports no PARTICLE_OR_GIVEN on the family-comma path, so `Freiherr von Berg Ma, Mr.` reads as its no-comma twin but without the report — group runs before assign decides which read applies, and plumbing a deferred report through the state is more than the shape is worth today. ### T1 — separators, not joiners @@ -544,7 +547,7 @@ Declined: ### comma-suffix-arc — #291/#296/#316 (2026-07-26 → 2026-08-01) #291 was filed 2026-07-26 out of the 2.0 vocabulary cleanup, with -its decline-by-measurement evidence in the issue body; "z domu" folded into it 2026-07-27 (see Excluded, MAIDEN_MARKERS). Intent for #291 and #296 is settled by the approved bundle spec but UNSHIPPED — rules.md carries both as deviates: markers on C1. The arc's bookkeeping: #291/#296 moved milestone v2.1 → v2.2 on 2026-08-01 (with #289/#293); the 2026-07-30 design doc was amended in place 2026-08-01 (A1–A6); its durable content is HERE rather than in the gitignored doc; #316 (trailing titles) was filed the same day, carrying the esq cleanup as a Related section. +its decline-by-measurement evidence in the issue body; "z domu" folded into it 2026-07-27 (see Excluded, MAIDEN_MARKERS). Intent for #291 and #296 is settled by the approved bundle spec; #296 (with #325) SHIPPED 2026-08-23 as the bundle's first three commits, with one deviation from the audit table recorded at C1 ('md' keeps dual membership — bare 'Md' is Muhammad); #291 is UNSHIPPED, and rules.md carries it as a deviates: marker on C1. The arc's bookkeeping: #291/#296 moved milestone v2.1 → v2.2 on 2026-08-01 (with #289/#293); the 2026-07-30 design doc was amended in place 2026-08-01 (A1–A6); its durable content is HERE rather than in the gitignored doc; #316 (trailing titles) was filed the same day, carrying the esq cleanup as a Related section. The approved 19-word TITLES∩suffix audit table (2026-07-30/08-01, intent for the unshipped bundle): jr, junior, phd, md, do and se DROP from TITLES (se: Structural Engineer is the PE/SE post-nominal pair, no prenominal convention — v1 residue); dr drops from SUFFIX_WORDS and sra from the suffix sets (v1 residue); sr, lt, cpl, cpt, cpo, csm, sgm, ra and vc KEEP dual membership (position decides); ms and sa keep dual membership AND join the ambiguous set (the periods gate). Derek's framing thesis for the whole table: v1's sets encoded where the v1 PARSER needed words to be, not where words can occur. Amendment A6: once SUFFIX_PHRASES ships, the glued peel's token-level post-nominal test cannot step over a phrase segment ("김민준씨, LEED AP" will not peel) — #291 owns deciding that, with a case row either way. diff --git a/docs/design/rules.md b/docs/design/rules.md index 80b01799..6b05d9bb 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -52,7 +52,9 @@ H2. Rationale: before a name, an abbreviation is almost always a initial does not, and neither does anything with interior periods, hyphens or digits. Where it fires, the shape outranks vocabulary: a period-marked opening word is a title even when - the word is suffix vocabulary. + the word is suffix vocabulary — except after a family comma, + where a part that is nothing but suffix words is the credential + run (C1) and the abbreviation opens nothing. "Rev. John Smith" → title="Rev." "Xyz. John Smith" → title="Xyz." "Smith, Major. John" → title="Major." @@ -69,6 +71,11 @@ H2. Rationale: before a name, an abbreviation is almost always a family name (C1), so no shape or vocabulary reading makes a title there. "Xyz. Smith, John" → family="Xyz. Smith" + Accepted: after a family comma a part that is nothing but suffix + words is the credential run (C1), which the abbreviation does + not open: the vocabulary decides, and "Esq." is the postnominal + it is. + "Smith, Esq." → suffix="Esq." history: decisions.md#H2 · interacts: C1, P4 · implemented: nameparser/_pipeline/_assign.py, nameparser/_pipeline/_group.py H3. Rationale: compound titles are written as a run of title words, @@ -437,7 +444,9 @@ S2. Rationale: generational suffixes and credentials are recognized unmistakably a credential when its periods are written. A trailing word of the suffix vocabulary reads as a suffix — generational forms and credential acronyms alike, and an - ambiguous acronym written with periods counts unambiguously. A + ambiguous acronym written with its periods, one after each + letter, counts unambiguously; a single trailing period is the + abbreviation shape any word can wear and does not. A BARE ambiguous acronym is consumed only when the name has words to spare — as the second of two words it stays the family name — and either reading carries the ambiguity flag. @@ -445,6 +454,7 @@ S2. Rationale: generational suffixes and credentials are recognized "John Smith M.A." → suffix="M.A." "John Smith PhD" → suffix="PhD" "John Ma" → family="Ma" · boundary + "Jack Ma." → family="Ma." · boundary Accepted: with words to spare, a bare ambiguous acronym reads as a suffix even beside an East Asian surname it more likely belongs to; and an unambiguous suffix is consumed even when @@ -593,7 +603,16 @@ C1. Rationale: a credential run after the comma means the name is in Only the part after the first comma decides. Both modes consult the vocabulary alone; by default a recognized suffix word counts even written like an initial ("V."), while strict mode vetoes - initial-shaped words. + initial-shaped words. In the listing form the part after the + comma is still read for what it is: a part that is nothing but + suffix words is the credential run and reads as suffixes, whole + — the slot after a family comma is postnominal position, so the + vocabulary's verdict comes before any title reading of the same + word — and a part that holds no name word at all, titles and suffixes + only, fixes no family boundary, so a part before the comma with + more than one name word keeps its positional read, order and + all. A name word in the part after the comma makes it the + given name, with titles before it and suffixes after. "Smith, John" → family="Smith" "سلمان، محمد" → family="سلمان" "田中、太郎" → family="" @@ -601,12 +620,35 @@ C1. Rationale: a credential run after the comma means the name is in "John Smith, V." → suffix="V." "John Smith, V." strict-comma-suffixes → family="John Smith" "Smith, PhD" → family="Smith" · boundary - "Smith, PhD" → suffix="PhD" deviates: #296 (today: suffix="") - (Today PhD lands in TITLE — the #316 trailing-title tangle - crossing C1; the marker tracks the suffix field only, so a - measured title="PhD" does not mean the marker is stale.) + "Smith, PhD" → suffix="PhD" + "Smith, Jr." → suffix="Jr." + "Smith, Sr." → suffix="Sr." + "Smith, Ph. D. Jr." → suffix="Ph. D., Jr." + "Smith, Dr." → title="Dr." + "Smith, Dr. Jr." → suffix="Jr." + "John Smith, Mr." → given="John" + "John Smith, Mr." → family="Smith" + "John Smith, Mr. Jr." → given="John" + "Smith Jr., Mr." → family="Smith" · boundary + "John Smith, Jones" → family="John Smith" + Accepted: a word of both the title and the unambiguous suffix + vocabulary reads as the postnominal after a family comma in + every spelling, the honorific's too — position decides for the + duals, and the slot is postnominal. + "Smith, Ms." → suffix="Ms." + "Smith, Ms. Jane" → title="Ms." + Accepted: a title-only part after a one-word family keeps the + family whole, and a glued honorific in it stays glued — the + honorific peel (W3) runs on the other structures, before the + comma is read; a credential after the comma still frees it. + "田中さん, Dr." → family="田中さん" + "田中さん, PhD" → suffix="さん, PhD" + Accepted: a delimiter core the policy names (T1) is a word here, + not structure — v1 applied the delimiter to the suffix-comma + form alone, and that limitation is kept as parity: "Smith, RN - + CRNA" reads given "RN" under the policy as without it. "John Smith, LEED AP" → family="Smith" deviates: #291 (today: family="John Smith") - history: decisions.md#C1 · implemented: nameparser/_pipeline/_segment.py + history: decisions.md#C1 · interacts: H2, P6 · implemented: nameparser/_pipeline/_segment.py, nameparser/_pipeline/_assign.py C2. Rationale: text beyond the recognized comma parts should be taken in without silent guessing. diff --git a/docs/release_log.rst b/docs/release_log.rst index 36cab410..e515800a 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -43,6 +43,12 @@ Release Log - Fix a particle chain and a maiden name taking a trailing generational numeral as a name word: ``"John van der Berg V"`` read family ``van der Berg V`` and ``"John née Jones Smith V"`` read maiden ``Jones Smith V``, where ``"John Smith V"`` reads suffix ``V``. Both stops asked "is this a suffix?" with the test that vetoes a bare ``V`` as an initial -- the same question the bound given-name join asked until #401 -- and now ask the suffix rule's own walk, over the name as they would leave it: family ``van der Berg``, maiden ``Jones Smith``, suffix ``V``, for ``I`` and ``X`` alike; ``"John van der J. V"`` and ``"J. née Jones Smith V"`` keep their readings, the word before the numeral being an initial as written or as left. The chain stops before a bare credential with words to spare too, as ``"John Smith Ma"`` does, where the suffix rule will still find them once the chain has run: ``"John van der Berg Ma"`` reads suffix ``Ma``, which is how 1.4.0 read it, while ``"Freiherr von Berg MA"`` keeps family ``von Berg MA``; a maiden name keeps such a credential, since the words the marker takes are the very words that made it one. A particle that is also suffix vocabulary now ends the chain where it opens the trailing run: ``"John Smith Mc V"`` reads suffix ``Mc, V``. The chain also no longer swallows the given name behind an unlisted abbreviation: ``"Xyz. van Johnson"`` and ``"Esq. van Gogh"`` read given ``van``, as ``"Dr. van Johnson"`` has since #367, the chain and the bound given-name join now asking the same test that reads the abbreviation as a title (closes #424) + - Fix a name losing its given/family split when a comma is followed only by an honorific: ``"John Smith, Mr."`` now gives given ``John``, family ``Smith``, title ``Mr.`` where it returned the whole of ``"John Smith"`` as the family name. The family-comma rule that the part before the comma is wholly the surname rests on the writer having marked where the surname ends, and a comma followed by nothing but titles marked no such thing -- ``"John Smith, Mr."`` is ``"Mr. John Smith"`` with the honorific moved to the end. A comma followed by an actual name still fixes the family (``"John Smith, Jones"`` keeps family ``John Smith``), a title followed by a postnominal is each read where it stands (``"Smith, Dr. Jr."`` is unchanged), and a single pre-comma piece has no split to keep, so ``"Smith, Dr."`` is unchanged. Because the pre-comma name is now read positionally, it also picks up the two things the merged read suppressed: the script-order table, the declared name order (``"de Mesnil Juan, Dr."`` keeps family ``de Mesnil`` under a family-first order) and the particle-or-given ambiguity report (``"Van Johnson, Mr."``). The same holds when the comma is followed by titles and postnominals with no name word between them: ``"John Smith, Mr. Jr."`` keeps its split, while ``"Smith Jr., Mr."`` -- one name word -- keeps its family + + - Fix pure postnominals being claimed as titles: ``jr``, ``junior``, ``phd``, ``do`` and ``se`` have left the default ``titles`` vocabulary, and ``dr``/``sra`` have left the suffix vocabulary they never belonged in. v1's word lists recorded where the v1 parser needed a word to be in order to do its job, not where the word can actually occur -- ``"Smith, PhD"`` returned title ``PhD`` because the title peel legitimately got there first. Each of the nineteen words in the ``titles``/suffix overlap was reviewed and given its true positional class; twelve are genuine duals and keep both memberships, with position deciding (``"Lt. Smith"`` is a title, ``"Smith, LT"`` a postnominal; ``"Ms. Smith"`` is the honorific, ``"John Smith, MS"`` the degree -- and so, after a family comma, is ``"Smith, Ms."``, the slot being postnominal). ``md`` is one of them: bare ``Md`` before a name is the Bengali and South Asian abbreviation of Muhammad (``"Md Abdul Karim"``), ``MD`` after it the degree. The cost is in leading position, where a dropped word now reads as a name: ``"PhD Smith"`` gives given ``PhD`` where 1.x gave title ``PhD``. That is the correct reading of a string nobody writes, and it is what makes ``"Do Nguyen"`` parse as the Vietnamese name it is -- and ``"Dr. Do Van Johnson"`` read given ``Do``. ``dr`` and ``sra`` are the only two words that lose *suffix* membership, so they also stop being recognized in trailing position: ``"John Smith Dr."`` now gives family ``Dr.`` rather than suffix ``Dr.``. That is the same reading every other title-only word already gets there -- ``"John Smith Prof."`` and ``"John Smith Mr."`` have always parsed this way, and the suffix entry was the only thing making ``dr`` an exception. Routing a trailing title word to ``title``, which the comma path already does and the no-comma path does not, is a separate open question (#316). An ambiguous credential acronym (``ma``, ``ed``, ``jd``, ``do``) now counts as a suffix only when written with its periods, one after each letter -- ``"John Smith M.A."`` -- and no longer on any period: ``"Jack Ma."`` keeps family ``Ma.``, as 1.4.0 read it, where 2.0 read a suffix + + - Fix a credential run after a one-word family comma reading as a title or a given name: ``"Smith, Jr."`` and ``"Smith, PhD"`` now give suffix ``Jr.``/``PhD`` where they gave title, and ``"Smith, Ph. D. Jr."`` gives suffix ``Ph. D., Jr.`` where the split credential landed in the given name (a regression from 1.4.0, which read suffix ``Ph. D.``). The position right after a family comma is postnominal position, so a run that is nothing but suffix words is read as the credential run it is before the title peel's whole-segment exception can claim it. Vocabulary still decides which words qualify -- ``"Smith, Dr."`` keeps title ``Dr.``, because ``dr`` is not postnominal vocabulary at all -- and the leading readings are untouched (``"Sr. Garcia"`` is still title ``Sr.`` while ``"Smith, Sr."`` is now suffix ``Sr.``); a name word in the run makes it the given-and-suffix reading it always had (``"Smith, John Jr."``) (closes #296, closes #325) + - Fix a name opening with a particle that is *never* a given name being split at the particle under a family-first name order -- ``Policy(name_order=FAMILY_FIRST)`` and ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)`` alike, and identically: ``"de Mesnil"`` read as family ``de``, given ``Mesnil``, and ``"de la Vega"`` as family ``de``, given ``la Vega``. Each is now the whole surname, as it has always been in the default order. The rule enforcing it asked for the particle by the ``GIVEN`` role, which under a family-first order belongs to the token *after* the particle, so the test read the wrong word and declined. It now also asks by position -- the piece that opens the name -- so both shapes of the same rule are caught: where such a particle stands alone as a piece, either opening the name or in the given position, the name is left with no given name at all, the given and the middles folding into the family. Standing *alone* is the whole of it, and the rule claims nothing wider: ``"Juan de la Vega"`` under ``FAMILY_FIRST`` still reports given ``de la Vega``, because there the particle chained onto the words after it rather than standing alone, and a bare ``"de"`` with nothing to fold into is still reported as the given name. The decision behind the fix: a word that can never be a given name leaves ``name_order`` nothing to decide, so declaring family-first is not a reason to make ``de`` a surname on its own. A leading particle that *may* be a given name is genuinely order-dependent and is untouched -- ``"van Gogh"`` still reads as family ``van``, given ``Gogh`` under both family-first orders. This is also what gives ``Lexicon.particles_ambiguous`` an effect outside the default order: taking a word out of it now changes the parsed fields under a family-first order, where before it moved only the ambiguity report. Seven of the 751 differential corpus names move, the same seven under each family-first order; default-order output is byte-identical over all 751, at the 1.4.0, 2.0.0 and 2.1.0 differential baselines alike (closes #359) - Fix a family name made only of particle words reporting no base on the 2.0 API, so the surname vanished from ``family_base`` and, on both APIs, from the initials: ``parse("Anh Do")`` gave family ``Do`` with ``family_base`` ``''`` and initials ``A.``, and under ``Policy(name_order=FAMILY_FIRST)`` ``"Del Toro"`` gave family ``Del`` the same way. The empty base was a 2.0 regression rather than a longstanding bug -- 1.4.0's own guard kept ``HumanName("Anh Do").last_base`` at ``Do``, and the facade has kept it right throughout; what 1.4.0 and 2.1 shared was the missing initial. A particle earns its name by joining forward to the word it modifies, so a particle standing alone in a name part is not doing a particle's work there and reads as an ordinary name word: it anchors the base, leaves the particles view, and contributes an initial. ``"Anh Do"`` is now base ``Do``, initials ``A. D.``; ``"Juan van der"`` is base ``van der``, initials ``J. v. d.``; ``"Nguyen, Van Le"`` initials ``V. L. N.`` where the middle name used to be dropped. Position decides this, not vocabulary -- whether the word is borne as a surname somewhere does not enter into it, which reverses the reasoning ``rules.md#R2`` carried before. The invariant it exists to hold: a non-empty family always has a non-empty base, because a particle needs a base to attach to. Where the particles DO join a name word nothing changes -- ``"Juan de la Vega"`` keeps base ``Vega``, particles ``de la`` and initials ``J. V.``. The parse fields themselves do not move: ``title``, ``given``, ``middle``, ``family``, ``suffix``, ``nickname`` and ``maiden`` are byte-identical over all 751 differential corpus names in all three name orders, and only the derived views and the initials change -- which is why the differential harness, which compares those seven roles, reports no diffs for this at any baseline. One consequence to know about: where the whole family is particles AND a tussenvoegsel was attached to it (#379), the two APIs order the initials differently -- ``parse("der, y van").initials()`` is ``y. d. v.`` against the facade's ``y. v. d.`` -- because ``initials()`` reads tokens in written order while the family FIELD renders folded words first. That affects 87 constructed inputs and no corpus name; it is a pre-existing ordering gap in ``initials()`` that this change makes visible, tracked at #408 (closes #385, closes #402) diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index c7509918..2459633a 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -19,10 +19,15 @@ last name-position piece, the rest are suffixes. The v1 single-name+ nickname rule lives here (decisions.md#N3): a nonempty nickname beside exactly one piece in total puts that piece in FAMILY. -FAMILY_COMMA: segment 0 wholly FAMILY (v1 parity); segment 1 gets -leading titles, then given, then middles with strict-suffix pieces to -suffix; segments 2+ are suffixes (lenient -- segment already flagged -non-suffixy ones COMMA_STRUCTURE). +FAMILY_COMMA: segment 0 wholly FAMILY (v1 parity) UNLESS segment 1 +holds no name word (titles and suffixes only), which fixed no family +boundary -- there segment 0 takes the NO_COMMA positional read instead, +order and all ('John Smith, Dr.', 'John Smith, Mr. Jr.'); segment +1 is wholly SUFFIX when it is nothing but suffix pieces ('Smith, Jr.', +'Smith, Ph. D. Jr.' -- the credential run C1 describes, in the listing +form), else gets leading titles, then given, then middles with +strict-suffix pieces to suffix; segments 2+ are suffixes (lenient -- +segment already flagged non-suffixy ones COMMA_STRUCTURE). SUFFIX_COMMA: segment 0 as NO_COMMA; segments 1+ wholly SUFFIX. Emits PARTICLE_OR_GIVEN when the leading name piece is a lone particles_ambiguous token with more pieces following ("Van Johnson", @@ -37,7 +42,8 @@ effective_script, is_suffix_lenient, resolve_script_set, ) from nameparser._pipeline._group import ( - _is_suffix_piece, _leading_titles, _peel_trailing, _peel_walk, + _is_leading_title, _is_suffix_piece, _leading_titles, _peel_trailing, + _peel_walk, ) from nameparser._pipeline._state import ( ParseState, PendingAmbiguity, Structure, WorkToken, @@ -250,6 +256,32 @@ def _assign_main(seg_idx: int, state: ParseState, return order +def _segment_holds_no_name(pieces: tuple[tuple[int, ...], ...], + ptags: tuple[frozenset[str], ...], + tokens: list[WorkToken]) -> bool: + """The segment is titles and suffixes only ('John Smith, Dr.', + 'John Smith, Mr. Jr.') -- nothing in it is a name word. + + The FAMILY_COMMA rule "segment 0 is wholly the family name" rests on + the writer having said where the family name ends. A comma followed + by no name word said no such thing -- 'John Smith, Dr.' is 'Dr. John + Smith' with the honorific moved, and 'John Smith, Mr. Jr.' the same + with the postnominal along -- so the pre-comma name keeps its + positional read instead of being merged. Uses the same + _is_leading_title predicate the peel does, period-abbreviation + inference included, so the two cannot disagree about what a title + is; a suffix piece counts as what it is, so a mixed run like + 'Smith, Dr. Jr.' is a title and a postnominal, each read where it + stands, and never a title run 'Dr. Jr.'. An empty segment + ('Doe,, Jr.') holds no title to read by. + """ + if not pieces: + return False + return all(_is_suffix_piece(pieces[k], ptags[k], tokens) + or _is_leading_title(pieces[k], ptags[k], tokens) + for k in range(len(pieces))) + + def assign(state: ParseState) -> ParseState: tokens = list(state.tokens) ambiguities = list(state.ambiguities) @@ -263,44 +295,93 @@ def assign(state: ParseState) -> ParseState: order = _assign_main(0, state, tokens, ambiguities) tail = 1 else: # FAMILY_COMMA - # PARTICLE_OR_GIVEN is deliberately not emitted here: after a - # comma the family is already fixed, so a leading given-position - # particle is not meaningfully ambiguous. script_orders is not - # consulted here for the parallel reason -- the comma already - # fixed the family, so there is no positional read to override. + # PARTICLE_OR_GIVEN is deliberately not emitted on the + # wholly-family read: after a comma that fixed the family, a + # leading given-position particle is not meaningfully + # ambiguous, and script_orders is not consulted for the parallel + # reason. The positional read below (a comma followed by no + # name word) emits and consults both, being the no-comma read + # of segment 0; group's chain emitter still does not, since + # group runs before assign decides which read applies (recorded + # at decisions.md#C1). # v1: "lastname part may have suffixes in it" -- the first # piece is always the family even if suffix-shaped; any later # strict-suffix piece goes to SUFFIX per piece ('Smith Jr., # John' -> family=Smith, suffix=Jr.) fam_pieces = state.pieces[0] fam_tags = state.piece_tags[0] - for k, piece in enumerate(fam_pieces): - if k > 0 and _is_suffix_piece(piece, fam_tags[k], tokens): - _set_roles(tokens, piece, Role.SUFFIX) - else: - _set_roles(tokens, piece, Role.FAMILY) + # A comma followed by no name word fixed nothing, so segment 0 + # keeps its positional read -- including script_orders, the + # particle fork, and the ORDER, which post_rules' family-first + # fold (P1) and its leading-piece scan key on: "assign records + # no order after a family comma" is the invariant those rules + # rest on, and this is the path that gives one, so they read + # segment 0 as the name it is ('de Mesnil Juan, Dr.' under a + # family-first order keeps family 'de Mesnil'; the test review + # found the fold missing it). The wholly-family branch below + # suppresses all three precisely because the comma HAD fixed + # the family. Needs two NAME pieces: with one, the positional + # read would make it a lone GIVEN, which is worse than what it + # replaces -- and the count is of name pieces, since the + # positional read peels a trailing suffix first: 'Smith Jr., + # Mr.' has two pieces and one name, and read positionally lost + # its family (the code review). + no_name = _segment_holds_no_name(state.pieces[1], + state.piece_tags[1], tokens) + if no_name and sum( + 1 for k, piece in enumerate(fam_pieces) + if not _is_suffix_piece(piece, fam_tags[k], tokens)) > 1: + order = _assign_main(0, state, tokens, ambiguities) + else: + for k, piece in enumerate(fam_pieces): + if k > 0 and _is_suffix_piece(piece, fam_tags[k], tokens): + _set_roles(tokens, piece, Role.SUFFIX) + else: + _set_roles(tokens, piece, Role.FAMILY) if len(state.segments) > 1: pieces = state.pieces[1] ptags = state.piece_tags[1] - n = _peel_leading_titles(pieces, ptags, tokens) - given_done = False - for m in range(n, len(pieces)): - # v1 walk order: the first non-title piece is ALWAYS - # the given, before any suffix check -- - # 'Hardman, RN - CRNA' keeps first='RN'. One deliberate - # 2.0 deviation, classified fix(comma-family): when that - # piece is the segment's ONLY piece and unambiguously - # suffix-shaped ('Andrews, M.D.'), it is a suffix -- v1 - # made it the given. - if not given_done: - if (m == len(pieces) - 1 - and _is_suffix_piece(pieces[m], ptags[m], - tokens)): - _set_roles(tokens, pieces[m], Role.SUFFIX) - continue - _set_roles(tokens, pieces[m], Role.GIVEN) - given_done = True - continue + # rules.md#C1: "a credential run after the comma means the + # name is in natural order with suffixes appended" -- and + # with one word before the comma the listing form holds, + # the family is that word, and the run is still the + # credential run. A no-name segment is read piece by + # piece, the suffix vocabulary's verdict BEFORE the title + # reading of the same word: the slot after a family comma + # is postnominal position, so a segment of nothing but + # suffix pieces is the credential run, whole (#296: + # 'Smith, Jr.' read title 'Jr.' through the period- + # abbreviation inference; #325: 'Smith, Ph. D. Jr.' put + # the split credential in the given name, the lone-piece + # route not applying), and a mixed run is a title and a + # postnominal, each where it stands ('Smith, Mr. Jr.'). + # Vocabulary decides which words qualify -- 'Smith, Dr.' + # reads the title, 'dr' not being suffix vocabulary since + # the audit -- and position breaks the tie for the genuine + # duals ('Smith, Sr.' is Senior, 'Sr. Garcia' Señor). A + # name word in the segment makes it v1's walk ('Smith, + # John Jr.'). + if no_name: + for k, piece in enumerate(pieces): + _set_roles(tokens, piece, + Role.SUFFIX if _is_suffix_piece( + piece, ptags[k], tokens) + else Role.TITLE) + n = len(pieces) + else: + n = _peel_leading_titles(pieces, ptags, tokens) + # v1 walk order: the first non-title piece is ALWAYS the + # given, before any suffix check -- 'Hardman, RN - CRNA' + # keeps first='RN'. The one deliberate 2.0 deviation, + # classified fix(comma-family) -- a last piece that is + # unambiguously suffix-shaped is a suffix, where v1 made + # it the given ('Andrews, M.D.', 'Smith, Dr. Jr.') -- is + # the no-name read above now: a segment whose only + # non-title piece is a suffix piece holds no name word, + # so the walk here never meets the case. + if n < len(pieces): + _set_roles(tokens, pieces[n], Role.GIVEN) + for m in range(n + 1, len(pieces)): # trailing piece of a two-part name is unambiguously # positioned: v1 accepts the lenient test there # ('Smith, John V' -> suffix='V', #144); with a third diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index 11e45d63..d27d6964 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -37,7 +37,9 @@ # rules.md#S2: "a trailing word of the suffix vocabulary reads as a # suffix — generational forms and credential acronyms alike, and an -# ambiguous acronym written with periods counts unambiguously. A +# ambiguous acronym written with its periods, one after each +# letter, counts unambiguously; a single trailing period is the +# abbreviation shape any word can wear and does not. A # bare ambiguous acronym is consumed only when the name has words to # spare" def _tags_for(token: WorkToken, state: ParseState) -> frozenset[str]: diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index 24c123d7..1a91c481 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -161,8 +161,10 @@ class Peel(NamedTuple): # rules.md#S2: "a trailing word of the suffix vocabulary reads as a # suffix — generational forms and credential acronyms alike, and an -# ambiguous acronym written with periods counts unambiguously. A BARE -# ambiguous acronym is consumed only when the name has words to spare" +# ambiguous acronym written with its periods, one after each letter, +# counts unambiguously; a single trailing period is the abbreviation +# shape any word can wear and does not. A BARE ambiguous acronym is +# consumed only when the name has words to spare" # (v1's are_suffixes tail rule, with the roman-numeral special) def _peel_walk(start: int, ptags: Sequence[Set[str]], skip: Set[int] = frozenset()) -> list[int]: diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index be910498..74adca1c 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -53,11 +53,15 @@ def _leading_name_piece(state: ParseState, those is walked past -- title and suffix pieces, but NICKNAME and MAIDEN as well, and anything assign left unroled -- and any number of them, not only a single leading title. The segment is 0, except - under a family comma, where segment 0 is already fixed as the - surname and the name continues in segment 1. Empty on either of - two exits: that segment does not exist, or none of its pieces - holds a name token.""" - seg = 1 if state.structure is Structure.FAMILY_COMMA else 0 + under a family comma that fixed the surname, where the name + continues in segment 1 -- assign records no order there. A family + comma followed by no name word fixed nothing, and assign reads + segment 0 positionally and records the order (#296's bundle), so + the name is segment 0 again. Empty on either of two exits: that + segment does not exist, or none of its pieces holds a name + token.""" + seg = (1 if state.structure is Structure.FAMILY_COMMA + and state.order is None else 0) if seg >= len(state.pieces): return () for piece in state.pieces[seg]: @@ -331,10 +335,12 @@ def post_rules(state: ParseState) -> ParseState: # -- 0 hits over 740,552 instrumented guard sites. The reason # is structural: the only rule that can leave a MIDDLE with no # GIVEN ahead of it in segment 1 is P1's family-first - # redistribution, which is gated on `state.order`, and assign - # never records an order on the FAMILY_COMMA path (the comma - # has already fixed the family). P6 runs only on that path, - # so the branch cannot be reached from here. + # redistribution, which is gated on `state.order`. On the + # FAMILY_COMMA path assign records an order only where + # segment 1 holds no name word (#296's positional read), and + # a no-name segment holds no MIDDLE for the fold to leave + # either. P6 runs only on that path, so the branch cannot be + # reached from here. if k and any(tokens[i].role is Role.GIVEN for piece in seg[:k] for i in piece): # A range, though only ever one piece today: grouping's diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index 8a3c92b0..aadcfe7c 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -132,6 +132,19 @@ def is_initial(text: str) -> bool: return is_initial_shaped(text) and not _in_initialless_script(text) +_DOTTED = re.compile(r"(?:[^\W\d_]\.)+") + + +def _dotted(text: str) -> bool: + """Written with its periods: one after each letter ('M.A.', + 'J.D.'), the acronym's own spelling. A single trailing period + ('Ma.', 'Ed.', 'Ms.') is the abbreviation shape any word can wear + -- the honorific's, a name's -- and is not the gate's "written + with periods" (rules.md#S2). Until #296's review the gate was + "any period", and 'Smith, Ms.' passed it as the degree.""" + return _DOTTED.fullmatch(text) is not None + + def suffix_as_written(n: str, text: str, lexicon: Lexicon) -> bool: """Counts as a suffix as written, with NO initial veto (the veto differs by caller): unambiguous suffix vocabulary, or an ambiguous @@ -149,7 +162,7 @@ def suffix_as_written(n: str, text: str, lexicon: Lexicon) -> bool: # removed periods only for the suffix_acronyms test); suffix WORDS # match on the plain normalized form a = n.replace(".", "") - if "." in text and a in lexicon.suffix_acronyms_ambiguous: + if a in lexicon.suffix_acronyms_ambiguous and _dotted(text): return True return (a in lexicon.suffix_acronyms and a not in lexicon.suffix_acronyms_ambiguous) \ @@ -159,7 +172,7 @@ def suffix_as_written(n: str, text: str, lexicon: Lexicon) -> bool: def _is_suffix_strict_n(n: str, text: str, lexicon: Lexicon) -> bool: if is_initial(text): # period-written ambiguous acronyms are exempt from the veto - return "." in text and \ + return _dotted(text) and \ n.replace(".", "") in lexicon.suffix_acronyms_ambiguous return suffix_as_written(n, text, lexicon) diff --git a/nameparser/config/suffixes.py b/nameparser/config/suffixes.py index d2b21bec..58ca748e 100644 --- a/nameparser/config/suffixes.py +++ b/nameparser/config/suffixes.py @@ -6,7 +6,6 @@ # 'ст' especially is a plausible false-positive risk (many two- # letter Cyrillic abbreviations exist), and both are short enough # to worry about. Not shipped in this pass. - 'dr', 'esq', 'esquire', 'jr', @@ -186,6 +185,15 @@ # also gates bare recognition: an ambiguous acronym counts as a # suffix only when written with periods ('M.A.' yes, 'Ma' no), so # 'Jack Ma' keeps its family name. + # + # NOT 'ms' or 'sa', though #296's audit table put them here for the + # leading-title collision (bare "Ms" the honorific, "M.S." the + # degree): the gate is position-blind and the collision is not. + # Gated, 'John Smith, MS' lost its suffix-comma route and read + # title 'MS', and 'Smith, Ms.' passed the gate on its one period + # and read as a credential anyway. Both words are genuine duals -- + # title and unambiguous suffix -- and position decides, as for + # 'sr' and 'lt' (decisions.md#C1). 'do', 'ed', 'jd', @@ -815,7 +823,6 @@ 'siie', 'smieee', 'sphr', - 'sra', 'sscp', 'stb', 'stmieee', diff --git a/nameparser/config/titles.py b/nameparser/config/titles.py index 4b9fe674..331d5d52 100644 --- a/nameparser/config/titles.py +++ b/nameparser/config/titles.py @@ -276,7 +276,6 @@ 'dissident', 'district', 'division', - 'do', 'docent', 'docket', 'doctor', @@ -394,10 +393,8 @@ 'investor', 'journalist', 'journeyman', - 'jr', 'judge', 'judicial', - 'junior', 'jurist', 'keyboardist', 'kingdom', @@ -517,7 +514,6 @@ 'petty', 'pfc', 'pharaoh', - 'phd', 'philantropist', 'philosopher', 'photographer', @@ -604,7 +600,6 @@ 'scientist', 'scpo', 'screenwriter', - 'se', 'secretary', 'security', 'seigneur', diff --git a/tests/test_particles.py b/tests/test_particles.py index 3edeb863..f625d614 100644 --- a/tests/test_particles.py +++ b/tests/test_particles.py @@ -48,36 +48,39 @@ def test_prefix_before_two_part_last_name_with_acronym_suffix(self) -> None: self.m(hn.suffix, "M.D.", hn) def test_title_before_and_after_prefixed_last_name(self) -> None: - # Issue #100: a repeated title/suffix token ("dr") before AND after a + # Issue #100: a repeated title/suffix token before AND after a # prefixed last name used to corrupt the middle name into # " dr Vincent van" because the suffix-boundary lookup matched the - # LEADING "dr" instead of the trailing one. - hn = HumanName("dr Vincent van Gogh dr") - self.m(hn.title, "dr", hn) + # LEADING token instead of the trailing one. The token was "dr" + # until #296's audit took 'dr' out of the suffix vocabulary (it + # is not a postnominal); "lt" is both a title and a suffix, which + # is the only property these three tests need. + hn = HumanName("lt Vincent van Gogh lt") + self.m(hn.title, "lt", hn) self.m(hn.first, "Vincent", hn) self.m(hn.middle, "", hn) self.m(hn.last, "van Gogh", hn) - self.m(hn.suffix, "dr", hn) + self.m(hn.suffix, "lt", hn) def test_suffix_token_collision_with_two_word_prefix(self) -> None: # Same fix as #100 but with a two-word prefix ("van der"). Exercises a # different iteration count through the prefix-joining loop. - hn = HumanName("dr Vincent van der Gogh dr") - self.m(hn.title, "dr", hn) + hn = HumanName("lt Vincent van der Gogh lt") + self.m(hn.title, "lt", hn) self.m(hn.first, "Vincent", hn) self.m(hn.middle, "", hn) self.m(hn.last, "van der Gogh", hn) - self.m(hn.suffix, "dr", hn) + self.m(hn.suffix, "lt", hn) def test_title_before_and_after_prefixed_last_name_with_middle(self) -> None: # The pre-fix bug corrupted the middle field; verify it is not disturbed # when a genuine middle name is present alongside the repeated token. - hn = HumanName("dr Vincent James van Gogh dr") - self.m(hn.title, "dr", hn) + hn = HumanName("lt Vincent James van Gogh lt") + self.m(hn.title, "lt", hn) self.m(hn.first, "Vincent", hn) self.m(hn.middle, "James", hn) self.m(hn.last, "van Gogh", hn) - self.m(hn.suffix, "dr", hn) + self.m(hn.suffix, "lt", hn) @pytest.mark.timeout(2) def test_many_repeated_prefixes_does_not_blow_up(self) -> None: diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 04c03fd1..210204e5 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -348,14 +348,15 @@ def __post_init__(self) -> None: "'first piece that is not a title' test broke: it " "skipped 'St'/'Do'/'Freiherr' and collapsed the " "untitled 'St John Smith' into one given name"), - Case("titled_ambiguous_particle_no_op_chain", "Do Van Jr.", - {"title": "Do", "given": "Van", "suffix": "Jr."}, + Case("titled_ambiguous_particle_no_op_chain", "St Van Jr.", + {"title": "St", "given": "Van", "suffix": "Jr."}, notes="the piece after the particle is a suffix, so the chain " "scan never advances and the merge is a no-op -- nothing " "was chained, so there is no fork to report (the emitter " "fired here for all 39 ambiguous particles, and _assign " - "double-reported the same token). Spelled with 'Do' " - "rather than the 'Dr.' this row carried until 2.2: " + "double-reported the same token). Spelled with 'St' " + "since #296's audit took 'do' out of TITLES; before that " + "with 'Do', and before 2.2 with 'Dr.': " "under #367 a plain title is transparent, so 'Dr. Van " "Jr.' leaves Van the leading name piece and the chain " "loop skips it without ever reaching the no-op. 'Do' is " @@ -1376,6 +1377,324 @@ def __post_init__(self) -> None: {"title": "Dr.", "family": "Smith"}, classification="fix(comma-family)", notes="pre-comma is definitionally family; v1 put it in first"), + Case("family_comma_all_title_segment_keeps_split", "John Smith, Mr.", + {"title": "Mr.", "given": "John", "family": "Smith"}, + classification="fix(comma-family)", + notes="a comma followed only by titles said nothing about " + "where the family name ends, so segment 0 keeps its " + "positional read instead of merging into one family " + "name (v1 read first 'John Smith'; 2.0 read it as the " + "family, the comma-precomma-family move)"), + Case("family_comma_all_title_segment_needs_two_pieces", "Smith, Dr.", + {"title": "Dr.", "family": "Smith"}, + classification="fix(comma-family)", + notes="the guard on family_comma_all_title_segment_keeps_split: " + "one pre-comma piece has no split to keep, and the " + "positional read would make it a lone GIVEN"), + Case("family_comma_untitled_segment_still_merges", "John Smith, Jones", + {"given": "Jones", "family": "John Smith"}, + notes="the non-flip: a post-comma NAME means the comma did fix " + "the family, so segment 0 stays wholly family (v1 parity)"), + # -- #296: the TITLES/suffix overlap audit. A word that is only ever + # a postnominal leaves TITLES, so the title peel stops claiming it + # first. Leading position is the price and is pinned here. + Case("audit_phd_leading_is_a_name", "PhD Smith", + {"given": "PhD", "family": "Smith"}, + classification="fix(#296)", + notes="'phd' left TITLES because M.D./Ph.D. are postnominal " + "only; nothing is prenominal-'PhD', so the leading " + "position falls through to the positional read"), + Case("audit_jr_leading_is_a_name", "Jr Smith", + {"given": "Jr", "family": "Smith"}, + classification="fix(#296)", + notes="same as audit_phd_leading_is_a_name: 'jr' is never " + "prenominal in any tradition"), + Case("audit_md_leading_stays_a_title", "Md Abdul Karim", + {"title": "Md", "given": "Abdul", "family": "Karim"}, + notes="the one word the 2026-07-30 table got wrong: 'md' KEEPS " + "dual membership. Bare 'Md' before a name is the " + "Bengali and South Asian Muslim abbreviation of " + "Muhammad (#343/#345's corpus rows), a prenominal use " + "the 'postnominal only' disposition did not consider; " + "'MD' after the name is the degree. Position decides, " + "as for 'sr'"), + Case("audit_md_after_comma_is_the_degree", "Smith, MD", + {"family": "Smith", "suffix": "MD"}, + classification="fix(#296)", + notes="the third leg of 'position decides' for 'md'"), + Case("audit_do_leading_is_a_name", "Do Nguyen", + {"given": "Do", "family": "Nguyen"}, + ambiguities=("particle-or-given",), + classification="fix(#296)", + notes="the disposition's own argument, realized: 'Do' is a " + "Vietnamese name, and dropping the title membership is " + "what lets it be read as one -- with the fork 'Van " + "Johnson' reports, since 'do' is an ambiguous particle " + "too and the title membership had been hiding the fork"), + Case("audit_se_leading_is_a_name", "SE Smith", + {"given": "SE", "family": "Smith"}, + classification="fix(#296)", + notes="Structural Engineer is a US licensure postnominal (the " + "PE/SE pair); no prenominal SE convention exists"), + Case("audit_junior_leading_is_a_name", "Junior Smith", + {"given": "Junior", "family": "Smith"}, + classification="fix(#296)", + notes="and 'Junior' is a real given name besides"), + # the non-flips: trailing position was already right and must stay + Case("audit_phd_trailing_unchanged", "John Smith PhD", + {"given": "John", "family": "Smith", "suffix": "PhD"}), + Case("audit_jr_trailing_unchanged", "John Smith Jr.", + {"given": "John", "family": "Smith", "suffix": "Jr."}), + Case("audit_lt_leading_stays_a_title", "Lt. Smith", + {"title": "Lt.", "family": "Smith"}, + notes="'lt' KEPT its dual membership -- a prenominal rank with " + "real retired-designation postnominal use; position " + "decides, so leading is untouched"), + Case("audit_sr_leading_stays_a_title", "Sr. Garcia", + {"title": "Sr.", "family": "Garcia"}, + notes="Señor (leading) vs Senior (trailing): kept dual, and " + "the leading read is the peel's normal path"), + Case("audit_sra_leading_stays_a_title", "Sra Garcia", + {"title": "Sra", "family": "Garcia"}, + notes="Señora is title-only; the SUFFIX_ACRONYMS entry was v1 " + "residue and is what got dropped, not this"), + Case("audit_dr_leading_stays_a_title", "Dr. Smith", + {"title": "Dr.", "family": "Smith"}), + Case("audit_dr_after_comma_is_a_title", "Smith, Dr.", + {"title": "Dr.", "family": "Smith"}, + classification="fix(comma-family)", + notes="'dr' left SUFFIX_WORDS: 'Dr.' is not a postnominal in " + "any tradition and the entry was v1 residue. This is " + "what keeps the postnominal reading of a lone post-comma " + "suffix piece from taking it -- the vocabulary decides, " + "not the position"), + Case("audit_dr_after_two_word_comma_keeps_the_split", "John Smith, Dr.", + {"title": "Dr.", "given": "John", "family": "Smith"}, + classification="fix(comma-family)", + notes="with 'dr' gone from the suffix sets the post-comma word " + "is a title only, so the no-name-word repair keeps the " + "pre-comma split. v1, and 2.0 through master, kept the " + "same split by a different route -- 'dr' was " + "suffix-tagged, making this a SUFFIX_COMMA -- but put " + "'Dr.' in suffix, not title: a change against every " + "baseline"), + Case("audit_sra_after_comma_is_a_title", "Smith, Sra", + {"title": "Sra", "family": "Smith"}, + classification="fix(comma-family)", + notes="the same removal on the acronym side"), + Case("audit_ms_after_comma_is_the_degree", "Smith, MS", + {"family": "Smith", "suffix": "MS"}, + classification="fix(#296)", + notes="'ms' is a genuine dual -- 'Ms.' leading, 'MS' the " + "degree trailing -- and position decides. The 2026-07-30 " + "table put it in the AMBIGUOUS set so bare 'MS' would " + "read as the honorific; measured, that gate is " + "position-blind: 'John Smith, MS' lost its suffix-comma " + "route and read title 'MS', and 'Smith, Ms.' passed the " + "gate on its one period anyway. The second deviation " + "from the table, with 'md'"), + Case("audit_ms_after_two_word_comma_is_the_degree", "John Smith, MS", + {"given": "John", "family": "Smith", "suffix": "MS"}, + notes="the common listing, unchanged at every baseline -- " + "what the ambiguous gate would have cost"), + Case("audit_ms_leading_is_the_honorific", "Ms. Smith", + {"title": "Ms.", "family": "Smith"}), + Case("audit_perioded_ms_is_the_degree", "Smith, M.S.", + {"family": "Smith", "suffix": "M.S."}), + Case("audit_ms_honorific_spelling_after_comma_is_the_postnominal", + "Smith, Ms.", + {"family": "Smith", "suffix": "Ms."}, + classification="fix(#296)", + notes="the cost of the dual, recorded: after a family comma " + "the slot is postnominal and the vocabulary says suffix, " + "whatever the period suggests -- as for 'Smith, Sr.'. " + "Every baseline read title 'Ms.' (the design-docs " + "review found the flip unrecorded). 'Smith, Ms. Jane' " + "still reads the title: a name word follows"), + Case("audit_ms_before_a_name_after_comma_is_the_title", + "Smith, Ms. Jane", + {"title": "Ms.", "given": "Jane", "family": "Smith"}), + Case("audit_sa_after_comma_is_the_postnominal", "Smith, SA", + {"family": "Smith", "suffix": "SA"}, + classification="fix(#296)", + notes="the same dual: Special Agent leading, the business " + "form trailing"), + Case("audit_perioded_sa_is_the_postnominal", "Smith, S.A.", + {"family": "Smith", "suffix": "S.A."}), + Case("audit_bare_do_after_comma_is_a_name", "Smith, DO", + {"given": "DO", "family": "Smith"}, + classification="fix(#296)", + notes="'do' left TITLES but was already AMBIGUOUS, so the bare " + "spelling is neither title nor suffix and falls to the " + "given position -- the period gate handles the real " + "collision, which is that 'Do' is a name"), + Case("audit_perioded_do_after_comma_is_a_suffix", "Smith, D.O.", + {"family": "Smith", "suffix": "D.O."}), + # -- the TRAILING half of the same two removals. `dr` and `sra` are + # the ONLY audit words that lose SUFFIX membership; every other + # audit word keeps its suffix membership, so trailing position is + # untouched for them. + Case("audit_dr_trailing_joins_the_title_word_gap", "John Smith Dr.", + {"given": "John", "middle": "Smith", "family": "Dr."}, + classification="fix(#296)", + notes="'dr' left SUFFIX_WORDS, so a trailing 'Dr.' is no " + "longer suffix vocabulary and falls to the positional " + "read, taking the family name with it. NOT a new defect " + "class -- no trailing title word routes to title on the " + "no-comma path, so 'John Smith Prof.' and 'John Smith " + "Mr.' already read this way (both pinned below; #316 is " + "the open question). The v1-residue suffix entry was the " + "only thing making 'dr' behave unlike every other " + "title-only word. This row records that 'dr' JOINED the " + "existing behavior, not that the behavior is right"), + Case("audit_sra_trailing_joins_the_title_word_gap", "John Smith Sra", + {"given": "John", "middle": "Smith", "family": "Sra"}, + classification="fix(#296)", + notes="the same move for the other word losing suffix " + "membership"), + Case("family_comma_lone_generational_suffix", "Smith, Jr.", + {"family": "Smith", "suffix": "Jr."}, + classification="fix(#296)", + notes="the issue as filed: the peel's whole-segment exception " + "claimed 'Jr.' through the period-abbreviation inference " + "even after the audit, so the ordering change is what " + "actually reaches this input"), + Case("family_comma_lone_generational_suffix_bare", "Smith, Jr", + {"family": "Smith", "suffix": "Jr"}, + classification="fix(#296)"), + Case("family_comma_lone_degree", "Smith, PhD", + {"family": "Smith", "suffix": "PhD"}, + classification="fix(#296)"), + Case("family_comma_dual_word_reads_postnominal", "Smith, Sr.", + {"family": "Smith", "suffix": "Sr."}, + classification="fix(#296)", + notes="Señor vs Senior: 'sr' keeps both memberships and this " + "is the position that picks Senior"), + Case("family_comma_dual_rank_reads_postnominal", "Smith, CPT", + {"family": "Smith", "suffix": "CPT"}, + classification="fix(#296)", + notes="the retired-designation reading of a prenominal rank"), + Case("family_comma_lone_esquire_is_the_postnominal", "Smith, Esq.", + {"family": "Smith", "suffix": "Esq."}, + classification="fix(#296)", + notes="H2's period-abbreviation inference reads a LEADING " + "'Esq.' as a title ('Esq. Smith'); after a family comma " + "the slot is postnominal and the vocabulary says " + "suffix, so the inference does not run"), + # -- #325: the whole credential run, not the lone piece + Case("family_comma_split_credential_run", "Smith, Ph. D. Jr.", + {"family": "Smith", "suffix": "Ph. D., Jr."}, + classification="fix(#325)", + notes="one word before the comma, the space-split 'Ph. D.' " + "and a suffix after it: the lone-piece route did not " + "apply and the merged credential fell through to the " + "given name (a 1.4.0 regression -- v1 read suffix 'Ph. " + "D.', title 'Jr.'). A run that is nothing but suffix " + "pieces is the credential run C1 describes, whole"), + Case("family_comma_credential_run_then_numeral", "Smith, Ph. D. III", + {"family": "Smith", "suffix": "Ph. D., III"}, + classification="fix(#325)"), + Case("family_comma_two_credentials", "Smith, PhD Jr.", + {"family": "Smith", "suffix": "PhD, Jr."}, + classification="fix(#325)", + notes="'PhD' led the run as a title until the audit; the audit " + "alone would have made it the given name, which is why " + "the ordering shipped in the same commit. The run is " + "suffixes"), + Case("family_comma_run_with_a_name_is_not_a_run", "Smith, John Jr.", + {"given": "John", "family": "Smith", "suffix": "Jr."}, + notes="the non-flip: a name word in the run makes it the " + "given-and-suffix walk v1 had"), + Case("family_comma_title_then_suffix", "Smith, Dr. Jr.", + {"title": "Dr.", "family": "Smith", "suffix": "Jr."}, + classification="fix(comma-family)", + notes="a title and a postnominal, each read where it stands " + "-- the 2.0 deviation's other case (v1 read first " + "'Jr.'), and what keeps the no-name-word test from " + "reading 'Dr. Jr.' as one title run"), + Case("family_comma_title_then_suffix_mr", "Smith, Mr. Jr.", + {"title": "Mr.", "family": "Smith", "suffix": "Jr."}, + classification="fix(comma-family)"), + Case("family_comma_title_then_suffix_keeps_the_split", + "John Smith, Mr. Jr.", + {"title": "Mr.", "given": "John", "family": "Smith", + "suffix": "Jr."}, + classification="fix(#296)", + notes="no name word after the comma, so it fixed no family " + "boundary -- the same reasoning as 'John Smith, Mr.'; " + "v1 read first 'Jr.', last 'John Smith', and 2.0 through " + "master family 'John Smith' (the design-docs review " + "found C1 silent on the shape)"), + Case("family_comma_title_run_keeps_the_split", "John Smith, Mr. Dr.", + {"title": "Mr. Dr.", "given": "John", "family": "Smith"}, + classification="fix(#296)"), + Case("family_comma_title_run_one_word", "Smith, Mr. Dr.", + {"title": "Mr. Dr.", "family": "Smith"}, + classification="fix(#296)", + notes="one pre-comma piece, no split to keep; the run is " + "titles (master read suffix 'Dr.')"), + Case("family_comma_three_segments_credential_run", "Smith, Jr., PhD", + {"family": "Smith", "suffix": "Jr., PhD"}, + classification="fix(#325)", + notes="segments 2+ compose with the credential run"), + Case("family_comma_suffixed_family_before_a_title", "Smith Jr., Dr.", + {"title": "Dr.", "family": "Smith", "suffix": "Jr."}, + classification="fix(#296)", + notes="two pre-comma pieces but ONE name piece: the positional " + "read peels the suffix first and would have left a lone " + "given and no family (the code review found 'Smith Jr., " + "Mr.' reading so), so the guard counts name pieces and " + "the family stays. v1 read first 'Smith', last 'Jr.', " + "suffix 'Dr.'; 2.0 through master family 'Smith', " + "suffix 'Jr.', title 'Mr.' for the Mr. spelling"), + Case("family_comma_suffixed_family_before_a_title_mr", "Smith Jr., Mr.", + {"title": "Mr.", "family": "Smith", "suffix": "Jr."}, + notes="unchanged at every baseline -- the shape the guard " + "protects"), + Case("family_comma_suffixed_two_word_family_before_a_title", + "John Smith Jr., Mr.", + {"title": "Mr.", "given": "John", "family": "Smith", + "suffix": "Jr."}, + classification="fix(#296)", + notes="two name pieces, so the split is kept"), + # -- the positional read keeps its ORDER, so the family-first fold + # (P1) reaches a particle-led pre-comma name as it does without + # the comma (the test review found it reading family 'de') + Case("family_comma_no_name_word_family_first", "de Mesnil Juan, Dr.", + {"title": "Dr.", "given": "Juan", "family": "de Mesnil"}, + policy=Policy(name_order=FAMILY_FIRST), + notes="as 'de Mesnil Juan' reads under the same order; master " + "read it through the suffix-comma route ('dr' was " + "suffix vocabulary) and got the fold that way"), + Case("family_comma_no_name_word_family_first_given_last", + "de la Cruz Juan Carlos, Dr.", + {"title": "Dr.", "given": "Carlos", "middle": "Juan", + "family": "de la Cruz"}, + policy=Policy(name_order=FAMILY_FIRST_GIVEN_LAST)), + Case("family_comma_no_name_word_family_first_plain", "John Smith, Dr.", + {"title": "Dr.", "given": "Smith", "family": "John"}, + policy=Policy(name_order=FAMILY_FIRST), + notes="the declared order applies to the pre-comma name as it " + "does to 'John Smith' alone -- deliberate"), + Case("title_word_trailing_is_not_a_title", "John Smith Prof.", + {"given": "John", "middle": "Smith", "family": "Prof."}, + notes="the pre-existing behavior the audit_dr_trailing and " + "audit_sra_trailing rows join, pinned so the pair reads " + "as consistency rather than as damage -- and so the " + "general fix (#316) has a row to flip when it lands. " + "Contrast 'Smith, Prof.', which the comma path DOES " + "route to title: the two paths disagree today"), + Case("ja_honorific_glued_family_comma_title_only", "田中さん, Dr.", + {"title": "Dr.", "family": "田中さん"}, + classification="fix(#296)", + notes="Accepted (C1): with 'dr' out of the suffix sets the " + "comma is a family comma, and the honorific peel runs " + "in script_segment on the other structures only, before " + "group or assign can say this comma fixed nothing -- so " + "the honorific stays glued, joining master's '田中さん, " + "Mr.'. Master peeled it through the suffix-comma route"), + Case("title_word_trailing_is_not_a_title_mr", "John Smith Mr.", + {"given": "John", "middle": "Smith", "family": "Mr."}), # -- #271: script-scoped order + segmentation (amendment 2026-07-27) Case("ko_unspaced_default", "김민준", @@ -2039,30 +2358,17 @@ def __post_init__(self) -> None: "extract_delimited claimed something passes the row " "above and fails only here"), Case("ja_honorific_glued_family_comma", "田中さん, PhD", - {"title": "PhD", "family": "田中", "suffix": "さん"}, + {"family": "田中", "suffix": "さん, PhD"}, classification="fix(#312)", notes="the peel reaches 田中さん across the comma, though no " "longer by crossing it: since #319 is_wholly_suffix " "declines the post-comma run outright (PhD is suffix " "vocabulary and it is the whole run), so the scan never " - "leaves segments[0] and never examines PhD at all. It " - "did cross before, stepping OVER PhD because that " - "spelling satisfies _is_post_nominal's strict test -- " - "the same fields by the older route. The spaced " - "田中さん PhD still peels that way, its single run " - "holding both tokens, so the two spellings now agree on " - "the outcome through DIFFERENT mechanisms; " - "ja_honorific_glued_family_comma_suffixy_second_run " - "cites this row for the outcome, not the route. Where " - "PhD itself lands still differs " - "between the two spellings -- suffix spaced, title " - "post-comma -- and that is fix(comma-family)'s, not " - "the peel's. The expectation bakes in TWO deviations " - "from 1.4.0 and only one of them is #312's: the peel " - "is, while first -> family is comma-family's, witnessed " - "on pure Latin by family_comma_lone_title (1.4.0 first " - "Smith, here family Smith) -- so no script-conditional " - "rule is reaching that half"), + "meets the comma. PhD reads as the postnominal it is " + "since #296's audit took 'phd' out of TITLES -- this row " + "carried title 'PhD' until then, which was the title " + "peel claiming a credential because v1's lists put it " + "where v1's parser needed it"), Case("ja_honorific_glued_family_comma_suffixy_second_run", "田中さん, V.", {"given": "V.", "family": "田中", "suffix": "さん"}, diff --git a/tests/v2/pipeline/test_assign.py b/tests/v2/pipeline/test_assign.py index 9eebb82b..e7a5b6f7 100644 --- a/tests/v2/pipeline/test_assign.py +++ b/tests/v2/pipeline/test_assign.py @@ -15,10 +15,10 @@ from nameparser._types import AmbiguityKind, Role _LEX = Lexicon( - titles=frozenset({"dr", "mr", "mrs", "sir"}), + titles=frozenset({"dr", "mr", "mrs", "sir", "sr"}), given_name_titles=frozenset({"sir"}), suffix_acronyms=frozenset({"phd", "md"}), - suffix_words=frozenset({"jr", "iii", "v"}), + suffix_words=frozenset({"jr", "iii", "v", "sr"}), particles=frozenset({"de", "la", "van"}), particles_ambiguous=frozenset({"van"}), conjunctions=frozenset({"and"}), @@ -288,3 +288,136 @@ def test_script_with_no_table_entry_falls_back() -> None: out = _assigned("毛 泽东", hangul_only) assert _by_role(out, Role.GIVEN) == "毛" assert _by_role(out, Role.FAMILY) == "泽东" + + +def test_all_title_post_comma_segment_leaves_segment_zero_positional() -> None: + # 'John Smith, Dr.' -- the comma is followed by nothing but a title, + # so it never said where the family name ends. Reading segment 0 + # wholly as family throws away a split the writer gave us. + out = _assigned("John Smith, Dr.") + assert _by_role(out, Role.TITLE) == "Dr." + assert _by_role(out, Role.GIVEN) == "John" + assert _by_role(out, Role.FAMILY) == "Smith" + + +def test_all_title_post_comma_segment_needs_two_pre_comma_pieces() -> None: + # 'Smith, Dr.' has nothing to split: one pre-comma piece stays FAMILY + # rather than becoming a lone GIVEN under the positional read. + out = _assigned("Smith, Dr.") + assert _by_role(out, Role.TITLE) == "Dr." + assert _by_role(out, Role.FAMILY) == "Smith" + assert _by_role(out, Role.GIVEN) == "" + + +def test_post_comma_title_run_is_all_titles() -> None: + out = _assigned("John Smith, Mr. Dr.") + assert _by_role(out, Role.TITLE) == "Mr. Dr." + assert _by_role(out, Role.GIVEN) == "John" + assert _by_role(out, Role.FAMILY) == "Smith" + + +def test_a_title_and_a_suffix_after_the_comma_fix_no_family_either() -> None: + # the condition is "no name word", not "all titles": a title and a + # postnominal with nothing between them said nothing about where + # the family ends (the design-docs review found C1 silent on it) + out = _assigned("John Smith, Mr. Jr.") + assert _by_role(out, Role.TITLE) == "Mr." + assert _by_role(out, Role.GIVEN) == "John" + assert _by_role(out, Role.FAMILY) == "Smith" + assert _by_role(out, Role.SUFFIX) == "Jr." + + +def test_the_positional_segment_zero_records_its_order() -> None: + # post_rules' family-first fold and its leading-piece scan key on + # "assign records no order after a family comma"; the positional + # read is the path that gives one (the test review found the fold + # missing 'de Mesnil Juan, Dr.' under a family-first order) + out = _assigned("John Smith, Dr.") + assert out.order is not None + out = _assigned("Smith, Dr. John") + assert out.order is None + + +def test_partly_title_post_comma_segment_keeps_family_comma() -> None: + # 'Smith, Dr. John' still has a name after the title, so the comma + # DID fix the family: segment 0 stays wholly family. + out = _assigned("Smith, Dr. John") + assert _by_role(out, Role.TITLE) == "Dr." + assert _by_role(out, Role.GIVEN) == "John" + assert _by_role(out, Role.FAMILY) == "Smith" + + +def test_non_title_post_comma_segment_is_untouched() -> None: + out = _assigned("John Smith, Jones") + assert _by_role(out, Role.FAMILY) == "John Smith" + assert _by_role(out, Role.GIVEN) == "Jones" + + +def test_positional_segment_zero_reports_the_particle_fork() -> None: + # The comma no longer fixed the family, so the leading ambiguous + # particle IS a live fork again -- emitted at the site that decides + # it, per the ambiguity doctrine. + out = _assigned("Van Johnson, Dr.") + assert _by_role(out, Role.GIVEN) == "Van" + assert _by_role(out, Role.FAMILY) == "Johnson" + assert [a.kind for a in out.ambiguities] == \ + [AmbiguityKind.PARTICLE_OR_GIVEN] + + +def test_a_credential_run_after_a_family_comma_reads_as_suffixes() -> None: + # 'Smith, Jr.' -- the peel's whole-segment exception claimed this + # even with 'jr' out of TITLES, because _is_leading_title also + # infers a title from the period-abbreviation shape. The slot after + # a family comma IS postnominal position, so a run that is nothing + # but suffix pieces is read as one before the peel gets a chance + # (#296) -- and the whole run, not the lone piece: 'Smith, Ph. D. + # Jr.' put the split credential in the given name (#325). + out = _assigned("Smith, Jr.") + assert _by_role(out, Role.FAMILY) == "Smith" + assert _by_role(out, Role.SUFFIX) == "Jr." + assert _by_role(out, Role.TITLE) == "" + out = _assigned("Smith, Ph. D. Jr.") + assert _by_role(out, Role.FAMILY) == "Smith" + assert _by_role(out, Role.SUFFIX) == "Ph. D. Jr." + assert _by_role(out, Role.GIVEN) == "" + out = _assigned("Smith, Jr. PhD") + assert _by_role(out, Role.SUFFIX) == "Jr. PhD" + assert _by_role(out, Role.TITLE) == "" + + +def test_lone_post_comma_dual_word_reads_as_the_postnominal() -> None: + # 'sr' kept BOTH memberships; position is what decides, and this is + # the position that decides postnominal. + out = _assigned("Smith, Sr.") + assert _by_role(out, Role.SUFFIX) == "Sr." + assert _by_role(out, Role.TITLE) == "" + + +def test_leading_dual_word_still_reads_as_the_title() -> None: + # The other half of the same fork, untouched: the peel's normal path. + out = _assigned("Sr. Garcia") + assert _by_role(out, Role.TITLE) == "Sr." + # assign leaves the one name word as the given; H1 (post_rules) + # makes it the family + assert _by_role(out, Role.GIVEN) == "Garcia" + + +def test_lone_post_comma_title_is_not_a_suffix() -> None: + # 'Smith, Dr.' decides WITHOUT consulting the ordering: after the + # audit 'dr' is not suffix-tagged, so the suffix test simply declines + # and the title peel takes it as before. + out = _assigned("Smith, Dr.") + assert _by_role(out, Role.TITLE) == "Dr." + assert _by_role(out, Role.SUFFIX) == "" + + +def test_a_mixed_post_comma_run_keeps_the_walk_order() -> None: + # a title then a suffix word is read where each stands (v1's walk: + # leading titles peel, the rest is given / middle / suffix), and a + # name word anywhere in the run makes it a name, not a credential run + out = _assigned("Smith, Dr. Jr.") + assert _by_role(out, Role.TITLE) == "Dr." + assert _by_role(out, Role.SUFFIX) == "Jr." + out = _assigned("Smith, John Jr.") + assert _by_role(out, Role.GIVEN) == "John" + assert _by_role(out, Role.SUFFIX) == "Jr." diff --git a/tests/v2/pipeline/test_post_rules.py b/tests/v2/pipeline/test_post_rules.py index 8678ddf9..2273d591 100644 --- a/tests/v2/pipeline/test_post_rules.py +++ b/tests/v2/pipeline/test_post_rules.py @@ -24,7 +24,7 @@ given_name_titles=frozenset({"sir"}), particles=frozenset({"de", "der", "ibn", "la", "van"}), particles_ambiguous=frozenset({"la", "van"}), - suffix_words=frozenset({"dr"}), + suffix_words=frozenset({"jr"}), suffix_acronyms=frozenset({"md"}), conjunctions=frozenset({"y"}), bound_given_names=frozenset({"abdul"}), diff --git a/tests/v2/test_facade_cases.py b/tests/v2/test_facade_cases.py index 1174bfce..2e51fdf9 100644 --- a/tests/v2/test_facade_cases.py +++ b/tests/v2/test_facade_cases.py @@ -71,6 +71,11 @@ #: to the unsubtracted default pushes every parenthesis-maiden row #: into this set and turns nothing red. _CORE_ONLY_IDS = frozenset({ + # #296's bundle: the positional read of the pre-comma name under a + # declared family-first order -- v1 has no name order to declare + "family_comma_no_name_word_family_first", + "family_comma_no_name_word_family_first_given_last", + "family_comma_no_name_word_family_first_plain", "leading_never_given_particle_two_leftovers_family_first", "leading_never_given_particle_two_leftovers_family_first_given_last", "maiden_marker_delimited_beside_a_nickname_clause", diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 62980cad..f92232ff 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -564,8 +564,6 @@ def test_cjk_corpus_matches_the_case_table() -> None: # The literal-anchored rules #413 added. Each claims exactly one # corpus name, so _CORPUS_CLAIMS cannot see a widening that reaches # only names the corpora lack -- these probes are the only wall. - "fix(#367) a title-and-particle word stops the scan instead of being chained": - ("Dr. Do Smith", "Dr Do Nguyen"), "fix(#399) a maiden marker bounds the particle chain: the geb. spelling": ("Berg, Ursula von der geb. Albrecht",), "fix(credential-pair-order) a split credential and a suffix render in written order": @@ -881,13 +879,15 @@ class _LatinCopy(NamedTuple): # snapshot is agreement rather than duplication: a divergence # would be a fact about one of the rules. # The title half of the #367 rule, widened when three more titled - # names arrived (#413). Partial on purpose: these are the four - # spellings that appear before `van` in the corpora, not every title - # that could -- seven titles sit before some particle, and two of the - # other three have rules of their own. + # names arrived (#413). Partial on purpose: these are the spellings + # that appear before `van` in the corpora, not every title that + # could -- the other titles before a particle in the corpora either + # have rules of their own or do not diff. 'jr' was a member until #296's audit took + # it out of TITLES (a postnominal only); 'Jr. Van Johnson' reads the + # same by the period-abbreviation inference and has its literal. "fix(#367) a title no longer displaces a leading particle out of the leading position": _LatinCopy(vocabulary=TITLES, - covers=frozenset({"mr", "dr", "sir", "jr"})), + covers=frozenset({"mr", "dr", "sir"})), "fix(#399)": _LatinCopy( vocabulary=PARTICLES, covers=frozenset({"de", "del", "den", "der", "di", "do", "dos", @@ -1240,7 +1240,7 @@ def _claim(rule: dict) -> _Claim: _CORPUS_CLAIMS: dict[str, dict[str, _Claim]] = { "expected_since_1.4.0.toml": { "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": - _Claim(107, ('family', 'given', 'middle'), "0191ac9143a6"), + _Claim(108, ('family', 'given', 'middle'), "9a814f70c2dc"), "fix(#274) maiden markers consumed": _Claim(29, ('family', 'maiden', 'middle'), "c0981c1c6557"), "fix(cjk-maiden-marker) maiden marker consumed, compounding with the CJK order flip": @@ -1248,11 +1248,29 @@ def _claim(rule: dict) -> _Claim: "fix(#379) a tussenvoegsel after a family comma attaches to the family": _Claim(13, ('family', 'middle'), "973617235cda"), "fix(comma-family) lone post-comma piece routes to suffix/title, not first": - _Claim(264, ('given', 'suffix', 'title'), "a639a0ca7ce2"), + _Claim(274, ('given', 'suffix', 'title'), "dc19a4822c61"), + "fix(comma-family) a comma followed only by titles keeps the given/family split": + _Claim(2, ('family', 'given'), "5bd9c6d96c38"), + "fix(comma-family) a comma followed only by titles keeps the given/family split, the C1 example": + _Claim(2, ('family', 'given', 'suffix'), "a3cfff4e78f4"), + "fix(#296) a dropped prenominal takes the name position it occupies": + _Claim(3, ('given', 'middle', 'title'), "263d5957cfc1"), + "fix(#296) dr is not postnominal vocabulary, so a trailing Dr. is a name word": + _Claim(9, ('family', 'middle', 'suffix'), "8d6e9a1b43c0"), + "fix(#296) a credential-only comma string reads a name and its postnominal": + _Claim(2, ('family', 'given', 'suffix', 'title'), "3f983ff71dee"), + "fix(#296) a lone post-comma credential is a suffix": + _Claim(18, ('family', 'given', 'suffix', 'title'), "1f79efa10444"), + "fix(#325) a split credential followed by another suffix after a one-word family comma reads as suffixes": + _Claim(6, ('given', 'suffix', 'title'), "7911e0158337"), + "fix(#325) a credential run across a second comma reads as suffixes": + _Claim(1, ('suffix', 'title'), "f025c5f70a4e"), + "fix(#367) an inferred title no longer displaces a leading particle either": + _Claim(1, ('family', 'given', 'middle'), "d8ee9cd5da5f"), "fix(comma-precomma-family) pre-comma run reads as family, not given": - _Claim(264, ('family', 'given'), "a639a0ca7ce2"), + _Claim(274, ('family', 'given'), "dc19a4822c61"), "fix(suffix-routing) two-token name with unambiguous trailing suffix stays suffix": - _Claim(1056, ('family', 'given', 'suffix'), "bfb2e274a968"), + _Claim(1068, ('family', 'given', 'suffix'), "c0770ee57f2a"), "fix(suffix-delimiter-rendering) no-space delimiter core token kept whole": _Claim(0, ('suffix',), "e3b0c44298fc"), "ambiguous-surname-acronym data change: parenthesized (MA)/(DO) now stays nickname": @@ -1266,11 +1284,11 @@ def _claim(rule: dict) -> _Claim: "fix(cjk-fullwidth-paren-nickname) fullwidth-parenthesis recognition compounds with the CJK order flip": _Claim(1, ('family', 'given', 'middle', 'nickname'), "cf370e856ae7"), "fix(cjk-comma-honorific-peel) glued honorific peels off a post-comma given name": - _Claim(22, ('given', 'suffix'), "1273a0e4e949"), + _Claim(23, ('given', 'suffix'), "344de804e2c6"), "fix(cjk-comma-compound) comma routing compounds with the CJK order flip": - _Claim(22, ('family', 'given', 'middle', 'suffix', 'title'), "1273a0e4e949"), + _Claim(23, ('family', 'given', 'middle', 'suffix', 'title'), "344de804e2c6"), "fix(cjk-glued-honorific-peel) glued honorific peels into suffix": - _Claim(36, ('family', 'given', 'suffix'), "9ba2bdd624de"), + _Claim(37, ('family', 'given', 'suffix'), "719c31233502"), "fix(cjk-honorific-suffix) postnominal honorifics recognized, compounding with the CJK order flip": _Claim(19, ('family', 'given', 'middle', 'suffix'), "aa475ddd4745"), "feat(#269) non-Latin titles/conjunctions recognized": @@ -1280,7 +1298,7 @@ def _claim(rule: dict) -> _Claim: "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle": _Claim(1, ('family', 'given'), "ca7b37af6cf8"), "fix(#367) a title no longer displaces a leading particle out of the leading position": - _Claim(4, ('family', 'given', 'middle'), "ae299117dd60"), + _Claim(3, ('family', 'given', 'middle'), "724967a4a117"), "fix(#400) abd joins the word after it as one given name": _Claim(11, ('given', 'middle'), "1eaed91fc574"), "fix(#272/#308) nakaguro division and a glued hangul honorific in one name": @@ -1328,9 +1346,9 @@ def _claim(rule: dict) -> _Claim: "fix(#379) a tussenvoegsel after a family comma attaches to the family": _Claim(13, ('family', 'middle'), "973617235cda"), "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": - _Claim(107, ('_ambiguities', 'family', 'given', 'middle'), "0191ac9143a6"), + _Claim(108, ('_ambiguities', 'family', 'given', 'middle'), "9a814f70c2dc"), "fix(#308/#312/#319/#320) glued CJK honorific peeled off the name into suffix": - _Claim(36, ('family', 'given', 'suffix'), "9ba2bdd624de"), + _Claim(37, ('family', 'given', 'suffix'), "719c31233502"), "fix(#307/#308/#320) spaced CJK postnominal honorific routed to suffix": _Claim(16, ('family', 'given', 'middle', 'suffix'), "6d390e518bd2"), "fix(#309) 旧姓 maiden marker consumed, compounding with the CJK order flip": @@ -1342,7 +1360,7 @@ def _claim(rule: dict) -> _Claim: "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle": _Claim(1, ('_ambiguities', 'family', 'given'), "ca7b37af6cf8"), "fix(#367) a title no longer displaces a leading particle out of the leading position": - _Claim(4, ('family', 'given', 'middle'), "ae299117dd60"), + _Claim(3, ('family', 'given', 'middle'), "724967a4a117"), "fix(#380) a trailing vd after a family comma is the tussenvoegsel, not a post-nominal": _Claim(2, ('family', 'suffix'), "ec0d45289dc1"), "fix(#399) a maiden marker bounds the particle chain that swallowed it": @@ -1363,8 +1381,6 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('family', 'given', 'middle'), "7923e6d3c5a7"), "fix(#418) accepted: a suffix word inside the maiden name ends it, connective or not": _Claim(1, ('family', 'maiden', 'middle'), "bedc18423d2a"), - "fix(#367) a title-and-particle word stops the scan instead of being chained": - _Claim(1, ('given', 'title'), "faa2c70fc49e"), "fix(#369) a given-name title licenses the bound given-name join with one word to spare": _Claim(3, ('family', 'given'), "724be3e6b926"), "fix(#401) the bound-given reserve counts the trailing numeral assign reads as the suffix": @@ -1381,6 +1397,32 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('_ambiguities', 'family', 'suffix'), "3e3aae6a5b4b"), "fix(#424) a title-led chain before the numeral is the one name piece": _Claim(1, ('_ambiguities', 'family', 'given', 'suffix'), "5b3a743f9e35"), + "fix(comma-family) a comma followed only by titles keeps the given/family split": + _Claim(2, ('family', 'given'), "5bd9c6d96c38"), + "fix(comma-family) a comma followed only by titles keeps the given/family split, the C1 example": + _Claim(2, ('family', 'given'), "a3cfff4e78f4"), + "fix(#296) a dropped prenominal takes the name position it occupies": + _Claim(3, ('_ambiguities', 'given', 'middle', 'title'), "263d5957cfc1"), + "fix(#296) dr is not postnominal vocabulary, so a trailing Dr. is a name word": + _Claim(9, ('family', 'middle', 'suffix'), "8d6e9a1b43c0"), + "fix(#296) a credential-only comma string reads a name and its postnominal": + _Claim(2, ('family', 'given', 'suffix', 'title'), "3f983ff71dee"), + "fix(#296) a lone post-comma credential is a suffix": + _Claim(18, ('family', 'given', 'suffix', 'title'), "1f79efa10444"), + "fix(#325) a split credential followed by another suffix after a one-word family comma reads as suffixes": + _Claim(6, ('given', 'suffix', 'title'), "7911e0158337"), + "fix(#325) a credential run across a second comma reads as suffixes": + _Claim(1, ('suffix', 'title'), "f025c5f70a4e"), + "fix(#296) a glued honorific before a lone credential: the credential is the postnominal": + _Claim(1, ('family', 'suffix', 'title'), "01bf2bd3f895"), + "fix(#296) do is a name, so it no longer stops the leading-particle scan as a title": + _Claim(1, ('family', 'given'), "faa2c70fc49e"), + "fix(#296) dr is not postnominal vocabulary, so 'John Smith, Dr.' keeps its split and its title": + _Claim(2, ('_ambiguities', 'suffix', 'title'), "34d3d96adb65"), + "fix(#296) an ambiguous acronym counts as a suffix only when written with its periods": + _Claim(1, ('_ambiguities', 'family', 'suffix'), "e13b3c769de4"), + "fix(#367) an inferred title no longer displaces a leading particle either": + _Claim(1, ('family', 'given', 'middle'), "d8ee9cd5da5f"), "fix(#424) accepted: a particle of the suffix vocabulary opening the trailing run is a suffix piece": _Claim(1, ('_ambiguities', 'family', 'middle', 'suffix'), "a564b97f7162"), "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle, the P4 example": @@ -1402,7 +1444,7 @@ def _claim(rule: dict) -> _Claim: "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle": _Claim(1, ('_ambiguities', 'family', 'given'), "ca7b37af6cf8"), "fix(#367) a title no longer displaces a leading particle out of the leading position": - _Claim(4, ('family', 'given', 'middle'), "ae299117dd60"), + _Claim(3, ('family', 'given', 'middle'), "724967a4a117"), "fix(#380) a trailing vd after a family comma is the tussenvoegsel, not a post-nominal": _Claim(2, ('family', 'suffix'), "ec0d45289dc1"), "fix(#399) a maiden marker bounds the particle chain that swallowed it": @@ -1421,8 +1463,6 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('family', 'given', 'middle'), "7923e6d3c5a7"), "fix(#418) accepted: a suffix word inside the maiden name ends it, connective or not": _Claim(1, ('family', 'maiden', 'middle'), "bedc18423d2a"), - "fix(#367) a title-and-particle word stops the scan instead of being chained": - _Claim(1, ('given', 'title'), "faa2c70fc49e"), "fix(#369) a given-name title licenses the bound given-name join with one word to spare": _Claim(3, ('family', 'given'), "724be3e6b926"), "fix(#401) the bound-given reserve counts the trailing numeral assign reads as the suffix": @@ -1439,6 +1479,32 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('_ambiguities', 'family', 'suffix'), "3e3aae6a5b4b"), "fix(#424) a title-led chain before the numeral is the one name piece": _Claim(1, ('_ambiguities', 'family', 'given', 'suffix'), "5b3a743f9e35"), + "fix(comma-family) a comma followed only by titles keeps the given/family split": + _Claim(2, ('family', 'given'), "5bd9c6d96c38"), + "fix(comma-family) a comma followed only by titles keeps the given/family split, the C1 example": + _Claim(2, ('family', 'given'), "a3cfff4e78f4"), + "fix(#296) a dropped prenominal takes the name position it occupies": + _Claim(3, ('_ambiguities', 'given', 'middle', 'title'), "263d5957cfc1"), + "fix(#296) dr is not postnominal vocabulary, so a trailing Dr. is a name word": + _Claim(9, ('family', 'middle', 'suffix'), "8d6e9a1b43c0"), + "fix(#296) a credential-only comma string reads a name and its postnominal": + _Claim(2, ('family', 'given', 'suffix', 'title'), "3f983ff71dee"), + "fix(#296) a lone post-comma credential is a suffix": + _Claim(18, ('family', 'given', 'suffix', 'title'), "1f79efa10444"), + "fix(#325) a split credential followed by another suffix after a one-word family comma reads as suffixes": + _Claim(6, ('given', 'suffix', 'title'), "7911e0158337"), + "fix(#325) a credential run across a second comma reads as suffixes": + _Claim(1, ('suffix', 'title'), "f025c5f70a4e"), + "fix(#296) a glued honorific before a lone credential: the credential is the postnominal": + _Claim(1, ('family', 'suffix', 'title'), "01bf2bd3f895"), + "fix(#296) do is a name, so it no longer stops the leading-particle scan as a title": + _Claim(1, ('family', 'given'), "faa2c70fc49e"), + "fix(#296) dr is not postnominal vocabulary, so 'John Smith, Dr.' keeps its split and its title": + _Claim(2, ('_ambiguities', 'suffix', 'title'), "34d3d96adb65"), + "fix(#296) an ambiguous acronym counts as a suffix only when written with its periods": + _Claim(1, ('_ambiguities', 'family', 'suffix'), "e13b3c769de4"), + "fix(#367) an inferred title no longer displaces a leading particle either": + _Claim(1, ('family', 'given', 'middle'), "d8ee9cd5da5f"), "fix(#424) accepted: a particle of the suffix vocabulary opening the trailing run is a suffix piece": _Claim(1, ('_ambiguities', 'family', 'middle', 'suffix'), "a564b97f7162"), "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle, the P4 example": @@ -1574,7 +1640,8 @@ def test_every_rule_claims_the_recorded_share_of_the_corpus() -> None: # exclude them and they must stay on the compound rule ("Dr 김민준씨, V.", ("family", "given", "suffix")): "fix(cjk-comma-compound)", - ("田中さん, PhD", ("family", "given", "suffix")): + # since #296's audit the title moves too (PhD is the postnominal) + ("田中さん, PhD", ("family", "given", "suffix", "title")): "fix(cjk-comma-compound)", ("田中さん, V.", ("family", "suffix")): "fix(cjk-comma-compound)", # #372's suffix-routing split. 'Bob Jones, author' moves NO @@ -1583,9 +1650,19 @@ def test_every_rule_claims_the_recorded_share_of_the_corpus() -> None: # glued honorifics are the majority it also legitimately takes, # and cannot be given a rule of their own -- '김민준씨' and the # given name '김지양' are the same string shape. + # since the all-titles repair (#296's bundle) the pre-comma name + # keeps its split here, and the rule written for that shape is + # ahead of the precomma merge in the file; 'MD, PHD' has one + # pre-comma piece, no split to keep, and stays merged ("Bob Jones, author", ("family", "given")): - "fix(comma-precomma-family)", - ("MD, PHD", ("family", "given")): "fix(comma-precomma-family)", + "fix(comma-family) a comma followed only by titles keeps " + "the given/family split", + # since #296's audit 'PHD' is a postnominal here and the string + # is credentials only; the rule written for that shape is ahead + # of the precomma merge in the file + ("MD, PHD", ("family", "given", "suffix", "title")): + "fix(#296) a credential-only comma string reads a name and " + "its postnominal", ("Smith Jr.", ("family", "suffix")): "fix(suffix-routing)", # the glued/spaced boundary. 'Andersonさん' and '김민준씨' left # suffix-routing for a rule that names them; '김민준 씨.' is diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 66117a9f..3868539a 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -890,15 +890,20 @@ def test_revise_clears_a_stale_unjoined_mark() -> None: def test_revise_sets_a_missing_unjoined_mark() -> None: # The other direction, and the one that made rules.md#R2's - # invariant false through this path: "Do" alone parses as a TITLE, - # so the sub-parse marks nothing, and the harvest then re-roles a - # bare particle into FAMILY. The recompute marks it there, so a - # non-empty family still has a non-empty base. + # invariant false through this path: "St" alone parses as a TITLE + # (a word in both the title and particle vocabularies), so the + # sub-parse marks nothing, and the harvest then re-roles a bare + # particle into FAMILY. The recompute marks it there, so a + # non-empty family still has a non-empty base. Spelled with "Do" + # until #296's audit took 'do' out of TITLES; "Do" alone is a + # marked given name now, which is the other path (kept below). p = Parser() - revised = p.revise(p.parse("Juan de la Vega"), family="Do") - assert revised.family == "Do" - assert revised.family_base == "Do" + revised = p.revise(p.parse("Juan de la Vega"), family="St") + assert revised.family == "St" + assert revised.family_base == "St" assert revised.family_particles == "" + revised = p.revise(p.parse("Juan de la Vega"), family="Do") + assert (revised.family, revised.family_base) == ("Do", "Do") def test_revise_sub_parse_structural_behavior() -> None: diff --git a/tests/v2/test_regex_sync.py b/tests/v2/test_regex_sync.py index f547a717..cf8d8b01 100644 --- a/tests/v2/test_regex_sync.py +++ b/tests/v2/test_regex_sync.py @@ -101,6 +101,7 @@ def test_initial_copies_agree_with_each_other_and_config() -> None: _SOURCES: dict[tuple[str, str], str | None] = { ("_group", "_PERIOD_ABBREV"): "period_abbreviation", ("_group", "_D"): None, + ("_vocab", "_DOTTED"): None, ("_group", "_PH"): None, ("_vocab", "_ROMAN"): "roman_numeral", ("_post_rules", "_EAST_SLAVIC"): "east_slavic_patronymic", diff --git a/tools/differential/corpus_cjk.jsonl b/tools/differential/corpus_cjk.jsonl index 34f198e9..55f8a00f 100644 --- a/tools/differential/corpus_cjk.jsonl +++ b/tools/differential/corpus_cjk.jsonl @@ -47,6 +47,7 @@ "田中さん II" "田中さん V." "田中さん 様." +"田中さん, Dr." "田中さん, Ph. D." "田中さん, PhD" "田中さん, V." diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index ea50a80a..dc26fc19 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -38,6 +38,7 @@ "J. Smith" "J. née Jones Smith V" "J.R. Smith" +"Jack Ma." "Jack Wei Ma" "Jane Smith (Nee)" "Jane Smith (Nee) (Jones)" @@ -66,9 +67,12 @@ "John Smith Mc V" "John Smith PhD" "John Smith Q.W.E.R.T." +"John Smith, Jones" "John Smith, LEED AP" "John Smith, MD, Bart" "John Smith, MD,, Jr." +"John Smith, Mr." +"John Smith, Mr. Jr." "John Smith, PhD" "John Smith, V." "John née Jones Smith Ma" @@ -124,10 +128,19 @@ "Sir abdul van der Berg" "Sir de Mesnil" "Smith Jr." +"Smith Jr., Mr." "Smith, Abd" +"Smith, Dr." +"Smith, Dr. Jr." +"Smith, Esq." "Smith, John" +"Smith, Jr." "Smith, Major. John" +"Smith, Ms." +"Smith, Ms. Jane" +"Smith, Ph. D. Jr." "Smith, PhD" +"Smith, Sr." "Smith, de Mesnil Juan" "Van Johnson" "Vega, Juan de la" @@ -180,6 +193,8 @@ "王君" "田中、太郎" "田中さん" +"田中さん, Dr." +"田中さん, PhD" "田中さん, V." "马丁·路德·金씨" "高橋 みなみ" diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index c5533985..c37cc5b2 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -163,6 +163,95 @@ issue = "fix(credential-pair-order) a split credential and a suffix render in wr name_regex = "(?i)^[\\u0000-\\u024f]*,\\s*jr\\.?\\s+ph\\.?\\s*d\\.?\\s*$" fields = ["suffix"] +[[change]] +issue = "fix(comma-family) a comma followed only by titles keeps the given/family split" +# 'Bob Jones, author' / 'Bob Jones, compositeur': the comma is followed +# by nothing but a title word, which said nothing about where the +# family name ends, so the pre-comma name keeps its positional read -- +# first 'Bob Jones' -> given 'Bob', family 'Jones', the title staying +# where it was (the post-comma word is a title on both sides). 2.0 had +# merged the pre-comma run into the family (the comma-precomma-family +# rule below, whose fields would absorb this move -- written ahead of +# it so the name files here and that rule's count tells). +name_regex = "(?i)^bob\\s+jones,\\s*[a-z]+$" +fields = ["given", "family"] + +[[change]] +issue = "fix(comma-family) a comma followed only by titles keeps the given/family split, the C1 example" +# 'John Smith, Mr.' and 'John Smith, Mr. Jr.': the rules.md#C1 examples +# of the rule above (first 'John Smith' -> given 'John', family +# 'Smith'; the title stays where it was; with the postnominal, v1's +# first 'Jr.' is the suffix). Without this rule the second lands on +# the fields-only fix(suffix-routing) catch-all below. Its own literal rather than +# an alternation with 'Bob Jones', which would be no vocabulary. +name_regex = "(?i)^john\\s+smith,\\s*mr\\.?(\\s+jr\\.?)?$" +fields = ["given", "family", "suffix"] + +[[change]] +issue = "fix(#296) a dropped prenominal takes the name position it occupies" +# 'Do Quang Minh': 'do' left TITLES -- a postnominal (D.O.) and a +# Vietnamese surname, never a prenominal -- so a leading 'Do' is no +# longer peeled as a title and falls through to the positional read: +# title 'Do', first 'Quang' -> given 'Do', middle 'Quang'. The +# 2026-07-30 audit table's disposition; 'phd', 'jr', 'junior' and +# 'se' left with it, and no corpus name leads with a bare one (the +# three that lead with 'Jr.' are read by the period inference and do +# not move). The rule's second name is 'MD, DO, DDS': title 'DO' -> +# given 'DO', the comma anchor's one corpus hit. `family` +# is deliberately OUT of the field list: the family name must not +# move on this rule, so a regression there stays loud. The anchors +# are the two openings a name has (start, or after a family comma). +name_regex = "(?i)(^|,\\s*)do\\b" +fields = ["title", "given", "middle"] + +[[change]] +issue = "fix(#296) dr is not postnominal vocabulary, so a trailing Dr. is a name word" +# 'John Smith Dr.' and 'dr Vincent van Gogh dr' with its #100 siblings +# (and, at 2.x only, 'Smith Dr', which 1.4.0 already read as the +# family): 'dr' left SUFFIX_WORDS, where it was v1 +# residue -- 'Dr.' is not a postnominal in any tradition -- so a +# trailing bare 'Dr' is no longer suffix vocabulary and falls to the +# positional read, taking the family name with it. NOT a new class: +# no trailing title-only word routes to title on the no-comma path +# ('John Smith Prof.' reads family 'Prof.' at every baseline), and +# the suffix entry was the only thing making 'dr' the exception. #316 +# is the open question for the class. +name_regex = "(?i)\\sdr\\.?$" +fields = ["middle", "family", "suffix"] + +[[change]] +issue = "fix(#296) a credential-only comma string reads a name and its postnominal" +# 'Jr., PhD' / 'MD, PHD': a string that is nothing BUT credentials. +# v1 read the trailing one as a title and the leading one as a first +# name; with 'phd' out of TITLES the comma structure reads it as a +# one-word name plus a postnominal -- a reading of a string nobody +# writes as a name, and the honest one for it. +name_regex = "(?i)^[a-z]{2,3}\\.?,\\s*phd$" +fields = ["title", "given", "family", "suffix"] + +[[change]] +issue = "fix(#325) a split credential followed by another suffix after a one-word family comma reads as suffixes" +# 'Smith, Ph. D. Jr.' and the #325 rows it leads ('Smith, Ph. D. MD', +# 'Smith, Ph. D. III', 'Smith, Ph. D. Jr. MD', 'Smith, Ph.D. Jr.', and +# 'Smith, PhD Jr.', which the regex reaches too -- title 'PhD' at +# every baseline, suffix 'PhD, Jr.' now): +# with one word before the comma the name is the listing form, and +# the space-split 'Ph. D.' followed by another suffix was no longer +# the LONE post-comma piece, so it fell through to the given name -- +# a 1.4.0 regression (v1 read suffix 'Ph. D.', title 'Jr.'). The +# whole run is suffixes now. Literal rather than the shape: a shape +# regex absorbed 'Smith, Dr. Jr.', which is the comma-family rule's +# (the comment review). +name_regex = "(?i)^smith,\\s*ph\\.?\\s?d\\.?\\s" +fields = ["title", "given", "suffix"] + +[[change]] +issue = "fix(#325) a credential run across a second comma reads as suffixes" +# 'Smith, Jr., PhD': title 'Jr.', suffix 'PhD' -> suffix 'Jr., PhD'; +# segments 2+ compose with the run. +name_regex = "(?i)^smith,\\s*jr\\.?,\\s*phd$" +fields = ["title", "suffix"] + [[change]] issue = "fix(comma-family) lone post-comma piece routes to suffix/title, not first" # 'Smith, Dr.' / 'Andrews, M.D.': v1 put the lone strict-suffix-or-title @@ -228,6 +317,27 @@ issue = "fix(comma-precomma-family) pre-comma run reads as family, not given" name_regex = "^[\\u0000-\\u024f]*,[\\u0000-\\u024f]*$" fields = ["given", "family"] +[[change]] +issue = "fix(#296) a lone post-comma credential is a suffix" +# 'Smith, Jr.' / 'Kim, Jr.' / 'Smith, PhD': rules.md#C1 -- the listing +# form holds with one word before the comma, and the word after it is +# the credential run. v1 put the family in `first` and the credential +# in `title`; 2.0 routed the pre-comma word to the family (the +# comma-family rule below) but the title peel still claimed the +# credential -- 'jr' and 'phd' were title vocabulary, and the +# period-abbreviation inference reads 'Jr.' as a title regardless -- +# so the postnominal position is read first now: a post-comma run +# that is nothing but suffix pieces is suffixes. A one-word name, a +# comma, one short word: the reach is pinned by the guards, and the +# rule sits BEHIND the two comma-family rules above so that a name +# whose only move is the pre-comma word going given -> family +# ('Smith, Dr.', 'Smith, Prof.') or a lone piece going given -> +# suffix ('Smith, Abd') keeps the rule written for it (the comment +# review found this rule absorbing all three). 'Smith, Ms.' is +# here too: the dual's postnominal reading, title at 1.4.0. +name_regex = "(?i)^[a-z]+,\\s*[a-z]{2,6}\\.?$" +fields = ["title", "given", "family", "suffix"] + [[change]] issue = "fix(#400/#274) bound-given join and maiden consumption in one name" # 'abd Allah Smith nee Jones': two behaviours meeting in one name, so @@ -1008,7 +1118,19 @@ issue = "fix(#367) a title no longer displaces a leading particle out of the lea # absorbing a leading-particle change, which is the mis-classification # the README warns about. A name_regex rule outranks every fields-only # one, so this claims it back. -name_regex = "(?i)^(mr|dr|sir|jr)\\.?\\s+van\\b" +name_regex = "(?i)^(mr|dr|sir)\\.?\\s+van\\b" +fields = ["given", "middle", "family"] + +[[change]] +issue = "fix(#367) an inferred title no longer displaces a leading particle either" +# 'Jr. Van Johnson': the same change as the rule above, for a leading +# piece the period-abbreviation inference reads as a title (H2). It +# sat in that rule's alternation as 'jr' while 'jr' was title +# vocabulary; #296's audit took it out (a postnominal only), so the +# member would now reach a word the vocabulary does not ship, and the +# shape gets its literal. The reading is unchanged: 'Jr.' is still a +# title by shape, and Van is still the leading name piece. +name_regex = "(?i)^jr\\.\\s+van\\b" fields = ["given", "middle", "family"] # Shapes that must never be explained. A [[change]] rule says "this diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index 01c323f4..49e9b3c3 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -346,17 +346,6 @@ issue = "fix(#272/#308) nakaguro division and a glued hangul honorific in one na name_regex = "^\\S*\u00b7\\S*\uc528$" fields = ["given", "middle", "family", "suffix"] -[[change]] -issue = "fix(#367) a title-and-particle word stops the scan instead of being chained" -# 'Dr. Do Van Johnson, MD': the arrival the fix(#367) rule below named -# as its reason for listing three roles -- "a corpus name that moves -# `title` or `maiden` should arrive UNEXPLAINED and be read once". This -# is the `title` one. `do` is both a title and an ambiguous particle, -# so it stops the transparency scan and stays a title piece instead of -# being chained onto the name. -name_regex = "(?i)^dr\\.?\\s+do\\s+van\\b" -fields = ["title", "given"] - [[change]] issue = "fix(#369) a given-name title licenses the bound given-name join with one word to spare" # 'Sheik abdul salam': rules.md#P5 -- "or after a given-name title, @@ -453,6 +442,151 @@ issue = "fix(#424) accepted: a particle of the suffix vocabulary opening the tra name_regex = "(?i)^john\\s+smith\\s+mc\\s+v$" fields = ["middle", "family", "suffix", "_ambiguities"] +[[change]] +issue = "fix(comma-family) a comma followed only by titles keeps the given/family split" +# 'Bob Jones, author' / 'Bob Jones, compositeur': the comma is followed +# by nothing but a title word, which said nothing about where the +# family name ends, so the pre-comma name keeps its positional read -- +# family 'Bob Jones' -> given 'Bob', family 'Jones', the title staying +# where it was. The FAMILY_COMMA rule that segment 0 is wholly the +# family rests on the writer having marked where it ends; a comma +# followed only by titles marked nothing ('John Smith, Mr.' is 'Mr. +# John Smith' with the honorific moved). +name_regex = "(?i)^bob\\s+jones,\\s*[a-z]+$" +fields = ["given", "family"] + +[[change]] +issue = "fix(comma-family) a comma followed only by titles keeps the given/family split, the C1 example" +# 'John Smith, Mr.' and 'John Smith, Mr. Jr.': the rules.md#C1 examples +# of the rule above (family 'John Smith' -> given 'John', family +# 'Smith'; title and suffix stay where they were). Its own literal rather than +# an alternation with 'Bob Jones', which would be no vocabulary. +name_regex = "(?i)^john\\s+smith,\\s*mr\\.?(\\s+jr\\.?)?$" +fields = ["given", "family"] + +[[change]] +issue = "fix(#296) a dropped prenominal takes the name position it occupies" +# 'Do Quang Minh': 'do' left TITLES -- a postnominal (D.O.) and a +# Vietnamese surname, never a prenominal -- so a leading 'Do' is no +# longer peeled as a title and falls through to the positional read: +# title 'Do', given 'Quang' -> given 'Do', middle 'Quang'. The +# 2026-07-30 audit table's disposition; 'phd', 'jr', 'junior' and +# 'se' left with it, and no corpus name leads with a bare one (the +# three that lead with 'Jr.' are read by the period inference and do +# not move). The rule's second name is 'MD, DO, DDS': title 'DO' -> +# given 'DO', the comma anchor's one corpus hit. `family` +# is deliberately OUT of the field list: the family name must not +# move on this rule, so a regression there stays loud. The anchors +# are the two openings a name has (start, or after a family comma). +name_regex = "(?i)(^|,\\s*)do\\b" +fields = ["title", "given", "middle", "_ambiguities"] + +[[change]] +issue = "fix(#296) dr is not postnominal vocabulary, so a trailing Dr. is a name word" +# 'John Smith Dr.' and 'dr Vincent van Gogh dr' with its #100 siblings +# (and, at 2.x only, 'Smith Dr', which 1.4.0 already read as the +# family): 'dr' left SUFFIX_WORDS, where it was v1 +# residue -- 'Dr.' is not a postnominal in any tradition -- so a +# trailing bare 'Dr' is no longer suffix vocabulary and falls to the +# positional read, taking the family name with it. NOT a new class: +# no trailing title-only word routes to title on the no-comma path +# ('John Smith Prof.' reads family 'Prof.' at every baseline), and +# the suffix entry was the only thing making 'dr' the exception. #316 +# is the open question for the class. +name_regex = "(?i)\\sdr\\.?$" +fields = ["middle", "family", "suffix"] + +[[change]] +issue = "fix(#296) a credential-only comma string reads a name and its postnominal" +# 'Jr., PhD' / 'MD, PHD': a string that is nothing BUT credentials. +# v1 read the trailing one as a title and the leading one as a first +# name; with 'phd' out of TITLES the comma structure reads it as a +# one-word name plus a postnominal -- a reading of a string nobody +# writes as a name, and the honest one for it. +name_regex = "(?i)^[a-z]{2,3}\\.?,\\s*phd$" +fields = ["title", "given", "family", "suffix"] + +[[change]] +issue = "fix(#296) a lone post-comma credential is a suffix" +# 'Smith, Jr.' / 'Kim, Jr.' / 'Smith, PhD': rules.md#C1 -- the listing +# form holds with one word before the comma, and the word after it is +# the credential run. v1 put the family in `first` and the credential +# in `title`; 2.0 routed the pre-comma word to the family (the +# comma-family rule below) but the title peel still claimed the +# credential -- 'jr' and 'phd' were title vocabulary, and the +# period-abbreviation inference reads 'Jr.' as a title regardless -- +# so the postnominal position is read first now: a post-comma run +# that is nothing but suffix pieces is suffixes. A one-word name, a +# comma, one short word: the reach is pinned by the guards. +name_regex = "(?i)^[a-z]+,\\s*[a-z]{2,6}\\.?$" +fields = ["title", "given", "family", "suffix"] + +[[change]] +issue = "fix(#296) a glued honorific before a lone credential: the credential is the postnominal" +# '田中さん, PhD': 'phd' left TITLES, so the lone post-comma credential +# is the suffix it is; the glued-honorific peel is #312's (2.1), so at +# this baseline both halves move: title 'PhD', family '田中さん' -> +# family '田中', suffix 'さん, PhD'. +name_regex = "(?i)^田中さん,\\s*phd$" +fields = ["title", "family", "suffix"] + +[[change]] +issue = "fix(#296) do is a name, so it no longer stops the leading-particle scan as a title" +# 'Dr. Do Van Johnson, MD': the fix(#367) title-and-particle rule that +# explained this name is gone with 'do' from TITLES -- a word in both +# the title and particle vocabularies stopped the transparency scan +# and stayed a title piece (master, after #367/#413, read title 'Dr. +# Do'); 'do' is a particle and a name now, and reads as the given +# name. At this baseline: given 'Do Van Johnson' -> given 'Do', +# family 'Van Johnson'; the title 'Dr.' is unchanged here. +name_regex = "(?i)^dr\\.?\\s+do\\s+van\\b" +fields = ["given", "family"] + +[[change]] +issue = "fix(#296) dr is not postnominal vocabulary, so 'John Smith, Dr.' keeps its split and its title" +# 'John Smith, Dr.' / 'John, Smith, Dr.': with 'dr' out of the suffix +# sets the post-comma 'Dr.' is a title only, so the comma is followed +# by nothing but titles and the pre-comma name keeps its split (the +# no-name-word repair): suffix 'Dr.' -> title 'Dr.' for the two-part +# spelling. The three-part spelling keeps suffix 'Dr.' (a third part +# is suffix by position) and moves only by gaining the +# COMMA_STRUCTURE report C2 gives a third part that is not suffix +# words. +name_regex = "(?i)^john,?\\s+smith,\\s*dr\\.?$" +fields = ["title", "suffix", "_ambiguities"] + +[[change]] +issue = "fix(#325) a split credential followed by another suffix after a one-word family comma reads as suffixes" +# 'Smith, Ph. D. Jr.' and the #325 rows it leads ('Smith, Ph. D. MD', +# 'Smith, Ph. D. III', 'Smith, Ph. D. Jr. MD', 'Smith, Ph.D. Jr.', and +# 'Smith, PhD Jr.', which the regex reaches too -- title 'PhD' at +# every baseline, suffix 'PhD, Jr.' now): +# with one word before the comma the name is the listing form, and +# the space-split 'Ph. D.' followed by another suffix was no longer +# the LONE post-comma piece, so it fell through to the given name -- +# a 1.4.0 regression (v1 read suffix 'Ph. D.', title 'Jr.'). The +# whole run is suffixes now. Literal rather than the shape: a shape +# regex absorbed 'Smith, Dr. Jr.', which is the comma-family rule's +# (the comment review). +name_regex = "(?i)^smith,\\s*ph\\.?\\s?d\\.?\\s" +fields = ["title", "given", "suffix"] + +[[change]] +issue = "fix(#325) a credential run across a second comma reads as suffixes" +# 'Smith, Jr., PhD': title 'Jr.', suffix 'PhD' -> suffix 'Jr., PhD'; +# segments 2+ compose with the run. +name_regex = "(?i)^smith,\\s*jr\\.?,\\s*phd$" +fields = ["title", "suffix"] + +[[change]] +issue = "fix(#296) an ambiguous acronym counts as a suffix only when written with its periods" +# 'Jack Ma.': rules.md#S2 -- "written with its periods, one after each +# letter"; a single trailing period is the abbreviation shape any +# word can wear. The gate was "any period" until #296's review, so +# suffix 'Ma.' -> family 'Ma.', which is how 1.4.0 read it. +name_regex = "(?i)^jack\\s+ma\\.$" +fields = ["family", "suffix", "_ambiguities"] + [[change]] issue = "fix(#424) the particle chain stops before the trailing numeral" # 'John van der Berg V': rules.md#P2 -- "a trailing suffix begins -- @@ -606,7 +740,19 @@ issue = "fix(#367) a title no longer displaces a leading particle out of the lea # ["given", "family", "suffix"] is a superset of this diff's fields. No # fields-only rule exists here, so this file's copy is the plain # classification. -name_regex = "(?i)^(mr|dr|sir|jr)\\.?\\s+van\\b" +name_regex = "(?i)^(mr|dr|sir)\\.?\\s+van\\b" +fields = ["given", "middle", "family"] + +[[change]] +issue = "fix(#367) an inferred title no longer displaces a leading particle either" +# 'Jr. Van Johnson': the same change as the rule above, for a leading +# piece the period-abbreviation inference reads as a title (H2). It +# sat in that rule's alternation as 'jr' while 'jr' was title +# vocabulary; #296's audit took it out (a postnominal only), so the +# member would now reach a word the vocabulary does not ship, and the +# shape gets its literal. The reading is unchanged: 'Jr.' is still a +# title by shape, and Van is still the leading name piece. +name_regex = "(?i)^jr\\.\\s+van\\b" fields = ["given", "middle", "family"] [[change]] diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index fdb0a7f1..bd5d5c65 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -40,21 +40,6 @@ # the top level here, since a mistyped table name would otherwise read # as an empty ledger rather than as a broken one. -[[change]] -issue = "fix(#367) a title-and-particle word stops the scan instead of being chained" -# 'Dr. Do Van Johnson, MD': the arrival the fix(#367) rule below named -# as the reason it lists three roles rather than two -- "a corpus name -# that moves `title` or `maiden` should arrive UNEXPLAINED and be read -# once, rather than be absorbed here." This is the `title` one. -# -# `do` is both a title and an ambiguous particle, so it stops the -# transparency scan and stays a title piece instead of being chained -# onto the name: title 'Dr.' -> 'Dr. Do', and the given name loses the -# word. Same mechanism as the 'Dr. St John Smith' example that rule -# describes; this is the first corpus name to exercise it. -name_regex = "(?i)^dr\\.?\\s+do\\s+van\\b" -fields = ["title", "given"] - [[change]] issue = "fix(#369) a given-name title licenses the bound given-name join with one word to spare" # 'Sheik abdul salam': rules.md#P5 -- "or after a given-name title, @@ -151,6 +136,150 @@ issue = "fix(#424) accepted: a particle of the suffix vocabulary opening the tra name_regex = "(?i)^john\\s+smith\\s+mc\\s+v$" fields = ["middle", "family", "suffix", "_ambiguities"] +[[change]] +issue = "fix(comma-family) a comma followed only by titles keeps the given/family split" +# 'Bob Jones, author' / 'Bob Jones, compositeur': the comma is followed +# by nothing but a title word, which said nothing about where the +# family name ends, so the pre-comma name keeps its positional read -- +# family 'Bob Jones' -> given 'Bob', family 'Jones', the title staying +# where it was. The FAMILY_COMMA rule that segment 0 is wholly the +# family rests on the writer having marked where it ends; a comma +# followed only by titles marked nothing ('John Smith, Mr.' is 'Mr. +# John Smith' with the honorific moved). +name_regex = "(?i)^bob\\s+jones,\\s*[a-z]+$" +fields = ["given", "family"] + +[[change]] +issue = "fix(comma-family) a comma followed only by titles keeps the given/family split, the C1 example" +# 'John Smith, Mr.' and 'John Smith, Mr. Jr.': the rules.md#C1 examples +# of the rule above (family 'John Smith' -> given 'John', family +# 'Smith'; title and suffix stay where they were). Its own literal rather than +# an alternation with 'Bob Jones', which would be no vocabulary. +name_regex = "(?i)^john\\s+smith,\\s*mr\\.?(\\s+jr\\.?)?$" +fields = ["given", "family"] + +[[change]] +issue = "fix(#296) a dropped prenominal takes the name position it occupies" +# 'Do Quang Minh': 'do' left TITLES -- a postnominal (D.O.) and a +# Vietnamese surname, never a prenominal -- so a leading 'Do' is no +# longer peeled as a title and falls through to the positional read: +# title 'Do', given 'Quang' -> given 'Do', middle 'Quang'. The +# 2026-07-30 audit table's disposition; 'phd', 'jr', 'junior' and +# 'se' left with it, and no corpus name leads with a bare one (the +# three that lead with 'Jr.' are read by the period inference and do +# not move). The rule's second name is 'MD, DO, DDS': title 'DO' -> +# given 'DO', the comma anchor's one corpus hit. `family` +# is deliberately OUT of the field list: the family name must not +# move on this rule, so a regression there stays loud. The anchors +# are the two openings a name has (start, or after a family comma). +name_regex = "(?i)(^|,\\s*)do\\b" +fields = ["title", "given", "middle", "_ambiguities"] + +[[change]] +issue = "fix(#296) dr is not postnominal vocabulary, so a trailing Dr. is a name word" +# 'John Smith Dr.' and 'dr Vincent van Gogh dr' with its #100 siblings +# (and, at 2.x only, 'Smith Dr', which 1.4.0 already read as the +# family): 'dr' left SUFFIX_WORDS, where it was v1 +# residue -- 'Dr.' is not a postnominal in any tradition -- so a +# trailing bare 'Dr' is no longer suffix vocabulary and falls to the +# positional read, taking the family name with it. NOT a new class: +# no trailing title-only word routes to title on the no-comma path +# ('John Smith Prof.' reads family 'Prof.' at every baseline), and +# the suffix entry was the only thing making 'dr' the exception. #316 +# is the open question for the class. +name_regex = "(?i)\\sdr\\.?$" +fields = ["middle", "family", "suffix"] + +[[change]] +issue = "fix(#296) a credential-only comma string reads a name and its postnominal" +# 'Jr., PhD' / 'MD, PHD': a string that is nothing BUT credentials. +# v1 read the trailing one as a title and the leading one as a first +# name; with 'phd' out of TITLES the comma structure reads it as a +# one-word name plus a postnominal -- a reading of a string nobody +# writes as a name, and the honest one for it. +name_regex = "(?i)^[a-z]{2,3}\\.?,\\s*phd$" +fields = ["title", "given", "family", "suffix"] + +[[change]] +issue = "fix(#296) a lone post-comma credential is a suffix" +# 'Smith, Jr.' / 'Kim, Jr.' / 'Smith, PhD': rules.md#C1 -- the listing +# form holds with one word before the comma, and the word after it is +# the credential run. v1 put the family in `first` and the credential +# in `title`; 2.0 routed the pre-comma word to the family (the +# comma-family rule below) but the title peel still claimed the +# credential -- 'jr' and 'phd' were title vocabulary, and the +# period-abbreviation inference reads 'Jr.' as a title regardless -- +# so the postnominal position is read first now: a post-comma run +# that is nothing but suffix pieces is suffixes. A one-word name, a +# comma, one short word: the reach is pinned by the guards. +name_regex = "(?i)^[a-z]+,\\s*[a-z]{2,6}\\.?$" +fields = ["title", "given", "family", "suffix"] + +[[change]] +issue = "fix(#296) a glued honorific before a lone credential: the credential is the postnominal" +# '田中さん, PhD': 'phd' left TITLES, so the lone post-comma credential +# is the suffix it is, and the glued-honorific peel (#312) reads 田中 +# and さん as before. title 'PhD' -> suffix 'さん, PhD'. +name_regex = "(?i)^田中さん,\\s*phd$" +fields = ["title", "family", "suffix"] + +[[change]] +issue = "fix(#296) do is a name, so it no longer stops the leading-particle scan as a title" +# 'Dr. Do Van Johnson, MD': the fix(#367) title-and-particle rule that +# explained this name is gone with 'do' from TITLES -- a word in both +# the title and particle vocabularies stopped the transparency scan +# and stayed a title piece (master, after #367/#413, read title 'Dr. +# Do'); 'do' is a particle and a name now, and reads as the given +# name. At this baseline: given 'Do Van Johnson' -> given 'Do', +# family 'Van Johnson'; the title 'Dr.' is unchanged here. +name_regex = "(?i)^dr\\.?\\s+do\\s+van\\b" +fields = ["given", "family"] + +[[change]] +issue = "fix(#296) dr is not postnominal vocabulary, so 'John Smith, Dr.' keeps its split and its title" +# 'John Smith, Dr.' / 'John, Smith, Dr.': with 'dr' out of the suffix +# sets the post-comma 'Dr.' is a title only, so the comma is followed +# by nothing but titles and the pre-comma name keeps its split (the +# no-name-word repair): suffix 'Dr.' -> title 'Dr.' for the two-part +# spelling. The three-part spelling keeps suffix 'Dr.' (a third part +# is suffix by position) and moves only by gaining the +# COMMA_STRUCTURE report C2 gives a third part that is not suffix +# words. +name_regex = "(?i)^john,?\\s+smith,\\s*dr\\.?$" +fields = ["title", "suffix", "_ambiguities"] + +[[change]] +issue = "fix(#325) a split credential followed by another suffix after a one-word family comma reads as suffixes" +# 'Smith, Ph. D. Jr.' and the #325 rows it leads ('Smith, Ph. D. MD', +# 'Smith, Ph. D. III', 'Smith, Ph. D. Jr. MD', 'Smith, Ph.D. Jr.', and +# 'Smith, PhD Jr.', which the regex reaches too -- title 'PhD' at +# every baseline, suffix 'PhD, Jr.' now): +# with one word before the comma the name is the listing form, and +# the space-split 'Ph. D.' followed by another suffix was no longer +# the LONE post-comma piece, so it fell through to the given name -- +# a 1.4.0 regression (v1 read suffix 'Ph. D.', title 'Jr.'). The +# whole run is suffixes now. Literal rather than the shape: a shape +# regex absorbed 'Smith, Dr. Jr.', which is the comma-family rule's +# (the comment review). +name_regex = "(?i)^smith,\\s*ph\\.?\\s?d\\.?\\s" +fields = ["title", "given", "suffix"] + +[[change]] +issue = "fix(#325) a credential run across a second comma reads as suffixes" +# 'Smith, Jr., PhD': title 'Jr.', suffix 'PhD' -> suffix 'Jr., PhD'; +# segments 2+ compose with the run. +name_regex = "(?i)^smith,\\s*jr\\.?,\\s*phd$" +fields = ["title", "suffix"] + +[[change]] +issue = "fix(#296) an ambiguous acronym counts as a suffix only when written with its periods" +# 'Jack Ma.': rules.md#S2 -- "written with its periods, one after each +# letter"; a single trailing period is the abbreviation shape any +# word can wear. The gate was "any period" until #296's review, so +# suffix 'Ma.' -> family 'Ma.', which is how 1.4.0 read it. +name_regex = "(?i)^jack\\s+ma\\.$" +fields = ["family", "suffix", "_ambiguities"] + [[change]] issue = "fix(#424) the particle chain stops before the trailing numeral" # 'John van der Berg V': rules.md#P2 -- "a trailing suffix begins -- @@ -330,7 +459,19 @@ issue = "fix(#367) a title no longer displaces a leading particle out of the lea # ["given", "family", "suffix"] is a superset of this diff's fields. No # fields-only rule exists here, so this file's copy is the plain # classification. -name_regex = "(?i)^(mr|dr|sir|jr)\\.?\\s+van\\b" +name_regex = "(?i)^(mr|dr|sir)\\.?\\s+van\\b" +fields = ["given", "middle", "family"] + +[[change]] +issue = "fix(#367) an inferred title no longer displaces a leading particle either" +# 'Jr. Van Johnson': the same change as the rule above, for a leading +# piece the period-abbreviation inference reads as a title (H2). It +# sat in that rule's alternation as 'jr' while 'jr' was title +# vocabulary; #296's audit took it out (a postnominal only), so the +# member would now reach a word the vocabulary does not ship, and the +# shape gets its literal. The reading is unchanged: 'Jr.' is still a +# title by shape, and Van is still the leading name piece. +name_regex = "(?i)^jr\\.\\s+van\\b" fields = ["given", "middle", "family"] [[change]]