nameparser 2.0 implementation#288
Draft
derek73 wants to merge 88 commits into
Draft
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The v2 core design uses enum.StrEnum (3.11+). Full #257 (dropping typing_extensions, CI matrix) remains tracked on the issue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Validate that all replace() field values are str (user error if None) - Append missing-role synthetics in canonical Role order, not kwargs order - Unquote return type annotation (postponed annotations in effect) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Casefolded seven-component tuple in canonical Role order for dedup, dict keys, and sorting. Semantic layer comparison; __eq__ remains strict. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds tests/v2/test_layering.py to mechanically enforce the conventions
doc's import layering and the public v2 export surface. Appends the v2
core types (Span, Role, Token, Ambiguity, AmbiguityKind, ParsedName,
Lexicon, Policy, PolicyPatch, PatronymicRule, UNSET, GIVEN_FIRST,
FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, Locale) to nameparser/__init__.py
alongside the existing v1 HumanName export.
Turns on component-flag mypy strictness for the four v2 modules and
check_untyped_defs for tests/v2, then fixes the resulting test-only
type mismatches by using the precise constructed types (Span(...),
frozenset({...}), tuple-of-pairs) where the literal type didn't matter
to the test, and adding narrow # type: ignore[arg-type] comments only
where a test deliberately exercises runtime coercion/validation of an
intentionally mismatched static type (e.g. Token span/Ambiguity kind
coercion tests, PatronymicRule string coercion, dict aliasing test).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pickle.dumps(Lexicon.default()) raised TypeError because the default slots-dataclass pickle path serializes the _cap_map MappingProxyType. Ship every other slot and rebuild the proxy from the canonical capitalization_exceptions tuple on load. Parser (Plan 3) is picklable by construction per the core spec, and a Parser holds a Lexicon. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
requires-python is >=3.11 on this branch, so the 3.10 job can no longer install the package (uv sync either fails or silently substitutes a managed 3.11). The rest of #257 (typing_extensions, classifiers) stays tracked on the issue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With requires-python at >=3.11, Self imports directly from typing and the conditional dependency can never activate; ruff (UP035/UP036) flags the dead version block once the floor is raised. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The v1 suite is fully annotated (since #250 put tests/ under mypy), so pyproject carries no per-file ANN ignores; the new tests/v2 modules must be annotated too or 'ruff check' fails in CI. Also drops an unused import ruff flagged (F401). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_types still claimed to import nothing from nameparser -- now false in the unqualified form (call-time _render imports, TYPE_CHECKING Lexicon) and contradicting the layering ALLOWED entry; AGENTS.md had the qualified wording but the docstring a reader actually opens did not. _render's docstring understated its _lexicon dependency (_normalize is used unconditionally) and now notes that only unknown KEYS get the enriched KeyError. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lexicon failed at the load site on field-layout mismatch; its five frozen siblings (Token, Ambiguity, ParsedName, Policy, PolicyPatch, Locale) restored slots silently and failed at a distant attribute read. Shared _guarded_getstate/_guarded_setstate functions in _types are ASSIGNED IN EACH CLASS BODY -- @DataClass(slots=True) regenerates the class and installs its own pickle methods unless the names are in the class's own __dict__, which is why a mixin cannot work. Lexicon keeps its own copy (layering forbids _lexicon importing _types) plus its mappingproxy rebuild. Policy, stated in code and AGENTS.md: unpickle validates field LAYOUT, not values -- pickle is not a security boundary, and canonical state comes only from a validated instance. _locale's ALLOWED entry gains the downward _types import. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
capitalized() with a non-Lexicon argument was a silent no-op on mixed-case input (the gate returned before touching the argument) and a deep AttributeError on single-case input; render()/initials() with non-str spec/delimiter/separator failed inside str.format. All three now raise the taxonomy's eager TypeError naming the argument, matching every constructor on the branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tuple('gmf') is ('g', 'm', 'f'), so a bare string fell through to the
permutation ValueError instead of the bare-string TypeError every
other iterable-valued field raises. Guarded in both Policy and
PolicyPatch (whose canonicalization would otherwise store the
shredded tuple until apply time).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The given-group tag immunity in initials() (a tagged given token still contributes -- the PARTICLE_OR_GIVEN reading) and the raw-error contract for malformed render specs (positional fields, bad conversions) were documented but untested. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AGENTS.md listed _render under 'later' modules one line above the bullet describing its shipped layering; the one-sanctioned-global rule did not acknowledge Lexicon.default()'s functools.cache (a lazily cached FROZEN singleton is a constant, not state); and the Lexicon repr comment cited '~700 entries' when the default vocabulary is ~1460 -- de-numbered so it cannot rot again. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extract _format_spec for the identical fill/enrich/collapse tail of render() and initials() (the two error templates could drift independently); replace initials()'s guarded join-plus-append with separator.join(letter + delimiter ...), which makes the empty-group conditional unnecessary; drop a backslash continuation for the house paren style; record WHY capitalized()'s gate reads joined token texts rather than render() output. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The identical TypeError lived in both Policy and PolicyPatch __post_init__ and could drift; extracted to a module helper. Also add the missing back-reference on Lexicon.__setstate__ pointing at its deliberately-duplicated twin in _types (the sync note was previously one-directional). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Forgetting the two class-body guard assignments on a future frozen type is silent -- @DataClass(slots=True) installs its own working pickle methods without skew detection. A conventions test now enumerates every frozen dataclass in the package and asserts the guards are present (Lexicon allow-listed for its private copy), turning a per-class convention into a CI gate. Also folds five vestigial function-local render imports into the top import block. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports the emoji/bidi character classes verbatim from config/regexes.py per the layering contract's duplication-by-design note; whitespace, emoji, bidi marks, and comma chars all act as separators without ever entering a token, so token spans always index original exactly. Wires tokenize into STAGES and registers the new module in test_layering's _MUST_EXIST. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds _vocab.py (is_initial/is_suffix_strict/is_suffix_lenient, shared across stages) and _segment.py (the family-comma/suffix-comma/no-comma decision). segment reads Lexicon suffix vocabulary directly rather than waiting for classify (recorded plan deviation #3: the suffix-comma vs family-comma call is definitionally vocabulary-dependent, mirroring v1's are_suffixes_after_comma). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports v1's join_on_conjunctions, prefix chains, and _join_bound_first_name as pure index operations over WorkToken runs (the anti-#100 invariant: no value-based lookup, no string joins). Adds two v2 additions from the plan: the "Ph. D."-split merge (plan deviation #1 -- lands in group since assign needs the merged piece to exist first) and the maiden-marker consuming rule (#274). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Completes the seven-stage STAGES tuple. Patronymic regexes verified byte-for-byte against nameparser/config/regexes.py's east_slavic/turkic entries -- no mismatch found. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Not a pipeline stage but the runner's tail: converts the seven-stage fold's ParseState into a public ParsedName, dropping structural marker tokens (e.g. maiden markers) and materializing PendingAmbiguity indices into real Ambiguity objects over the final token tuple. Also adds _pipeline/_assemble.py to test_layering's _MUST_EXIST now that the file exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds tests/v2/cases.py (Case dataclass + CASES tuple) and tests/v2/test_cases.py (the core runner), seeded with the v1 parity battery pinned in the plan header plus three carried-forward rows verified live against v1.4.0: - suffix_stays_suffix / suffix_stays_suffix_title: v1 routes a lone trailing recognized suffix (PhD) to the family/last field; v2 keeps it in suffix. Classified fix(suffix-routing). - family_comma_lone_title: v1 puts the pre-comma piece in first when it's followed only by a title; v2 treats pre-comma as definitionally family. Classified fix(comma-family). All 35 case rows pass against the pipeline as built through Task 11 with no corrections needed to the plan's predicted values; the East Slavic/Turkic policy rows were cross-checked against v1's equivalent patronymic_name_order opt-in and confirmed as true parity. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| # patterns are duplicated by design with this provenance note. When | ||
| # editing, keep both copies in sync. | ||
| _EMOJI = re.compile('[' # lgtm[py/overly-large-range] | ||
| '\U0001F300-\U0001F64F' |
The ParseState ownership map said roles belong to assign/post_rules exclusively, but group writes maiden roles -- an ownership contract wrong in one place teaches readers to distrust it everywhere. Fixed the map, noted post-group segments staleness and the sole-producer text invariant on WorkToken, documented PendingAmbiguity's dangling- index tolerance and Parser's None-default resolution, made assemble's role fallback say what it means, and pinned the whole ownership map mechanically: a new test folds the case corpus stage by stage and asserts each stage changes only the fields it owns. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
In the real data suffix_acronyms_ambiguous is a SUBSET of
suffix_acronyms (v1 shape), so the plain membership test in classify
and is_suffix_strict fired before the period gate could withhold
vocab:suffix -- the gate was dead code for the overlap members
({'ed', 'jd'}), and 'Smith, Ed' silently dropped the given name into
suffix. Both sites now exclude the ambiguous subset, Lexicon enforces
the subset invariant like particles_ambiguous, the classify/vocab
fixtures mirror the real subset shape instead of hiding the bug with
disjoint sets, and three v1-verified case rows pin the parity.
Found by cross-session PR review (C1/I1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The group-stage merge reunited the two tokens as a piece, but the suffix view joins SUFFIX tokens with ', ' -- 'John Ph. D.' rendered as suffix 'Ph., D.' -- and a mid-name credential never reached the trailing suffix peel, landing in MIDDLE. Continuation tokens of a suffix-merged piece now carry a new stable 'joined' tag (spec's provenance-tag sketch anticipated it) which the suffix view uses to attach with a space; assign peels group-flagged suffix pieces at ANY position, matching v1's fix_phd which extracted the credential before parsing. Contract-test trigger added for the new stable tag; two v1-verified case rows pin the behavior. Found by cross-session PR review (I2/I3). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
v1's handle_non_first_name_prefix was unported: 'de la Vega' produced
given='de' with no ambiguity ('de' is not particles_ambiguous, so
PARTICLE_OR_GIVEN never fires -- a silent misassignment). Port it as
post_rules rule 1b keyed on the parsed given exactly like v1: a lone
given token tagged particle-but-not-ambiguous with middles or family
present folds into FAMILY; 'Jean de Mesnil' (non-leading) and bare
'de' (degenerate guard) stay untouched, and ambiguous 'van Gogh'
keeps the given reading. v1-verified case row added. Found by
cross-session PR review (I4).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
middle_as_family was validated but consumed by no stage -- ported v1's handle_middle_name_as_last as post_rules rule 4 (runs after the patronymic rotations, comma or not, matching v1's post_process order; span order reproduces v1's prepend). lenient_comma_suffixes now gates segment's post-comma predicate (strict mode vetoes initial-shaped suffix words). extra_suffix_delimiters cannot be honored until token re-splitting lands with the migration work, so setting it warns instead of silently doing nothing. Found by cross-session PR review (I5). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PR review I6/I7/I8/I9: - assemble omits an ambiguity whose referent tokens were ALL dropped (born-empty ambiguities like UNBALANCED_DELIMITER are kept -- they are token-independent by design) - Policy.name_order now accepts only GIVEN_FIRST / FAMILY_FIRST / FAMILY_FIRST_GIVEN_LAST: the unnamed permutations had no implemented assignment semantics and would silently misassign. Non-Role elements get the taxonomy's TypeError. - Span.__add__ comment no longer cites a nonexistent join stage; the real rationale is that no covering-span operation exists (anti-#100) - the stage ownership test now also pins token-level ownership: texts and spans fixed at tokenize, classify changes only tags, the role-assigning stages only roles (group also tags, for 'joined') - docstring polish: ParseState names the three ambiguity-recording stages; _types names the matches() call-time import; _assign's PARTICLE_OR_GIVEN wording is order-neutral Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- T1: three-piece FAMILY_FIRST_GIVEN_LAST pins given-from-the-END
semantics (not a rotation of FAMILY_FIRST)
- T2: reverse-coverage property -- every input char lies in a token
span, a masked delimited span, or is individually ignorable; no
character silently vanishes
- T3: case row for post-comma non-suffix extras ('Smith, John, Extra,
Jr.' -> suffix 'Extra, Jr.' + COMMA_STRUCTURE; v1 parity pinned live)
- T4: multiple unbalanced delimiters are each reported; the scan does
not stop at the first unmatched opener
- T6: digits join the Hypothesis stress alphabet
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implementation branch for the 2.0 design — see the RFC in #285 and the umbrella issue #284 for the design discussion and feedback questions. Implementation learnings that change the design will land as amendment commits on #285, per the plan described there.
Progress
Span,Role,Token,Ambiguity/AmbiguityKind,ParsedName;Lexicon,Policy/PolicyPatch/apply_patch,Locale. Frozen, slotted, hashable, picklable dataclasses with eager fail-loud validation; ~110 dedicated tests undertests/v2/.render()/initials()/capitalized()/__str__)Parser/parse()/parser_for()/matches()+ shared case tableHumanNamefacade +CONSTANTSshim (existing test corpus as regression harness)ru,tr_az) + differential-parse harnessNotes on what's here so far
TypeError, well-typed bad value →ValueError, enum lookups stayValueErrorper stdlib precedent.tests/v2/test_layering.py), and the seven-field canonical order derives fromRoledeclaration order everywhere — no ordering is ever stated twice.nameparser/configdata modules through 2.x, including the newmaiden_markers.py(Recognize maiden-name markers like "née" and "geb." (Jane Smith née Jones) #274 — marker words like "née"/"geb."; decisions recorded on the issue).🤖 Generated with Claude Code