Align the prep-diff tool with the exporter's normalization rules - #279
Open
sirreal wants to merge 19 commits into
Open
Align the prep-diff tool with the exporter's normalization rules#279sirreal wants to merge 19 commits into
sirreal wants to merge 19 commits into
Conversation
`Hook_Reflector::getName()` short-circuited on `Scalar\String_` nodes and returned the interpreted string value. Escape sequences were therefore resolved, so `do_action( "\x09tab" )` exported a literal tab and `do_action( "\xC0 bad" )` exported a raw 0xC0 byte. That byte is not valid UTF-8, `json_encode()` returns `false` for it, and `wp parser export` silently produced no JSON at all for the whole run. Route string nodes through `Pretty_Printer` like every other expression. The printer returns php-parser's `rawValue` attribute, which is the source-verbatim spelling, and `cleanupName()` strips the quotes. Also check `json_encode()` for failure in `Command::_get_phpdoc_data()` and fail loudly with `json_last_error_msg()` instead of writing an empty file.
The pretty printer inherits an override that returns PHP-Parser's `rawValue` attribute so escape sequences are not interpreted. PHP-Parser sets that attribute to the body of a doc string, without the delimiters, so `apply_filters( 'f', <<<EOT ... EOT, 2 )` exported its argument as a bare `body` string with embedded newlines instead of PHP source. Print heredoc and nowdoc nodes with the default printer, which reproduces the `<<<LABEL ... LABEL` form and does not interpret escape sequences in doc strings either.
`pName_FullyQualified()` prints single-segment fully-qualified names without the leading backslash regardless of namespace context, so inside a namespaced file the printed form denotes a namespaced symbol rather than the global one. This is an accepted limitation because the parser targets global-namespace WordPress core code.
The global namespace prefixes are stripped from inline `{@link}` and
`{@see}` references after the DocBlock text has been rendered, so the
stripping also reached into rendered code regions and silently deleted
the backslash an author had written in a verbatim code sample.
Carve out `<code>` regions before stripping, the same way `fix_newlines()`
protects the newlines in those regions, so code samples are exported as
they were written.
The quote-stripping pattern required a body free of quote characters, so a hook name that contained one, like `do_action( "it's" );`, was exported with the quotes that surround it in the source. Match the opening quote and require the same quote at the end, allowing the body to hold the other quote character or an escaped copy of the delimiter. Only that pair is stripped; the body keeps its source spelling, so `do_action( 'it\'s' );` exports as `it\'s`. Concatenated expressions still fall through to the dynamic-name handling below.
`Method_Call_Reflector::_getClassMapping()` maps a handful of WordPress factory functions to the class they return, so that `get_current_screen()->add_help_tab()` is exported as a use of `WP_Screen::add_help_tab()`. The lookup never matched before this branch, because the printed receiver carried a leading backslash that the mapping keys do not have. Pin the restored behavior with a test.
Inline `{@see}`/`{@link}` references in prose and the reference tokens
of `@see`/`@link` tags are documentation text: the export should carry
the author's spelling, backslash included. Pin that, and pin the
boundary on the other side: `@param` types pass through phpDocumentor's
type resolution, which prefixes every non-keyword type with a synthetic
`\`, so the author's spelling is unrecoverable there and the synthetic
prefix must keep being stripped.
Stop stripping global namespace prefixes from documentation text:
inline `{@see}`/`{@link}` references in descriptions and tag content,
and the reference tokens of `@see`/`@link` tags. These fields carry the
author's own words — phpDocumentor hands them over verbatim — so any
leading backslash present was written by hand and any rewrite imposes
a spelling the author didn't choose. The consumer that renders these
references (the developer.wordpress.org theme re-parses the inline tags
at render time) owns display policy.
This removes the code-region protection machinery along with the strip:
with nothing rewriting the text, code samples and inline code spans are
verbatim by construction, and the existing tests for them become plain
regression pins.
Tag types and argument types keep the strip: those values pass through
phpDocumentor's type resolution, which synthesizes a leading `\` on
every non-keyword type regardless of how the author spelled it, so the
prefix there is resolver output, not authorship.
Extracted from the original "Preserve expression output" commit so the diff tool's changes can land separately from the exporter's. The diff normalization must follow the same rules the exporter uses when it strips PHP-Parser's synthetic global namespace prefixes, otherwise the normalized output no longer matches what the exporter produces.
`tests/prep-diff-test.php` was a standalone script that nothing executed: `phpunit.xml.dist` only loads `tests/phpunit/tests/`, and both `composer test` and CI run PHPUnit with that configuration. Move it under `tests/phpunit/tests/` as a test case so that it runs with the rest of the suite. `prep-diff.php` normalized printed expressions with its own prose-oriented regular expression, which also matched global names appearing inside string literals. An argument default of `'see \Foo bar'` was rewritten to `'see Foo bar'`, which is exactly the mangled output the script is meant to expose, so a regression in expression printing diffed clean. Reuse the exporter's `WP_Parser\strip_global_namespace_prefix()` instead, which only strips a prefix at the start of a value. Inline documentation references are unaffected: they never contain whitespace, so both rules produce the same result for them. Diffs of output generated before and after this branch now show the string-literal fix instead of hiding it, which is intended.
The exporter now hands documentation text through as authored: inline references in descriptions and tag content, and the reference tokens of `@see`/`@link` tags. A difference in those fields between two builds is a real behavior change, so the diff normalization must not erase it.
Match the exporter: documentation text is exported as authored, so the diff tool has nothing to reconcile in descriptions, tag content, or the `refers`/`link` tokens — and normalizing them would hide a real change in exactly the fields the exporter promises not to touch. Expression and type fields keep the anchored-prefix normalization the exporter still applies.
sirreal
force-pushed
the
prep-diff-updates
branch
from
August 20, 2026 10:10
87520b5 to
9b41ff9
Compare
sirreal
commented
Aug 24, 2026
Member
Author
There was a problem hiding this comment.
I think we should add unescaped slashes, unescaped unicode, and unescaped line terminators here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #262 (base branch:
fix-254). Extracts the diff-tool work from that PR so it can be reviewed separately from the export-semantics changes.The rule
prep-diff.phpnormalizes generated JSON so two exports can be compared without incidental churn. Its normalization must mirror the exporter's own rules exactly — anything it normalizes beyond them is a real behavior change it will hide.After this branch, the tool follows the same split #262 establishes:
description,long_description, and tagcontent,refers, andlink. The exporter now emits these as authored, so the diff tool has nothing to reconcile. Normalizing them would cancel out an intended change on both sides of a diff and report it as no change at all.class,default,extends,type,value, and thetypes/aliases/implementscollections, matching what the exporter still normalizes.Why it matters
Without the final commit, a corpus diff of #262 reports roughly zero documentation-text hunks — the tool strips the author-written backslash from
{@see \parse_blocks()}on both sides, so exactly the change #262 exists to make becomes invisible to review. With this branch, that change shows up: the manual corpus verification quoted in #262's description was produced with this branch'sprep-diff.php.The earlier hand-rolled prose regex had the same failure mode in miniature: it matched inside string literals, rewriting
'see \Foo bar'to'see Foo bar'on both sides. Replacing it with the exporter's own anchored\WP_Parser\strip_global_namespace_prefix()means a regression that re-introduces string corruption now surfaces in the diff instead of vanishing.Commits
02a2035) — the prep-diff changes originally bundled into Preserve documentation and hook string semantics #262's "Preserve expression output" commit: normalization for the fields whose output Preserve documentation and hook string semantics #262 changes.da20168) — relocatestests/prep-diff-test.phptotests/phpunit/tests/prep-diff.phpas a PHPUnit class so CI actually executes it (previously no runner invoked it), and swaps the prose regex for the exporter's shared helper.6ce9808) — pins the pass-through behavior.9b41ff9) — drops the{@see}/{@link}rewrite and removeslinkandrefersfrom the normalized keys.Testing
prep-diff.phpstill runs standalone:php prep-diff.php < export.json.