Fix normalize_content() dropping text between shortcodes and entity c…

[sonali195]

What?

Fixes normalize_content() deleting parts of post content before it is sent to AI features.

Why?

get_post_context() passes post content through normalize_content(), which had two bugs:

1. The shortcode regex #\[.+\](.+)\[/.+\]# was greedy and ran after newlines were removed, so it matched from the first [ to the last ] in the whole post.
   Intro [note]First[/note] middle text [tip]Second[/tip] outro became Intro Second outro. Plain brackets like [1] also triggered it.
2. HTML entities were deleted instead of decoded. the_content texturizes quotes into ’ and similar, so don't became dont and R&D became RD.

How?

• Unwrap only a shortcode whose opening tag matches its own closing tag ([name ...]…[/name]), non-greedily, repeating so nested shortcodes unwrap from the inside out.
• Decode entities with html_entity_decode() after wp_strip_all_tags(), so escaped markup like <div> stays as literal text; convert non-breaking spaces to regular spaces.
• Make the <br> replacement case-insensitive and accept <br />.

The function signature and both filters are unchanged.

Use of AI Tools

AI assistance: Yes
Tool(s): Claude
Used for: Finding the bug, drafting the fix and regression tests. I reviewed the changes and take responsibility for them.

Testing Instructions

1. npm run wp-env:test start
2. npm run test:php -- --filter normalize_content
3. Or call WordPress\AI\normalize_content( 'Intro [note]First[/note] middle [tip]Second[/tip] outro' ) and confirm all text is kept.

Changelog Entry

Fixed - Preserve HTML entities and text between shortcodes when normalizing content, while still encoding entity-encoded tags.

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: sonali195 <[email protected]>
Co-authored-by: dkotter <[email protected]>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.59%. Comparing base (f3f2bd2) to head (ef5206e).

Additional details and impacted files

@@            Coverage Diff             @@
##             develop    #1077   +/-   ##
==========================================
  Coverage      81.58%   81.59%           
  Complexity      3081     3081           
==========================================
  Files            129      129           
  Lines          12275    12281    +6     
==========================================
+ Hits           10015    10021    +6     
  Misses          2260     2260           

Flag	Coverage Δ
unit	81.59% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[dkotter]

Left a few comments that need action.

In addition, for future reference, would be great to open an Issue first, documenting the actual problem, and then proceeding to the PR stage. While I overall understand the bug, there's really no context in the PR around the actual scenario you're running into where the bug crops up or real-life reproduction steps, making it harder to assess the severity here.

[sonali195]

Good catch, thanks. Moved html_entity_decode() before wp_strip_all_tags(), so entity-encoded markup like </block-content> is decoded and then stripped instead of coming back as a real tag. Added test_normalize_content_strips_entity_encoded_tags to cover this, and updated the existing entity test since <test> is now stripped.