Repository navigation
Fix forward word bounds when ZWJ + emoji follows MidLetter or MidNum - #187
Merged
Merged
Conversation
After a letter (or digit) and a MidLetter/MidNumLet/MidNum/quote
character, the forward iterator waits for a letter (or digit) to apply
WB6/WB7, WB7b/WB7c or WB11/WB12. A ZWJ followed by an
Extended_Pictographic character took the WB3c shortcut instead, which
dropped that pending lookahead, so "a.\u{200d}\u{1f44d}" came out as
one segment. The reverse iterator gives ["a", ".\u{200d}\u{1f44d}"],
which matches the rules. Rewind in this case, as for any other
character that isn't the required letter or digit.
Manishearth
approved these changes
Oct 26, 2026
Member
|
MSRV failure unrelated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
split_word_bounds()and its reverse iterator disagree when a letter or digit is followed by a MidLetter/MidNumLet/MidNum/quote character and then ZWJ + an Extended_Pictographic character:"a.\u{200d}\u{1f44d}"["a.\u{200d}👍"]["a", ".\u{200d}👍"]"1,\u{200d}\u{1f44d}"["1,\u{200d}👍"]["1", ",\u{200d}👍"]"\u{5d0}\"\u{200d}\u{1f44d}"["\u{5d0}\"\u{200d}👍"]["\u{5d0}", "\"\u{200d}👍"]The reverse result is the correct one. WB6/WB7 (and WB7b/WB7c, WB11/WB12) only join across the middle character when a letter (or digit) follows it after WB4. Here the next character is the emoji, so there is a break before the
.; the., ZWJ and emoji stay together by WB4 and WB3c.In
UWordBounds::next, the state isFormatExtend(RequireLetter | RequireHLetter | RequireNumeric)while it waits for that letter or digit. The WB3c shortcut (prev_zwj && is_emoji(ch)) switched toEmojiand continued, which dropped the pending lookahead and its rewind. This change breaks out in that case, so the existing rewind code emits the letter or digit on its own, as it does for any other character that isn't the required one.Tests: three cases added to
EXTRA_TESTSintest_words, which checks the forward and reverse iterators and their indices. They fail before the change and pass after it.cargo test,cargo clippy --all-targets --allandcargo fmt --all --checkpass, and the library builds with 1.85.0. (On 1.85.0 the dev-dependency proptest 1.12 needs rustc 1.88, which is unrelated to this change.)How I found it: forward vs reverse word bounds on 2M random strings drawn from one or two characters of each Word_Break class. Before: 60 mismatches, all of this shape. After: none. A cross-check of grapheme and word bounds against libunibreak (on characters whose properties didn't change between Unicode 17 and 18) agrees too.