Repository navigation
feat: Upgrade to Unicode 18.0.0 - #184
Merged
Merged
Conversation
Implement GB9c rule change from Unicode Standard Annex unicode-rs#29: simplify Indic conjunct break from `Consonant [Extend Linker]* Linker [Extend Linker]* × Consonant` to `Linker Extend* × Consonant` This also handles three new InCB=Linker characters (U+1CF5, U+1CF6, U+11A3A) that have gcb categories other than Extend/ZWJ, requiring fixes to the linker count tracking in next_boundary and prev_boundary.
Manishearth
approved these changes
Sep 17, 2026
Manishearth
left a comment
Manishearth
left a comment
Member
There was a problem hiding this comment.
I think this looks correct.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR updates Unicode support to version 18.0.0 (Unicode Standard Annex #29 in Unicode 18.0.0). See changes done in UAX#29 in Unicode 18.0.0 here.
Algorithmic (
src/grapheme.rs)Rewrite
handle_incb_consonantfor the new GB9c rule. The old rule required a preceding InCB=Consonant before the Linker sequence; the new rule only requires an InCB=Linker (with optional InCB=Extend) before the InCB=Consonant. The backward walk now simply skips extends, looks for a linker, and suppresses the break.Fix
next_boundarylinker count tracking. Since Unicode 18, some InCB=Linker characters (U+1CF5, U+1CF6, U+11A3A) havegcb=Otherinstead of Extend/ZWJ. The old code reset the linker count before reaching the linker check. Nowis_incb_linkeris checked first.Fix
prev_boundarylinker count tracking. Handle the case where a linker is encountered with count=0 (previously fell through to None).Data generation (
scripts/)unicode.py: Relaxedcheck_incb_derivableto warn (not crash) when InCB=Linker characters exist outside Extend∪ZWJ. Fixed two pre-existing bugs inis_incb_linkergeneration: range upper bound usedloinstead ofhi, and..instead of..=.unicode_gen_breaktests.py: Updated regex for Unicode 18 test data format (sot/eot markers changed from[0.2]/[0.3]to[1.0]/[2.0]).Tests
handle_incb_consonant(basic, no-linker, across-chunks) andis_incb_linkercoverage of the three newUnicode 18 linkers.
test_grapheme_cursor_incb_consonant_chunkedintegration test verifying cursor-based chunked mode matches the iterator for all five GB9c test sequences.EXTRA_DIFFentry for Linker+Extend+Consonant extended vs legacy behavior.