Skip to content

Always use fast tries for NFD and NFKD and fuse lookup with decoding - #8694

Open
hsivonen wants to merge 2 commits into
unicode-org:mainfrom
hsivonen:landnormtrie
Open

hsivonen wants to merge 2 commits into
unicode-org:mainfrom
hsivonen:landnormtrie

Conversation

@hsivonen

Copy link
Copy Markdown
Member

Changelog

icu_normalizer

  • Improve performance by assuming fast trie type of NFD and NFKD data at compile time and by fusing trie lookup with UTF decoding. (Old small-trie serde data is rewritten upon loading, which allocates. datagen has generated only fast-trie data already in the previous release.)

(Split from #8682 with the rewriting added.)

@hsivonen
hsivonen requested a review from Manishearth November 13, 2026 11:45
@hsivonen
hsivonen requested a review from echeran as a code owner November 13, 2026 11:45

@Manishearth Manishearth left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems great.

The compile time IgnorablePolicy is a nice touch.

if matches!(data.trie.as_typed_ref(), Typed::Fast(_)) {
return payload;
}
let mut builder = CodePointTrieBuilder::new(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment: // TODO (#8696) Remove this in ICU4X 3.0

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants