icukit

A comprehensive Python toolkit for Unicode and internationalization, built on ICU (International Components for Unicode).

icukit provides a Pythonic interface to ICU's powerful text processing, localization, and internationalization capabilities. It includes both a library API and a command-line interface.

Installation

pip install icukit

This pulls in icukit-pyicu, which bundles pre-built ICU libraries and PyICU, and cartlet>=0.8 from PyPI. There are no system dependencies to install on either macOS or Linux — the wheels are self-contained.

See Installation Guide for details, including how to use a system PyICU instead.

Features

Text Processing

• Transliteration: Convert between scripts (Latin to Cyrillic, Hangul to Latin, etc.)
• Normalization: NFC, NFD, NFKC, NFKD Unicode normalization forms
• Character Names: Formal names, name aliases of every type (corrections, control names, abbreviations), and extended names of every code point, and lookup from a name or alias to its character
• Text Segmentation: Break text into words, sentences, lines, or grapheme clusters
• Word shapes: shape provides coarse@1, cased@1, and the approximate orthographic cvletters@1, with uncapped cvletters_counts. It labels vowel letters, not phones: the English table is curated, other alphabets remain L unless caller material supplies labels, and phone-level speakability is out of scope.
• Unicode Regex: Full Unicode-aware regular expressions with script and property support

Sentence breaking

For English locales, Breaker, SentenceOverride, and ik break sentences (as well as ik break tokenize) use ICU sentence candidates with token integrity and the locale-fallback break="suppress" entries from the shipped abbreviation lexicon. This applies to English locales with any region or script except the POSIX variant. Candidate decisions follow ICU, token integrity, caller-before rules, exceptions, an optional base, then caller-after rules. The learned en-tn@1 rule set, en-tn-cart@1 synthetic-text model, and en-real-cart@1 real-text model remain available as opt-in bases. Pass base="none" to either Python API or --base none to the CLI for raw ICU sentence boundaries without the shipped list. Other locale languages keep raw ICU as their default. AbbreviationSentenceBreaker remains the explicit alternative that also deposits ambiguous lexicon boundaries. The TN bases' measurements are agreement with a corpus splitter on synthetic text. The real-text model's receipt reports source-record seam-break measurements and its GUM and Hansard training sources; neither claim is a general sentence-breaking accuracy result.

Recognition

• Value readings: Find numbers, dates, times, measures, currencies, ordinals, and abbreviations in text by inverting ICU's own formatting, keeping every plausible reading of a span
• Language-wide forms: Read the forms every locale of a language writes (en_US text reads en_GB's "12 kilometres"), or only the locales a caller chooses

Readers come assembled in sets. generated_detectors(locale) holds the readers that invert ICU's canonical formatting (each date skeleton, dates with an era among them in ICU's own forms with four-digit years, each interval, compact, spelled-out, and relative form, and ranges of numbers as ICU's NumberRangeFormatter writes them: "3–5", "10–15%", "~3"; a set that reads currencies or measures reads their ranges too, "$3–5", "10–15 kg", through range_detectors); flexible_detectors(locale) holds the flexible readers, which read the forms text writes beyond ICU's own (negative and accounting currency, dates with eras in other layouts, mixed measures, other decimal styles, times with zone names), each reader that takes a currency or unit built for those it chooses from ICU. A unit is read where CLDR's unit preferences give it to a region of the language, and every duration and data-size unit everywhere, so German text's "°F" is not read unless you pass units=. The flexible set costs more than the generated one: on the order of 5 to 10 seconds to build for en_US (a few for a language of few locales), so build it once and reuse it, and each detect a small multiple of the generated set's (the readers share the numbers they read, kept for the 16 texts read last, about 110 bytes per character each for en_US). For every reading, use both: generated_detectors(locale).with_(*flexible_detectors(locale).detectors). Pass guarded=True to flexible_detectors, or build generated_detectors(locale, (*DEFAULT_FAMILIES, *GUARDED_FAMILIES)) (GUARDED_FAMILIES alone builds only the guarded readers), to add the readings the default readers refuse on purpose ("May" alone as a month; a year of under four digits beside an era in ICU's own forms, "Mar 15, 44 BC", which the generated readers refuse; a range of such years, "44–45"; or a detached quote or prime mark that ICU writes attached, such as 84 "). Detached quote and prime readings have their own measure:detached-unit:<unit> type and DETACHED_UNIT_FAMILY; letter unit symbols retain the default measure reader's flexible spacing.

Recognition reads plain text strings. Markdown, rich text, HTML, XML, and other markup must be turned into text before icukit reads it, by the caller or on the client; icukit has no mode for them. It also has no URL or email detector, so readings can fall inside a URL ("2004" in a path).

Localization

• Number Formatting: Decimal, currency, percent, scientific, spelled-out numbers
• Date/Time Formatting: Locale-aware date and time formatting with multiple styles
• Duration Formatting: Human-readable time durations ("2 hours, 30 minutes")
• List Formatting: Locale-aware list formatting ("A, B, and C")
• Plural Rules: Determine plural categories (one, few, many, other) for any locale
• Message Formatting: ICU MessageFormat for complex localized strings

Internationalization Utilities

• Collation: Locale-aware string sorting and comparison
• Locale Information: Parse, validate, and query locale data
• Script Detection: Identify writing scripts in text
• Bidirectional Text: Detect and handle RTL/LTR text
• IDNA: Internationalized domain name encoding/decoding
• Spoof Detection: Detect confusable characters and homograph attacks

Reference Data

• Regions: Country and region codes with containment relationships
• Scripts: Writing system information and properties
• Timezones: Timezone data with offsets and equivalents
• Calendars: Calendar system information (Gregorian, Hebrew, Islamic, etc.)

Quick Start

Library API

from icukit import (
    transliterate,
    sort_strings,
    format_number,
    format_datetime,
    get_plural_category,
    break_words,
)

# Transliterate text between scripts
transliterate("Привет мир", "Russian-Latin/BGN")  # "Privet mir"
transliterate("hello", "Latin-Cyrillic")  # "хелло"

# Sort strings with locale-aware collation
sort_strings(["cafe", "café", "CAFE"], "en_US")  # ['cafe', 'CAFE', 'café']
sort_strings(["Öl", "Ol", "öl"], "de_DE")  # ['Ol', 'öl', 'Öl']

# Format numbers for different locales
format_number(1234567.89, "en_US")  # "1,234,567.89"
format_number(1234567.89, "de_DE")  # "1.234.567,89"
format_number(1234567.89, "hi_IN")  # "12,34,567.89"

# Format dates
from datetime import datetime

now = datetime.now()
format_datetime(now, "en_US", style="LONG")  # "January 19, 2026 at 4:00:00 PM PST"
format_datetime(now, "ja_JP", style="LONG")  # "2026年1月19日 16:00:00 PST"

# Determine plural category
get_plural_category(1, "en")  # "one"
get_plural_category(2, "en")  # "other"
get_plural_category(2, "ru")  # "few"
get_plural_category(5, "ru")  # "many"

# Break text into words
break_words("Hello, world!")  # ['Hello', ',', 'world', '!']

Recognition

from icukit import flexible_detectors, generated_detectors, icu_abbreviations

readers = generated_detectors("en_US")
for d in readers.detect("Due March 5, 2024."):
    print(d["start"], d["end"], d["type"], d["text"])
# 4 17 date:yMMMMd March 5, 2024
# 4 11 date:MMMMd March 5

d = readers.detect("Due March 5, 2024.")[0]
# The value: DateTimeValue(fields=(('y', 2024), ('M', 3), ('d', 5)), calendar='gregorian')
d["value"]
# Each part with its own span: (Capture(name='y', start=13, end=17, text='2024', ...), ...)
d["captures"]
# How to write it again: DateFormatSpec(locale='en_US', skeleton='yMMMMd', pattern='MMMM d, y', ...)
d["spec"]

# Every locale of the language by default; `locales=` chooses them, and () is the locale alone
flexible_detectors("en_US").detect("12 kilometres")  # measure:kilometer and number:decimal
flexible_detectors("en_US", locales=()).detect("12 kilometres")  # number:decimal only

# The short forms ICU writes, with their expansions
icu_abbreviations("en_US", kinds=("unit",))  # (..., IcuAbbreviation(surface='m/s²', ...), ...)
# Symbols by CLDR's names in the language, read from a snapshot of CLDR's annotations
icu_abbreviations("de_DE", kinds=("symbol",))  # (..., IcuAbbreviation(surface='&', ...), ...)

Offsets are code points. date_detectors(locale, skeletons) and number_detectors(locale, currencies=...) build smaller sets by hand.

Command-Line Interface

icukit includes a full-featured CLI accessible via icukit or ik:

# Transliterate text
ik transliterate name Russian-Latin/BGN -t "Москва"
# Output: Moskva

# List transliterators
ik transliterate list --name "Russian-.*"

# Recognize typed values in running text
ik detect --currency USD -t 'Paid $1,234.50 on March 5, 2024'

# Add the flexible readers (accounting currency, mixed measures); --locales '' reads en_US alone
ik detect --flexible --locales '' -t 'Paid ($12.50) for 5 ft 3 in'
# --currency USD also reads the strict "$12.50" inside "($12.50)"
ik detect --flexible --locales '' --currency USD -t 'Paid ($12.50)'

# Format numbers
ik locale format 1234567.89 --locale de_DE
# Output: 1.234.567,89
ik compact 1234567 --locale de_DE
# Output: 1,2 Mio.

# Get locale information
ik locale info en_US

# Sort lines with locale collation
cat names.txt | ik sort --locale sv_SE

# Detect the scripts in text
ik script detect --all -t "Hello Мир 世界"

# Get Unicode character information
ik unicode info -t "Aé"

# Look up a character by its name
ik unicode lookup -t "GREEK SMALL LETTER ALPHA"

From Python: char_from_name("GREEK SMALL LETTER ALPHA") returns "α", char_from_name("NBSP") returns "\xa0", get_char_names("Ƣ")["alias"] returns "LATIN CAPITAL LETTER GHA", and get_char_aliases("\x07") lists ALERT (a control alias) and BEL (an abbreviation).

Run ik help or ik <command> --help for detailed usage information.

Supported Python Versions

• Python 3.11+
• Tested on Linux and macOS

Documentation

• API Reference
• CLI Reference
• Language Availability
• Locale Material
• English Real-Text Sentence Model

License

BSD 2-Clause License

The CLDR data bundled in icukit/data/cldr_symbols is Unicode, Inc.'s, under the Unicode License v3.

The UCD data bundled in icukit/data/ucd_name_aliases is Unicode, Inc.'s, under the Unicode License v3.

The opt-in en-tn@1 sentence rules, en-tn-cart@1 synthetic-text model, and en-real-cart@1 real-text model in icukit/data/break_rules/en are distributed under CC BY-SA 4.0; their bundled NOTICE gives attribution and describes the modifications. The TN artifacts derive from the Google Text Normalization English corpus. The real-text model derives from commercially compatible GUM documents and UK House of Commons Hansard; REAL_TEXT_RECEIPT.json lists the documents, source URLs, licenses, and measurements. The model is distributed under CC BY-SA 4.0 because part of its training text is CC BY-SA; attribution is provided in the bundled NOTICE. No corpus sentence or row is bundled in these artifacts.