Decoder

Decoder is a C11 Unicode library plus a small set of text-processing tools built around it. The core library covers normalization, case mapping, text segmentation, character properties, script/security analysis, emoji helpers, and UTF conversion. The repository also includes CLI tools for PDF/EPUB text extraction, normalization, and OCR-assisted workflows.

Status

This is a pre-1.0 release. The public API may change between minor versions until it stabilises. Specifically:

• Versioning — 0.x series. Breaking changes are possible until 1.0.
• Platforms — tested on Linux and macOS (x86_64 and arm64). Windows and big-endian platforms are not yet supported.
• Thread safety — most read-only property lookups (e.g. decoder_get_script, normalization, case folding) are safe for concurrent use after decoder_init(). Mutable engines such as decoder_script_unification_t are not thread-safe; do not share an instance across threads without external synchronisation.
• Memory ownership — pointers returned by decoder_script_unification_get_script reference data owned by the engine and are invalidated by subsequent add_range calls (which may realloc) and by cleanup. Copy the string if you need a longer lifetime.
• Build flags — -march=native is the default for development speed. For portable distribution builds, override via make ARCH_FLAGS="-march=x86-64-v2 -mtune=generic" (or similar).
• Unicode version — UCD 17.0.0. Updates require regenerating tables via make regenerate-lut.
• Open punch list — see TODO.md for completed work and known follow-ups (fuzz testing, Windows, additional locale tailorings, etc.).

Bug reports and API feedback welcome via GitHub issues.

Scope

• Unicode 17.0 data and conformance-focused algorithms
• UAX #15 normalization: NFC, NFD, NFKC, NFKD
• UAX #21 case mapping and case folding, including full expansions
• UAX #24 / UAX #44 script, block, and character property lookup
• UAX #29 grapheme, word, and sentence segmentation
• UTS #39 confusable, skeleton, and mixed-script checks
• UTF-8 / UTF-16 / UTF-32 validation and conversion

Build

The core library builds with the repository Makefile.

make -j

Main outputs:

• build/libunicode.a
• public headers under include/

The public umbrella header is decoder.h.

Quick Start

#include <decoder.h>
#include <stdio.h>

int main(void) {
    decoder_init();

    const uint32_t text[] = { 'A', 0x0308, 'B' };
    size_t boundaries[8];
    size_t count = 0;

    if (decoder_find_grapheme_boundaries(text, 3, boundaries, 8, &count) == DECODER_SUCCESS) {
        printf("grapheme boundaries: %zu\n", count);
    }

    decoder_cleanup();
    return 0;
}

For full API coverage, see API.md.

Verification

Recommended verification targets:

make test-public-api
make test-conformance-all
make test_simd_optimizations

Static analysis helpers:

make inspect DIR=src/algorithms
make inspect-fixlist DIR=src/core SEVERITIES=error,warning

Benchmarks

Boundary performance notes live in docs/performance.md.

Useful benchmark targets:

make benchmark-boundaries
make benchmark-pdf
make bench-api

Tools

CLI tools live under tools/.

• pdf_normalize: MuPDF-backed PDF/EPUB extraction plus Decoder cleanup and Markdown/text emission
• text_normalize: plain text normalization
• pdf_ocr: Apple Vision OCR fallback for scanned PDFs
• tools/decoder: wrapper script around the toolchain

Tool-specific usage notes are in tools/README.md.

Repository Layout

• include/: public headers
• src/algorithms/: Unicode algorithm implementations
• src/core/: public wrapper layer
• src/api/: high-level API glue
• src/platform/: SIMD/platform support
• generated/: generated Unicode tables
• tests/: public API, conformance, SIMD, and benchmark code

Notes

• The core library is intended to stay small and logic-focused; generated Unicode data lives in generated/.
• The PDF extraction tools have optional external dependencies. pdf_normalize uses MuPDF. pdf_ocr uses Apple Vision on macOS.

License

MIT. See LICENSE.