Skip to content

Shape Indic scripts, with the shaped cluster as the spacing unit - #186

Open
ikashapov wants to merge 1 commit into
nerdy-pro:mainfrom
ikashapov:issue-153-shaped-clusters
Open

ikashapov wants to merge 1 commit into
nerdy-pro:mainfrom
ikashapov:issue-153-shaped-clusters

Conversation

@ikashapov

Copy link
Copy Markdown
Contributor

Shape Indic scripts, with the shaped cluster as the spacing unit

Closes #153.

What

Two halves, in the order the issue names them:

The predicate. needs_shaping gains a second property: the Brahmic
scripts whose OpenType shaping reorders glyphs relative to characters — a
Devanagari prebase matra is stored after its consonant and drawn before it,
which Joining_Type cannot see because those letters never join. The list
(Devanagari, Bengali, Gurmukhi, Gujarati, Oriya, Tamil, Telugu, Kannada,
Malayalam, Sinhala, Khmer, Myanmar) is drawn from the OpenType shaping models
themselves; Thai and Lao are deliberately excluded — their vowels are stored
in visual order, so the cmap walk is already right and their measured widths
must not move. New dependency: unicode-script.

The unit. §17.3.2.35 w:spacing and §17.3.1.13 distribute on a shaped
run now operate on the shaped cluster — HarfBuzz cluster values collected
from Skia's shaper and folded to UAX #29 grapheme starts, so an unreordered
combining mark can never become a spacing unit of its own (Skia reports
character-level clusters; unfolded, distribute opened a gap between ह and
its own matra). One HarfBuzz pass per shaped fragment measures both the
advance and the unit count; the count rides on the fragment so distribution
can count units without a measurer, and the painter groups its own shaped
glyphs by the same values, shifting each visual rank by one spacing step in
the same visual-gap model distribution reserves by.

Standing defects repaired along the way

  • A w:spacing run in Arabic was measured and painted with no spacing at
    all
    (the documented pre-Complex-script shaping: make the spacing unit the shaped cluster so Indic reordering works #153 state).
  • distribute stretched a shaped run's underline, borders and hyperlink rect
    while the painter left its glyphs unspread.
  • §17.3.2.45 w:w scaled the reserved box but not the painted glyph
    positions of a shaped run — glyphs overlapped under expansion and the
    underline overran them.
  • The emergency splitter measured pieces of a shaped word by cmap while the
    painter drew them shaped; for an Indic conjunct the cmap sum runs ~1.8× the
    ligature width. Pieces are now measured through the shaper (SplitMeasure
    trait, implemented by TextMeasurer).

Verification

  • The 28-document corpus renders pixel-identical — the
    Latin/Cyrillic/CJK/Thai fast path is untouched.
  • New fixture test-files/devanagari.docx (built by
    scripts/make_devanagari_fixture.py, OPC-verified) pins: the matra
    reorder, conjunct-safe w:spacing, distribute over Devanagari and Arabic
    lines, and a Latin w:spacing control that must stay off the shaping path.
    Exercised by tests/devanagari_shaping.rs with structural assertions (no
    host font names) and graceful skips for hosts without Devanagari coverage
    (ubuntu CI).
  • The cluster-fold, the shift geometry, the predicate, the spacing formula,
    the shaped split measure and the RTL tie-break each carry a
    mutation-checked test — every deliberate break was caught by exactly the
    tests written for it.
  • Rendering verified visually against LibreOffice; cargo fmt /
    clippy -D warnings / test --all / doc / --no-default-features all
    green.

README coverage table updated (76 ✅ / 12 ⚠️ / 10 ❌; the Indic row flips to
✅, the FAQ gap list drops it), AGENTS.md corpus table and entry-point notes
updated.

@ikashapov
ikashapov force-pushed the issue-153-shaped-clusters branch 6 times, most recently from 603216b to ef48207 Compare September 12, 2026 09:48
@ikashapov
ikashapov force-pushed the issue-153-shaped-clusters branch 2 times, most recently from 137dd33 to e3c277d Compare September 18, 2026 02:49
Closes nerdy-pro#153. Two halves, in the order the issue names them:

The predicate: `needs_shaping` gains a second property — the Brahmic
scripts whose OpenType shaping reorders glyphs relative to characters
(Devanagari's prebase matra is stored after its consonant and drawn
before it), which `Joining_Type` cannot see because those letters never
join. The list is drawn from the shaping models themselves and excludes
Thai and Lao (vowels stored in visual order; their widths must not move).

The unit: §17.3.2.35 `w:spacing` and §17.3.1.13 `distribute` on a shaped
run now operate on the shaped cluster — HarfBuzz cluster values, folded
to UAX nerdy-pro#29 grapheme starts so an unreordered combining mark can never
become a unit of its own. One HarfBuzz pass per shaped fragment measures
the advance and the unit count; the count rides on the fragment for
distribution to read without a measurer, and the painter groups its own
shaped glyphs by the same values, shifting each visual rank by one
spacing step in the same visual-gap model distribution reserves by.

Along the way this repairs four standing defects on shaped runs: a
`w:spacing` run in Arabic was measured and painted with no spacing at
all; `distribute` stretched a shaped run's underline and borders while
the painter left its glyphs unspread; §17.3.2.45 `w:w` scaled the
reserved box but not the painted glyph positions (the painter now scales
shaped positions); and the emergency splitter measured pieces of a
shaped word by cmap while the painter drew them shaped — for an Indic
conjunct the cmap sum runs half again wider than the ligature, so the
splitter now measures pieces through the shaper (`SplitMeasure`).

The 28-document corpus renders pixel-identical (the Latin fast path is
untouched); the new devanagari.docx fixture pins the reorder, the
conjunct-safe spacing, both distribute cases, and the Latin control.
@ikashapov
ikashapov force-pushed the issue-153-shaped-clusters branch from e3c277d to 65739f6 Compare September 22, 2026 09:39
ikashapov added a commit to ikashapov/dxpdf that referenced this pull request Oct 14, 2026
One squashed commit carrying the tree of integration/2026-10-14, which was
previously a chain of ten merges ending at ac5787e (kept locally as
archive/integration-2026-10-14-merges). Base: upstream main 3a5c7ca.

Integrated, each at the head of its open upstream pull request:

- nerdy-pro#252  fix-layout-defects-228-232 (a40d5ce) — issues nerdy-pro#228–nerdy-pro#232: bookmark-only
        paragraphs, nested-table width, list instances over one abstract
        numbering, table width resolution (pct and tcW autofit), column
        balancing and separators
- nerdy-pro#183  issue-150-wmf-svg (6e5aa99) — WMF/EMF and SVG images
- nerdy-pro#184  issue-151-pgborders-docgrid (27204b3) — page borders, document grid
- nerdy-pro#186  issue-153-shaped-clusters (65739f6) — shaped text clusters
- nerdy-pro#187  issue-154-tracked-changes (6982b34) — tracked changes and comments
- nerdy-pro#185  issue-155-smartart-charts (3683223) — SmartArt and charts
- nerdy-pro#256  issue-253-del-instr-text (0656a52) — w:delInstrText tolerated
- nerdy-pro#257  issue-250-image-hyperlinks (cb5ce2b) — a:hlinkClick on pictures
- nerdy-pro#258  issue-188-font-absent-cost (9a63bf0) — no deep catalogue tier before
        the substitution table, no theme script-font preload
- nerdy-pro#259  issue-158-auto-hyphenation (9bfbae2) — w:autoHyphenation

Beyond the union of those branches, the integration itself changed:
- README keeps one row per feature after the merges; the matrix tally is
  recomputed from the rows.
- nerdy-pro#253's tests assert that a deleted field code is never drawn and never run
  text, because nerdy-pro#154 keeps w:del runs instead of dropping them at parse.
- nerdy-pro#250's push_link_over sits after nerdy-pro#151's resolve_line_metrics; nerdy-pro#158's
  HyphenationCtx.measure and rejoin_hyphenated take &dyn SplitMeasure (nerdy-pro#153).

Verified on this exact tree: cargo fmt, clippy and doc clean; cargo test
2977 passed / 0 failed (skipping the capi library freshness guards);
--no-default-features and --features capi build; the twelve-document
harness passes 12/12.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Complex-script shaping: make the spacing unit the shaped cluster so Indic reordering works

1 participant