[ikashapov]

Implement automatic hyphenation (§17.15.1.10)

Closes #158.

What Word does

A document that declares <w:autoHyphenation/> in settings.xml is hyphenated by the consumer: when the next word does not fit on the line, Word cuts it at a point its proofing dictionary allows for the run's language, paints the head with a hyphen and carries the tail to the next line. Three companion settings control it — w:hyphenationZone (§17.15.1.53, default 360 twips; "the amount of whitespace which can be left at the end of a line … before hyphenation should be attempted"), w:consecutiveHyphenLimit (§17.15.1.22, how many lines in a row may end in a hyphen; 0 or omitted is no limit) and w:doNotHyphenateCaps (§17.15.1.27) — and a paragraph exempts itself with w:suppressAutoHyphens (§17.3.1.34), a §17.7.2 cascade property that docDefaults and styles can set too. The language is the run's w:lang (§17.3.2.20), so a German quotation inside an English paragraph is cut as German.

dxpdf parsed none of the four settings and dropped w:lang before it reached a fragment, so a hyphenating document rendered with ragged or over-stretched lines and a different line count from Word.

What this does

Parse → model → resolve. w:autoHyphenation, w:hyphenationZone, w:consecutiveHyphenLimit and w:doNotHyphenateCaps land in HyphenationSettings on DocumentSettings, whose Default carries the spec's omitted values (off, 360 twips, no limit, capitals hyphenated), and are copied onto ResolvedDocument. The zone shares w:defaultTabStop's deserializer, so the universal-measure spellings parse, and 0 is a value — hyphenate on any gap — not an absence (test-files/document_outline.docx declares it).

Language to the fragment. FontProps.hyphenation: Option<HyphenLanguage> is resolved in font_props_from_run from the run's cascaded w:lang/@w:val, beside rtl and for the same reason: it is per run and every text fragment of a run shares it by Rc. An untagged run, or one in a language with no vendored patterns, is never hyphenated — falling back to English was considered and rejected, since it would cut German text at English syllables.

Patterns. hyphenation = "=0.8.4" with default-features = false: Knuth–Liang over the TeX hyph-utf8 pattern sets, pinned exactly because the vendored dictionaries are serialized in that version's bincode schema and the crate's last release is from 2021. No embed_all (82 dictionaries, ~2.8 MB — the #124 wheel-size risk). scripts/make_hyphenation_data.sh, in the shape of make_icu_data.sh, copies four .standard.bincode files out of the pinned crate source into src/i18n/data/hyph/ — en-us, en-gb, de-1996, ru; 385,596 bytes raw, 214,624 deflated — with ATTRIBUTION.md carrying each pattern set's licence (MIT for de-1996 and en-gb, Kuiken's permissive notice for en-us, LPPL 1.2+ for ru). src/i18n/hyphenate.rs maps a tag by primary subtag (every de-* to de-1996, because the crate's only Swiss set is the 1901 orthography; en-GB to the British patterns, every other en to American) and answers opportunities(word, lang) from one OnceLock<Standard> per language.

The fitter. Hyphenation happens in one place, layout::line::fit_lines_in, at the one moment a word is known not to fit. The fitter now works over a Cow<[Fragment]> with a FitWindow (start index and optional line limit) and an optional HyphenationCtx; fit_lines/fit_lines_with_first keep their signatures as wrappers passing a borrowed slice and no context, so a document without the setting takes exactly the old code path and never materialises a copy — pinned by a bit-for-bit test. At overflow, hyphenate_at cuts the word that did not fit only when it is a whole, unshaped, non-footnote-mark word of a run with a dictionary; only when the gap the visible line would leave exceeds the zone (strictly, so zone 0 is "any gap"); only while the consecutive count is under the limit; never an all-caps word under doNotHyphenateCaps; never a token with digits or an interrupted run of letters (don't, hard-nosed). Among the pattern points the longest head whose head + "-" re-measures to fit wins. The head is spliced in place with the new BreakAfter::Hyphenated and its text already ending in U+002D; the tail follows with the word's own break status and hanging trailing space. Emission needs no change: it paints the fragment's own text, and justification already measures from the fragments.

Threading. ParagraphStyle.hyphenation: Option<AutoHyphenation> is Some only when the document setting is on and the paragraph's cascaded w:suppressAutoHyphens is not Some(true); carried on the style so body paragraphs, table cells and split continuations all inherit it. place_paragraph builds the context when a measurer exists; the floats path now fits from an index into the whole vector with max_lines: Some(1) instead of slicing and remapping — identical for every paragraph without the setting, and the only shape that can hold a spliced word.

The continuation re-fit. The remainder of a paragraph split across a page or column is re-placed at the next page's width, and a word cut for the first width may fit whole there; re-fitting the cut fragments would paint hyphen- beside ation on one line. fragments_from_line now runs line::rejoin_hyphenated over the remainder — every Hyphenated head folded back into its tail, re-measured, the tail's break status restored — which is also the argument for a variant on the fragment rather than a flag on the line. The hole was reproduced before the fix by a paragraph flowing from a 66 pt column into a 200 pt one.

What is deliberately left out

Each is stated in a comment at the site that makes the choice.

• The zone's compatibility-mode dependence. §17.15.1.53's definition and its own example read in opposite directions, and Microsoft's VBA documentation says Word 2013+ in native layout (compatibilityMode 15) ignores the zone altogether. This implements the normative reading with the 360-twip default; dxpdf does not parse w:compat, so Word may hyphenate more than this does, never less. test-files/auto-hyphenation-compat15.docx is kept as a probe for the Word render that settles it.
• Word's dictionary versus TeX patterns. Cut points can differ per word, and Word's minimum word length and left/right minima are its own (the patterns' are 2/3 for English, 2/2 for German and Russian, so ei-ne is a legal cut here that Word would not make). The fixtures use compounds both agree on; the README cell says so.
• w:softHyphen is still dropped at parse; a word already cut per cluster by split_oversized_fragments for being wider than its line is not hyphenated (Word would cut it at a syllable); a word glued across a <w:r> boundary, or to a list label's suffix, is left whole; shaped runs (Arabic, Hebrew, RTL) and every script without patterns are never hyphenated; @eastAsia/@bidi tags are not consulted; lines governed by a margin-relative position tab are skipped; the consecutive count restarts on a page continuation; the wheel delta was not measured here (no release build in this environment — the Cargo.toml comment says what to record).
• Languages: four. Extend LANGUAGES in the regen script when a fixture and a caller justify one, the rule make_icu_data.sh already applies.

How it was verified

Every test was written from the clause first and watched fail, and each commit message records what failed; every new test was mutation-checked against the implementation it guards (the commit messages name the mutations and which tests caught them).

• Unit: the four settings elements and their defaults (docx::parse::settings, model::types::settings); the resolve passthrough; the tag map, the crate's documented examples, a German compound's seams, Russian syllables, the minima and the boundary contract, and that every vendored dictionary loads (i18n::hyphenate); the run's cascaded w:lang reaching FontProps; nineteen fitter tests over a fixed-advance measurer and a table of cuts — the zone both ways and its strictness, the visible-width gap, longest head, no fitting head, the consecutive limit and its reset, caps, glued words, untagged/shaped/footnote fragments, punctuation and trailing-space handling, the window, the rejoin, and bit-for-bit identity with no context; the §17.3.1.34 cascade in convert.rs; the paragraph layer end to end with the real patterns, with its no-measurer and no-setting controls; and a section-level narrow→wide column paragraph that reproduced the continuation hole before the fix; and the floats path at the paragraph layer — one line per call, each at its own width, a float narrowing the first three lines — which hyphenates at both widths and never leaves an inserted hyphen mid-line, mutation-checked against max_lines: None at its call site (the fitter's own window test builds the window itself and stays green under that mutation; this one does not).
• Integration (tests/auto_hyphenation.rs) over seven fixtures built by scripts/make_auto_hyphenation_fixtures.py (all pass scripts/verify_docx.py): every cut rejoins to a word of the paragraph and lies at a pattern point of the run's language (the en-US run inside the German paragraph is cut at understand-ing); an inserted hyphen is only ever the last piece of its line; the exempted twin, the 7200-twip zone and the off document insert nothing; no two consecutive lines end in one under limit 1, with the limitless twin as the control that they otherwise do; capitals stay whole only under doNotHyphenateCaps while Kraftfahr-zeug beside them is still cut; the unequal-column paragraph never shows a hyphen mid-line; and document_outline.docx — autoHyphenation on, zone 0, docDefaults suppressAutoHyphens — paints no hyphen its text does not contain (the §17.7.5.2 control).
• Battery: cargo fmt --check, cargo clippy --all-targets -- -D warnings, RUSTDOCFLAGS="-D warnings" cargo doc --no-deps, cargo test --all --no-fail-fast (skipping the two committed-Go-library freshness tests, which need a release build), cargo check --no-default-features, cargo check --features capi — all green; see the branch report for counts.
• Corpus before/after: every test-files/*.docx rendered with the upstream-main debug binary and with this branch's, comparing pdfinfo page counts and pdftotext -layout output. 56 documents (49 pre-existing + the 7 new fixtures): no page count changed anywhere, and the text of all 49 pre-existing documents is byte-identical — among them document_outline.docx (autoHyphenation on with a docDefaults exemption), sample-emoji.docx (autoHyphenation w:val="0") and font_scaling.docx (a hyphenationZone with no autoHyphenation). Text differs in exactly five documents, all new and all carrying <w:autoHyphenation/> with a zone that permits cuts: auto-hyphenation-{base,compat15,limit-1,nocaps,columns}. The two new fixtures that must not hyphenate — zone-wide (a 7200-twip zone, wider than the column) and off — render identically to the reference binary, which hyphenates nothing.