A split paragraph's first chunk carries its headings once

[Maya-Kid]

When the first block after a heading (a paragraph, or more often a list) is too long for one chunk, its first piece carried the new headings twice, sometimes going over budget.

Problem

Packer::text_unit computes the prefix while the new headings are still in pending, which leaves them out, then moves them into the first piece's body. But the loop recomputed it for every piece, so with pending now empty the first piece's prefix had them too, though the splitter had left room for one copy.

On dev 2293ec9, chunk_with_budget(text, 40) on # Report, Intro., ## Part A and a paragraph of 60 short sentences gives the paragraph a 42-token first chunk:

# Report

## Part A

## Part A

One sentence here. …

Without Intro., both headings repeat: 45 tokens. At the default budget of 300, with 240 sentences, the two shapes give 303 and 306. Extraction sees those headings twice.

Change

• The first piece keeps the prefix computed before its headings moved into its body, as table_unit already does; only continuation pieces recompute it.
• Chunk boundaries, offsets, heading and chunk counts stay the same; only that first chunk's text changes.

Verification

• the_first_chunk_of_a_split_paragraph_carries_its_headings_once chunks the example: the paragraph's first chunk must be # Report plus its source (from ## Part A), its second both headings plus its source. With only the test added, dev 2293ec9 fails it: ## Part A twice, 42 tokens against 40. With the change, the four cases above give 38, 38, 299 and 299 tokens.

Linux x86_64 container, Rust 1.98.1, PostgreSQL 16 + pgvector 0.8.6, Poppler 25.03, DB required, commit 078b8d5 on dev 2293ec9: fmt and clippy clean; cargo test --locked --workspace --no-fail-fast 1562 passed, 1 failed, 6 ignored; a chat timing test timed out under load, then passed with its binary run alone.

Limits

• Documents already ingested keep their chunks until reprocessed.
• Split pieces can still run a token or two over, as continuation pieces did on dev: the splitter's room leaves out the blank lines between prefix, headings and text.

[WaylandYang]

Thank you. The diagnosis is exact: the first piece's prefix was recomputed after its headings had left pending, so they were printed twice and could push the chunk over budget. Keeping the earlier prefix for that piece, as table_unit does, fixes it without moving any boundaries. A possible simplification for later: if the prefix is recomputed after each flush() instead of before it, the first flag and the unreachable!() copy of headings can both go. Merging.