Skip to content

add <track> for time-aligned media transcripts - #175

Merged
vagenas merged 5 commits into
mainfrom
add-tracks
Sep 10, 2026
Merged

add <track> for time-aligned media transcripts#175
vagenas merged 5 commits into
mainfrom
add-tracks

Conversation

@vagenas

@vagenas vagenas commented Sep 1, 2026

Copy link
Copy Markdown
Member
  • New <track> semantic element: <bdiv/>-delimited cue blocks, each with a mandatory start time and optional end time, an optional <frame> and <audio>, and a transcript of <voice>-attributed turns
  • Semantics: cue blocks cover inclusive [start, end] intervals (missing end means [start, start]) and are ordered by non-decreasing start time, overlaps allowed
  • Validation: XSD elements/groups + top_level_cat entry + range facets; Schematron rules for cue-block structure, timestamp order, and block sequence
  • Archive: whole-track media in audio/ and video/ ({N}.{ext} by <track> position); pack() and doclang pack gain --audio/--video; media content types added; archive-demo gains a track and a spoken audio/1.wav
  • Docs/tests: spec ### Tracks section, regenerated ### Reference (drops the abandoned <timestamp>/<hour>/… draft rows), token vocab, and track + packaging fixtures

- New `<track>` semantic element: `<bdiv/>`-delimited cue blocks, each with a mandatory start
  time and optional end time, an optional `<frame>` and `<audio>`, and a transcript of
  `<voice>`-attributed turns
- Semantics: cue blocks cover inclusive `[start, end]` intervals (missing end means
  `[start, start]`) and are ordered by non-decreasing start time, overlaps allowed
- Validation: XSD elements/groups + `top_level_cat` entry + range facets; Schematron rules for
  cue-block structure, timestamp order, and block sequence
- Archive: whole-track media in `audio/` and `video/` (`{N}.{ext}` by `<track>` position);
  `pack()` and `doclang pack` gain `--audio`/`--video`; media content types added; archive-demo
  gains a track and a spoken `audio/1.wav`
- Docs/tests: spec `### Tracks` section, regenerated `### Reference` (drops the abandoned
  `<timestamp>`/`<hour>`/… draft rows), token vocab, and track + packaging fixtures

Signed-off-by: Panos Vagenas <[email protected]>
- `<chapter>`: an optional cue-block title, placed after the timestamps and before
  `<frame>`; it marks a chapter boundary at the cue block's start time only, and
  chapters form a flat non-overlapping partition of the timeline (no nesting, no
  explicit end — the next `<chapter>` or `</track>` closes it)
- `<audio>` is now the recording over the cue block's `[start, end]` interval — a
  temporal crop, the way a `<picture>` crop corresponds to its page region;
  timestamps are authoritative, the clip is a best-effort fragment, and its exact
  duration is not checked. `<frame>` stays the still at the start time
- Schematron: `track-audio-requires-end` (a cue block with `<audio>` must carry an
  end time) and `track-chapter-strictly-increasing` (two chapters cannot mark the
  same instant)
- token vocabulary gains `<chapter>` / `</chapter>`; `### Reference` regenerated
  with the `<chapter>` entry and the revised `<frame>`/`<audio>` wording
- tests: chapter fixtures (valid + invalid) and `nok_track_audio_without_end`

Signed-off-by: Panos Vagenas <[email protected]>
Signed-off-by: Panos Vagenas <[email protected]>
@vagenas
vagenas marked this pull request as ready for review September 9, 2026 08:31
PeterStaar-IBM
PeterStaar-IBM previously approved these changes Sep 9, 2026

@PeterStaar-IBM PeterStaar-IBM left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@ceberam ceberam left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a couple of comments below.

And just a reflection on the design choice for <voice>, compared to WebvTT <v> span (not highlighting any defect though and going very fine-grain). In WebVTT, the <v speaker> span wraps the spoken text inline, so speaker and utterance are in the same node. In DocLang's <track>, the <voice> precedes the text as a sibling. This means <voice> has no explicit close: a turn runs until the next <voice> or </bdiv>. The structural intent is the same, but the representation differs and it may create some gaps:

  • no XML structure encodes that boundary: a tree walker (from a consumer application) has to reconstruct it by walking forward from each <voice> until the next one
  • mixed attributed and unattributed content within the same cue could not be captured, like in the 3rd block of WebVTT EXAMPLE 6.
  • word-level timing is not expressible (this is not supported in Docling's WebVTT parser either, by the way), just at cue block interval.

Comment thread doclang/doclang.sch Outdated
Comment thread doclang/tokenization.py Outdated
@vagenas

vagenas commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

@ceberam thanks for your comments

  • no XML structure encodes that boundary: a tree walker (from a consumer application) has to reconstruct it by walking forward from each until the next one

This is consistent with DocLang's delimiter + virtual-text pattern (e.g. in lists, tables)

  • mixed attributed and unattributed content within the same cue could not be captured, like in the 3rd block of WebVTT EXAMPLE 6.

It actually can with an empty <voice/> — I now even refined the explanation in the spec.

@vagenas
vagenas requested a review from ceberam September 10, 2026 14:24
@vagenas

vagenas commented Sep 10, 2026

Copy link
Copy Markdown
Member Author
  • word-level timing is not expressible (this is not supported in Docling's WebVTT parser either, by the way), just at cue block interval.

This can indeed be done via cue blocks (with hardly any information loss), e.g.

00:00:00.030 --> 00:00:02.669 align:start position:0%
the<00:00:00.389> quick<00:00:00.750> brown<00:00:01.110> fox

from #175 could become:

<doclang>
  <track>
    <bdiv/>
    <seconds value="0"/>
    <msecs value="30"/>
    <seconds value="0"/>
    <msecs value="389"/>
    the
    <bdiv/>
    <seconds value="0"/>
    <msecs value="389"/>
    <seconds value="0"/>
    <msecs value="750"/>
    quick
    <bdiv/>
    <seconds value="0"/>
    <msecs value="750"/>
    <seconds value="1"/>
    <msecs value="110"/>
    brown
    <bdiv/>
    <seconds value="1"/>
    <msecs value="110"/>
    <seconds value="2"/>
    <msecs value="669"/>
    fox
  </track>
</doclang>

@vagenas
vagenas merged commit 663e9f5 into main Sep 10, 2026
10 checks passed
@vagenas
vagenas deleted the add-tracks branch September 10, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants