Skip to content

Split large artwork images across multiple messages - #188

Merged
maximmaxim345 merged 14 commits into
mainfrom
134-artwork-chunking
Sep 2, 2026
Merged

Split large artwork images across multiple messages#188
maximmaxim345 merged 14 commits into
mainfrom
134-artwork-chunking

Conversation

@kahrendt

Copy link
Copy Markdown
Contributor

An artwork image larger than one Noise frame (65518 bytes of payload) currently relies on transport fragmentation, which admits no interleaving: the sender must finish the fragmented message before any other frame, so the whole image is an indivisible burst that audio chunks cannot preempt. A player holding only min_buffer_ms of audio can underrun behind it; on a live stream that truncates audio. With BMP gone (#168) the worst case shrank, but photographic JPEG still exceeds one frame beyond roughly 500x500 (the spec's own stream/start example, 800x800 JPEG, is 3-5 frames), and PNG exceeds it at almost any artwork size.

Changes:

  • An image is transferred as a fixed 14-byte announce message carrying the timestamp and the image's total_size (uint32), followed by parts carrying only image data, all on the channel's message type; a flags byte after the type byte distinguishes the message kinds, following the shape Give fragmentation a single ID #172 gave fragmentation
  • The announce carries no image data, so a client can allocate its image buffer before any image bytes arrive and write each part directly into it; the transfer completes when the received data reaches total_size
  • Every artwork message is capped at 65518 bytes after the type byte so it never needs transport fragmentation; any other messages MAY be sent between the messages of a transfer, and transfers on different channels are independent
  • The pending image is now the most recently announced image, from its announce until it becomes current: an announce discards any held pending image, partly received or complete, and the pending image becomes current once its transfer completes and its timestamp is reached
  • A rejected announce (no active stream, or client not available) still starts its transfer, so its parts are rejected rather than treated as malformed
  • The clear message becomes an announce with total_size 0, which completes immediately with no parts
  • A two-byte cancel message (flags bit 1) discards the channel's pending image, partly received or complete, leaving the current image showing; it replaces resend-the-current-image as the cancel recipe in the scheduled-artwork rules, which under chunking would cost a full image retransfer
  • Malformed sequences (a message under 2 bytes or over the frame cap, an announce whose length is not 14 bytes, a cancel longer than 2 bytes, a part with no transfer in flight, data extending past total_size, a nonzero reserved flag bit, both flag bits set) close the connection, matching fragmentation

The pending-image redefinition keeps the newest-wins rule from #135 applying at the announce, as it did at message arrival before chunking, and means a client needs at most two image buffers per channel, receiving the transfer directly into the pending one.

Every image costs one extra small message, including images that fit in a single frame; the uniform format was judged worth ~31 bytes of ciphertext per image to give clients flexibility to transfer directly into a proper sized buffer.

There is no explicit last-part flag; total_size already determines completion, and a redundant end marker would add mismatch states to define.

Every artwork message changes shape: the flags byte shifts the timestamp, the header and image data travel in separate messages, and the empty clear message becomes a bare announce with total_size 0.

Closes #134

Breaking changes:

The wire format for artwork is completely changed. All servers and clients will have to re-implement to handle:

  • the announce binary message
  • the artwork specific split chunks versus using the protocol's generalized noise split
  • handling the pending slot; i.e., servers should now send a cancel message instead of re-sending the current image

An artwork image larger than one Noise frame (65518 bytes of payload)
currently relies on transport fragmentation, which admits no
interleaving: the sender must finish the fragmented message before any
other frame, so the whole image is an indivisible burst that audio
chunks cannot preempt. A player holding only min_buffer_ms of audio
can underrun behind it; on a live stream that truncates audio. With
BMP gone (#168) the worst case shrank, but photographic JPEG still
exceeds one frame beyond roughly 500x500 (the spec's own stream/start
example, 800x800 JPEG, is 3-5 frames), and PNG exceeds it at almost
any artwork size.

Changes:

- An image is transferred as a fixed 14-byte announce message
carrying the timestamp and the image's total_size (uint32), followed
by parts carrying only image data, all on the channel's message type;
a flags byte after the type byte distinguishes the message kinds,
following the shape #172 gave fragmentation
- The announce carries no image data, so a client can allocate its
image buffer before any image bytes arrive and write each part
directly into it; the transfer completes when the received data
reaches total_size
- Every artwork message is capped at 65518 bytes after the type byte
so it never needs transport fragmentation; any other messages MAY be
sent between the messages of a transfer, and transfers on different
channels are independent
- The pending image is now the most recently announced image, from
its announce until it becomes current: an announce discards any held
pending image, partly received or complete, and the pending image
becomes current once its transfer completes and its timestamp is
reached. The newest-wins rule from #135 thus applies at the announce,
as it did at message arrival before chunking, and a client needs at
most two image buffers per channel, receiving the transfer directly
into the pending one
- A rejected announce (no active stream, or client not available)
still starts its transfer, so its parts are rejected rather than
treated as malformed
- The clear message becomes an announce with total_size 0, which
completes immediately with no parts
- A two-byte cancel message (flags bit 1) discards the channel's
pending image, partly received or complete, leaving the current image
showing; it replaces resend-the-current-image as the cancel recipe in
the scheduled-artwork rules, which under chunking would cost a full
image retransfer
- Malformed sequences (a message under 2 bytes or over the frame cap,
an announce whose length is not 14 bytes, a cancel longer than 2
bytes, a part with no transfer in flight, data extending past
total_size, a nonzero reserved flag bit, both flag bits set) close
the connection, matching fragmentation

Every image costs one extra small message, including images that fit
in a single frame; the uniform format was judged worth ~31 bytes of
ciphertext per image.

There is no explicit last-part flag; total_size already determines
completion, and a redundant end marker would add mismatch states to
define.

Every artwork message changes shape: the flags byte shifts the
timestamp, the header and image data travel in separate messages, and
the empty clear message becomes a bare announce with total_size 0.

Out of scope: the fragmentation rules in messaging.md, which stay as
they are, and the metadata/color cancel recipe, which remains
resend-the-current-state since those are single small JSON messages.

Closes #134
Comment thread roles/artwork/v1.md Outdated
Comment thread roles/artwork/v1.md Outdated
Comment thread roles/artwork/v1.md Outdated
Comment thread roles/artwork/v1.md Outdated
Comment thread roles/artwork/v1.md Outdated
Comment thread roles/artwork/v1.md Outdated
Comment thread roles/artwork/v1.md Outdated
Comment thread roles/artwork/v1.md Outdated
Comment thread roles/artwork/v1.md
Comment thread roles/artwork/v1.md Outdated

@maximmaxim345 maximmaxim345 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good now (I have only one last nit, but that's more to give server implementers a hint so they avoid the pitfall this tries to solve).
Thanks @kahrendt!

Comment thread roles/artwork/v1.md
@kahrendt

kahrendt commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

@maximmaxim345 Thanks for the thorough feedback on this one, it improved the PR greatly!

@maximmaxim345
maximmaxim345 merged commit 5bea14e into main Sep 2, 2026
1 check passed
@maximmaxim345
maximmaxim345 deleted the 134-artwork-chunking branch September 2, 2026 06:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Artwork transfers can starve audio: head-of-line blocking on the shared connection

2 participants