Preserve partially understood markup and define durable wire format compatibility #61

Activity

1. HarryCordewener commented on Oct 11, 2026

   

   HarryCordewener

   on Oct 11, 2026 – with ChatGPT Codex Connector

   MemberAuthor

   More actions

   Investigation update: preservation, test coverage, and rollout

   Evidence status: the findings below come from source review at MarkupString 09b995c3 (v2.18.0) and SharpMUSH b8211f16, which pins that package. New runtime reproductions are still pending. Existing CI passing does not verify these cases. No code fixes, migrations, or production-data changes have been made.

   Further source findings

   • Unknown-kind preservation does not cover the whole stored document. The envelope reader does not dispatch on a wire version, defaults missing/non-string t to empty, and can fall back to plain text when p/r is absent or malformed. A hypothetical future structural rename can therefore be accepted and then saved with content removed. An extra v field alone cannot protect against this existing reader.
   • Known kinds can discard new properties. ANSI and HTML reconstruct their known fields. By contrast, unknown kinds retain their raw layer object. These are different compatibility guarantees.
   • Nested layout materialization has additional loss boundaries. An unknown block becomes an empty stack; incomplete layout metadata preserves the rendered-text fallback, but writing reconstructs a tree without the original unknown subtree. Gradient decoding filters stops to IColorMarkup, so an opaque unknown-color layer can disappear at that projection.
   • Default and explicit registry paths need separate tests. LayoutJson.CodecFor consults the passed registry only; a null argument does not fall back to MarkupRegistry.Default for custom block codecs.
   • Registering two historical kinds for one runtime type is not currently sufficient. MarkupRegistry.With removes the prior kind mapping for that type. Historical read aliases or separate read/write codec selection would need an explicit design.

   A test trap worth addressing first

   LayoutMarkup.Equals compares identity and layout metadata, but not the Root tree. MarkupSet interning can consequently reuse an earlier equivalent layer in the same process and hide a lossy reconstructed tree.

   Use literal fixtures with fresh identities and isolated reader/writer processes, and assert nested fields and retained JSON. Neither MarkupText's plain-text equality nor render-equivalence establishes storage fidelity.

   Small source-derived cases to execute:

   • {"v":99,"t":"x","future":true}: current read/write is expected to remove the unrecognized envelope fields.
   • {"v":2,"text":"important"}: hypothetical renamed text field is expected to become empty; this is an illustration, not an actual v2 format.
   • Extra properties on a known kind versus the same properties on an unknown kind.
   • Unknown nested layout blocks, unknown gradient colors, custom blocks with explicit/default/missing registries, malformed covers, and throwing codecs.
   • Historical-package writer → current reader → historical reader, including a fresh-process new → old → new roundtrip.

   SharpMUSH impact is record/path dependent

   Attribute hydration calls the decoder without a contextual recovery boundary; metadata and values are stored separately and values can be lazy. A bad value can fail its read or a scan that reaches it, but this does not establish that the entire LMDB database becomes unopenable.

   Channels, mail, feeds, scene edit history, and object snapshots also contain markup. Scene fallback treats a parse failure as bare text, which should not by itself authorize saving that projection. Future conversion must account for feed byte totals and snapshot digests.

   Established precedents and the smallest fit

   Keeping JSON is a reasonable first choice:

   • .NET extension data retains unknown properties for re-emission. MarkupString's handwritten codecs would need to implement this behavior explicitly and recursively.
   • Protobuf binary unknown fields survive serialization, but JSON conversion and copying only known fields can lose them. Changing serializers does not fix lossy higher-level projections.
   • Avro schema resolution separates writer and reader schemas, but ignores writer fields absent from the reader schema. Read compatibility is not automatically a lossless editing guarantee.
   • Lexical NodeState provides opaque state preservation; its serialization guidance favors additive defaults or new types for incompatible changes. This is a bounded mechanism, not a guarantee for every unknown node.
   • Tiptap documents detecting unsupported content and suppressing updates/collaboration or making the editor read-only. Validation alone should not be treated as unknown-field preservation.
   • tldraw uses schema-bearing snapshots and explicit migration sequences, including deliberate downgrade support where provided.
   • Azure upcasting and Axon versioning show how historical reads can adapt data without rewriting the stored originals. Confluent's compatibility guidance supports reader-first rollout and checking all retained historical versions.

   Proposed implementation sequence

   1. Freeze the contract and reproduce. Treat today's unversioned t/p/r as a legacy format. Commit historical golden payloads with producing package/commit, including existing legacy readers, nested layouts, plugins, Unicode, and malformed cases. Run isolated historical/current package runners.
   2. Harden reads while keeping the writer unchanged. Add a diagnostic read result distinguishing complete, safely opaque, unsupported/partial, and damaged values. Retain original stored bytes in SharpMUSH. Separate “can display” from “can safely rewrite.” Preserve raw content through unrelated metadata updates; reject unsafe edits rather than silently save a fallback.
   3. Add explicit version dispatch and historical readers. Keep wire version distinct from NuGet/API and database schema versions. Dispatch before interpreting fields; unknown/malformed versions must not fall through the legacy parser. Apply this to nested serializer calls too. Extension bags, opaque subtrees, and codec aliases can progressively permit more safe edits.
   4. Deploy readers before enabling new writes. Upgrade server, workers, plugins, and browser consumers. Gate storage and transport writer targets separately. A world minimum-reader policy must be checked before migrations/writes; binaries predating that gate still require deployment-level downgrade prevention.
   5. Convert only when genuinely necessary. Prefer mixed-version records plus historical readers. Any later converter needs raw-field inventory, a verified/pinned native backup and restore drill, dry-run report, revision/hash checks, bounded atomic checkpoints, idempotent resume, preservation of unsupported originals, and explicit accounting/digest handling.

   The smallest safe first release is preservation plus read-only handling for partially understood values. A broad format replacement or an automatic rewrite of the world is not required to get that protection.
2. HarryCordewener

   added 5 commits that reference this issue on Oct 13, 2026

   Never fail a read on a newer wire version or a throwing codec

   ...

   bbbc017

   Keep a layout read from an edited copy from being interned as the ori…

   ...

   20dcde4

   VersionOf and NeedsUpgrade, and the rules an upgrade step keeps

   ...

   bdf95f9

   Count a value written back as another kind as lost

   ...

   a85fe59

   Say whether a stored text can be saved back without loss; cut 2.19.0 (#…

   ...

   e44a412
3. HarryCordewener commented on Oct 13, 2026

   

   HarryCordewener

   on Oct 13, 2026 – with Claude

   MemberAuthor

   More actions

   The library part of this issue is done in #63 (2.19.0):

   • golden payloads from the released 1.0.0, 2.0.0, 2.8.0 and 2.18.0 writers, plus the pre-"k" legacy forms
   • every reproduction case listed here, asserted on text, cover, layer order and preserved raw JSON
   • Inspect with four outcomes (Complete, Opaque, Unsupported, Damaged) and CanRewrite
   • an envelope wire version with a plain-text fallback for newer versions, VersionOf/NeedsUpgrade, and the upgrade-step rules in docs/serialization.md

   Unknown properties on a known kind, and unknown layout blocks, are reported through CanRewrite (false) rather than preserved. That is the "or reject rewrites" branch above.

   Separate pinned package runners are not added. The writer's bytes are unchanged, and the golden fixtures pin every released writer's output, so they cover new-reader/old-writer. Old-reader/new-writer does not arise until a new wire version exists.

   The SharpMUSH side (rewrite checks on edit, copy, snapshot and pose paths, integration tests, and the rollback policy) moves to SharpMUSH/SharpMUSH#1818.

   ---

   Generated by Claude Code
4. HarryCordewener

   closed this as completedon Oct 13, 2026