Capture-doc audit metadata + hub admin maintainer tools (bd-eiku4ymo)#415
Open
cscheid wants to merge 7 commits into
Open
Capture-doc audit metadata + hub admin maintainer tools (bd-eiku4ymo)#415cscheid wants to merge 7 commits into
cscheid wants to merge 7 commits into
Conversation
Capture docs gain a top-level automerge `meta` map beside
content/mimeType/hash: kind ("engine-capture"), schemaVersion (1),
createdAt (RFC 3339 UTC), sourcePath (project-relative), engines.
Everything else about a capture is inside the gzipped payload,
invisible to sync-server audits; this envelope is what `hub admin
scan` (Phase B) reads for inventory and its age gate — without
decompressing anything.
- quarto-hub/resource.rs: CaptureDocMeta, create_capture_document
(+ _at test seam for fabricating old captures), read_capture_meta
(every field optional — future/buggy writers stay inspectable),
and CAPTURE_MIME_TYPE moves here as the single source of truth.
- The three capture writers (quarto-preview eager driver +
re-execute, quarto-hub-provider exec server) stamp the envelope;
the previously-duplicated MIME constants become re-exports.
- Written once at creation, never mutated — no CRDT merge concerns.
Legacy docs (no meta) read back as None and stay valid; readers of
`content` are untouched.
- chrono promoted from dev-dep to dep of quarto-hub (RFC 3339
formatting).
Tests: 4 unit tests on the envelope (schema intact, meta roundtrip,
RFC 3339 createdAt, explicit-timestamp seam, legacy None) + the
eager-capture integration test now asserts the recorded doc carries
meta with the right sourcePath/engines. Plan (incl. the reviewed
admin-tool design for Phases B1-B4):
claude-notes/plans/2026-07-24-capture-meta-and-hub-admin-tools.md
Co-Authored-By: Claude Fable 5 <[email protected]>
quarto-hub/src/admin/classify.rs: pure, storage-free classification of stored automerge docs by ROOT shape (project-index, project-set, text-file, engine-capture, binary-file, unknown) and extraction of the liveness-graph edges (index files map + captures sidecar; project-set keys + entry indexDocIds, automerge:-prefix normalized). Unknown is load-bearing: a future schema classifies as Unknown and the collector's allowlist (EngineCapture only) protects it automatically. Roots = index + project-set docs (their inbound pointers live outside the store). 7 unit tests incl. the future-schema and defensive-reference cases. Co-Authored-By: Claude Fable 5 <[email protected]>
admin/scan.rs: enumerate a samod store, load + classify every doc, compute liveness as a worklist closure from the roots (project indexes + project sets), and emit a versioned camelCase manifest (admin/manifest.rs) of orphaned engine-capture docs past the age gate — with per-doc evidence strings, a full by-kind inventory, and a notCollectible section for unreferenced-but-protected docs. Design discovery, recorded in the module docs: whole-store enumeration must NOT go through Storage::load_range([]) — samod's filesystem adapter splays a key's first component across two path segments and load_range rebuilds keys from raw path components, so an empty-prefix listing returns doc ids split in half (samod itself only ever uses per-doc prefixes, where the mapping round-trips). Enumeration is therefore an explicit backend-specific step (list_doc_ids_filesystem mirrors the splay; the adapter's storage-adapter-id file is skipped structurally), while per-doc chunk loading stays on the adapter and is correct on every backend. Caught by the real-store integration test before it could ever mislead a scan. Tests: 6 unit tests on in-memory stores (orphan detection, age gate, --include-unstamped, unknown/binary protection, project-set liveness closure, JSON roundtrip) + a real-store integration test (new tests/integration/ aggregator): docs created through a live HubContext repo, flushed via Repo::stop, scanned offline through TokioFilesystemStorage — exactly the superseded capture is flagged. Co-Authored-By: Claude Fable 5 <[email protected]>
admin/collect.rs implements the mutating maintainer tools with every damage-minimization gate from the reviewed design: - collect accepts ONLY a scan manifest (version-checked; data-dir match enforced so a manifest from server A can't authorize collection on server B), re-verifies each candidate against CURRENT storage (exists, still capture-kind, still unreferenced, still past the manifest's own age gate), and QUARANTINES — one rename per doc directory into <data_dir>/trash/<batch>/docs/<id>, chunk SHA-256s recorded first, batch.json embedding the authorizing manifest + skips for the audit trail. Never unlinks. Dry-run by default. - restore moves a batch (or subset) back, refusing per-doc on hash mismatch or when the target dir reappeared (doc re-created). - purge is the only unlink in the tool set: whole trash batches, batch-level, older than the retention window; batches without a readable batch.json have no age evidence and are never eligible. - AdminLock: all three ACQUIRE the server's own hub.lock (exclusive fs2 flock) for their duration — a running server makes them refuse, and they symmetrically block a server starting mid-operation. Integration tests (real HubContext-built store): full lifecycle (dry-run tree-fingerprint inertness, quarantine, project still serves live docs, hash-verified restore, retention-gated purge), re-verification skip when the orphan is re-referenced between scan and collect, and refusals for wrong data dir + held lock. Co-Authored-By: Claude Fable 5 <[email protected]>
hub grows a subcommand layer (serve stays the default — the flat CLI
is unchanged): hub admin scan/collect/restore/purge wiring the
Phase B1-B3 machinery, with negative --older-than-days /
--retention-days as explicit gate-disable escape hatches
(allow_hyphen_values). `quarto hub admin` comes along for free.
Runbook: claude-notes/instructions/hub-storage-hygiene.md (pipeline,
safety properties, peer + ephemeral-data-dir caveats).
E2E (recorded in the plan): store built entirely by shipped binaries
(q2 preview + a real re-execute to orphan a capture), then the hub
binary drove live scan -> offline scan -> dry-run -> quarantine ->
restore -> retention-gated purge; output inspected at each step. The
first attempt ran a stale pre-Phase-A q2 whose captures were
genuinely unstamped — scan correctly protected them ("unstamped
(pre-envelope), excluded by default"), live-validating the
legacy-capture gate.
Co-Authored-By: Claude Fable 5 <[email protected]>
cargo xtask verify --skip-hub-build passes (Rust-only change: quarto-hub, quarto-preview, quarto-hub-provider — hub-client/WASM do not depend on quarto-hub). Co-Authored-By: Claude Fable 5 <[email protected]>
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
Conflict: both sides created quarto-hub's tests/integration/main.rs aggregator (this branch for the admin tests, main for the sliding-sessions move of auth_bearer + session_auth + support). Resolved as the union, alphabetized. 391 quarto-hub tests pass on the merged tree; clippy clean. Co-Authored-By: Claude Fable 5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two deliverables (design reviewed in
claude-notes/plans/2026-07-24-capture-meta-and-hub-admin-tools.md):Part A — uncompressed audit metadata on capture docs. Engine-capture binary docs gain a top-level automerge
metamap (kind,schemaVersion,createdAtRFC 3339,sourcePath,engines) besidecontent/mimeType/hash— everything else about a capture is inside the gzipped payload, invisible to sync-server audits. All three capture writers stamp it;CAPTURE_MIME_TYPEconsolidates into quarto-hub as the single source of truth. Legacy docs (nometa) stay valid; readers ofcontentuntouched; written once, never mutated (no CRDT merge concerns).Part B — sync-server maintainer tools (
hub admin …). Motivated by the orphaned-capture lifecycle: every re-execute mints a new capture doc and repoints the sidecar, orphaning the old one forever (and #410 made captures big). The safety model is a pipeline of independent gates — a mistake must survive all of them before bytes are gone:scan(read-only, live-safe): classifies every doc by ROOT shape (project-index / project-set / text-file / binary-file / engine-capture / unknown), computes liveness as a closure from the roots, applies an age gate onmeta.createdAt, and emits a versioned camelCase manifest with per-doc evidence + a full inventory. Unknown shapes and unstamped captures are structurally protected.collect: accepts only a scan manifest (version- and data-dir-checked), re-verifies every candidate against current storage, and quarantines (renames into<data-dir>/trash/<batch>/with per-chunk SHA-256s and the authorizing manifest embedded inbatch.json). Never unlinks. Dry-run by default.restore: hash-verified undo of a batch or named docs.purge: the only unlink in the tool set — whole trash batches past a retention window, dry-run by default.hub.lock(exclusive flock) for their duration: they refuse under a live server, and a server can't start mid-operation. Serve remains the default command — the flathubCLI is unchanged;quarto hub admincomes for free.Operator runbook:
claude-notes/instructions/hub-storage-hygiene.md(pipeline, safety properties, peer/ephemeral-dir caveats).Implementation note worth review: whole-store enumeration deliberately does not use
Storage::load_range([])— samod's filesystem adapter splays a key's first component across two path segments andload_rangerebuilds keys from raw path components, so an empty-prefix listing returns doc ids split in half. Enumeration is an explicit backend-specific step (list_doc_ids_filesystem); per-doc chunk loads stay on the adapter (correct on every backend). Caught by the real-store integration test.Test plan
q2 preview+ a real/api/preview/re-executeto orphan a capture), then thehubbinary drove live scan → offline scan → dry-run → quarantine → restore → purge; output inspected at each step. A first attempt with a stale pre-Part-Aq2produced genuinely unstamped captures — scan correctly refused to collect them, live-validating the legacy-capture gate.cargo xtask verify --skip-hub-buildpasses (Rust-only change; hub-client/WASM don't depend on quarto-hub).Related: bd-qbhp2cvv (#410). Follow-up material surfaced by scan's
notCollectiblereporting (unreferenced non-capture docs) is deferred per design review.🤖 Generated with Claude Code