Repository navigation
feat(data): bring-your-own-data — shot-bundle import through a DataSource seam - #64
Merged
Merged
Conversation
…ingestor One named door for every surface: a DataSource protocol (name, MachineConfig, load_shot/load_db/cached_shots), FairMastSource delegating to the mast.py facade (the tests' monkeypatch surface, kept intact), and UserFileSource — npz bundles validated against ShotBundle v1 on every read, identity stamped from the file name, machine.json config overlay that fails loudly on unknown fields, and a validate-then-publish import through fusionlab.atomicio so a rejected upload never leaves a partial file. The derived catalog evaluates each bundle at its peak-current slice with tau_E and beta_t_pct computed by the documented formulas. Part of DATA-2 (#34). FUSIONLAB_DATA_SOURCE selects the default source; FUSIONLAB_USER_BUNDLES points at an imported-bundle directory. Co-authored-by: Tapan Ganatma Nakkina <[email protected]> Human author: Tapan Ganatma Nakkina ([email protected])
Every route, cache and tool now addresses shots by string id — archive
numbers and imported bundle ids ride the same paths. Replay, psi,
field-lines, exports, USD naming and attribution resolve through the
source's machine config; /shots, /db and /db/search serve the configured
source's catalog beside imported bundles.
POST /data/bundles/{id} imports an npz bundle through the user-file
source (validated against ShotBundle v1, published atomically, optional
meta_json); a bare POST on a digit id imports that archive shot through
the cache-then-S3 path. DELETE removes imported bundles only. The
benchmark resolves its shot through the source seam (bench_shot config,
first cached bundle fallback) instead of a hard-coded archive id.
Agent tools take string ids and the action_proposed event now records
the canonical proposal args the confirm applies.
Closes #34 (with the ingestor and docs commits).
Co-authored-by: Tapan Ganatma Nakkina <[email protected]>
Shot ids stay strings across picker, deep links, search selection, compare, retries and exports; the picker refreshes from the source's own /shots list. An import control uploads an npz bundle (or self-imports an uncached archive shot by number) and a delete control removes imported bundles. Co-authored-by: Tapan Ganatma Nakkina <[email protected]>
docs/DATA_CONTRACT.md describes the ShotBundle v1 contract for a second machine: the units table (m, T, MA, 1e20 m^-3, keV, MW, MJ), field shapes and optionality, machine.json, the two import paths, what works without equilibrium, and the no-validation-claims rule (compared with, never "validated" or "predictive"). Repo map gains the data-module row; the env docs cover the data-source selection vars. Co-authored-by: Tapan Ganatma Nakkina <[email protected]>
…id surface New tests: source selection and facade delegation, valid/invalid bundle ids, validated atomic imports (no partial file on rejection), machine config, env-selected directories, derived catalogs, multipart upload through replay/compare/CSV export, deletion, trace-only bundles without equilibrium, malformed and truncated archives, numeric-upload-ID rejection, imported-catalog search, and the benchmark endpoint on an uploaded bundle. Existing pins that froze the pre-DATA-2 surface (int ids, no machine kwarg, raw model args in proposal events) are reframed to the string-id one; the invariants under test are unchanged. Co-authored-by: Tapan Ganatma Nakkina <[email protected]>
The rebase onto the moved main left semantic joins the textual merge missed: imported catalogs now carry the derived mode column (FEAT-4's lh_mode over this machine's isotope and enhancement cut, never MAST's calibration), _archive_flags takes the source and degrades imported ids to flagless (they carry no archive bookkeeping), and the HARN-4 tests that reached into the pre-DATA-2 internals (no-arg _catalog, dict-returning _shot, integer tool-data ids) are reframed to the source-aware string-id surface. Co-authored-by: Tapan Ganatma Nakkina <[email protected]>
The load_shot/limit_distances error hints listed the archive cache's ids while the replay route now serves the configured source's shots — archive plus imported bundles, as string ids. Pin the tool's data to the source's own list so the hint matches what the route can load. Co-authored-by: Tapan Ganatma Nakkina <[email protected]>
Co-authored-by: Tapan Ganatma Nakkina <[email protected]>
obvious-autobuild Bot
added a commit
that referenced
this pull request
Oct 12, 2026
Task-mode IA: Replay / Compare / Explore workspaces behind one hash grammar; shared shot-load state machine; anchored what-if as a Replay drawer; command palette; catalog absorbed into Compare with MJ units and catalog naming; a11y tokens; integrated main (#61 #62 #64) + post-audit fixes (CSV-export error surfacing 02c16c3, mobile compare-bar wrap 73b4048). V1-V10 sweep PASS at 73b4048; V8 capability parity PASS (72 controls, art_A29dHIEm). Co-authored-by: Tapan Ganatma Nakkina <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The spec's external research settled the multi-machine question: FAIR-MAST is the only substantially open tokamak archive, so "support another machine" cannot mean "connect to the cloud" — it has to mean a physicist with AUG/DIII-D/JET access brings their own data. Until now there was no door: every route assumed the MAST loader, the benchmark hard-coded
mast.load_shot(30166), and a bundle without equilibrium broke/replayoutright. This PR builds that door on the ShotBundle v1 contract DATA-1 landed.What
DataSourceseam (fusionlab/data/sources.py): a small protocol (load_shot/load_db/cached_shots/clean_db/import_bundle) with two implementations —fair-mast(delegating through the existingmast.pyfacade, preserving its monkeypatch surface) anduser-file(imported bundles). Source selection:FUSIONLAB_DATA_SOURCE, with the archive as the zero-config default.FUSIONLAB_USER_BUNDLESat) an npz bundle; it is validated in full against ShotBundle v1 before anything is written and published atomically (atomicio) — a malformed upload fails with the first bad field named (B_T: expected (21,) on the t_s base, got ()) and leaves no partial file. An optionalmachine.jsonbeside the bundles configures the imported machine (attribution, Te ranges, USD prim root, campaign pattern, isotope, L/H cut) — defaults are deliberately modest and never borrow MAST's calibrations.AUG-34578) ride the same routes — replay, ψ, field lines, USD export naming, CSV/JSON export, the catalog search, and the agent tools. The benchmark resolves its shot through the seam (bench_shotin the machine config, first cached bundle as fallback) instead of a hard-coded id; harness tasks run on uploaded bundles with no worker-registry changes.POST /data/bundles/{id}(multipart import; a bare POST on a shot number self-imports it through the cache-then-S3 path) andDELETE /data/bundles/{id}(imported bundles only — the archive's cache belongs to the loader).equilibrium_absenthint; imported ids carry no archiveuseful/abortflags and degrade to flagless rather than inventing them./shotslist.docs/DATA_CONTRACT.md— the contract for a second machine: units table (m, T, MA, 1e20 m⁻³, keV, MW, MJ), field shapes and optionality, machine.json reference, what works without equilibrium, and the house rule: quantities are compared with your data, never "validated" or "predictive".Deliberately out of scope: IMAS-netCDF via
omas(a follow-on ingestor — the seam is shaped for it), per-machine model retraining (foreign bundles flag out-of-distribution instead), machine parameters in routes/UI beyond the config.How to Review
fusionlab/data/sources.py— the seam + ingestor + machine config (start here).fusionlab/api_replay.py— the string-id rewrite and the two new routes;_shot()is the one resolver every surface shares.tests/test_sources.py— the DATA-2 acceptance criteria as tests;tests/test_api.py/test_agent_tools.pyshow the pin reframes (integer-id → string-id;_catalog()→ source-aware).docs/DATA_CONTRACT.md— the machine-owner-facing contract.Test Evidence
Bundle import control: replay picker before and after importing a synthetic non-MAST bundle (AUG-34578, foreign geometry + time base) — the imported shot renders beside the archive shots with out-of-distribution shading
AUG-34578, foreign geometry R=1.65 m a=0.5 m κ=1.7, its own time base) imports, appears in/shotsbeside archive numbers, replays with source-aware attribution, exports to CSV with provenance columns, and co-groups with shot 30166 in a compare export; the benchmark ran on the uploaded bundle with no worker changes; malformed upload → HTTP 400 namingB_T; trace-only bundle replays with the hint; deletes clean up. Before/after screenshots attached as QA evidence.Human author: Tapan Ganatma Nakkina ([email protected])
Closes #34
🔗 Obvious Project · 🧵 Obvious Thread · 🔀 Obvious PR Artifact