Verified structures: catalog, gates, transactional attach, and provider export - #150
Open
LiangSu8899 wants to merge 265 commits into
Open
Verified structures: catalog, gates, transactional attach, and provider export#150LiangSu8899 wants to merge 265 commits into
LiangSu8899 wants to merge 265 commits into
Conversation
Introduce the first schedule-layer structure. stage_pipeline specs declare a stage graph with cadence attributes instead of a tensor boundary; their parity ground truth is the host's own eager path under an explicit noise window, so the registry now carries kind/family/stages/conformance fields and region entries are unchanged. vla_tick_pipeline (cond_iter_pipeline family, tick specialization): obs_encode at observation cadence feeding a fixed-shape K-step denoise loop at tick cadence, noise as a SWAP window, condition buffers read-only. Includes the GR00T N1.6 binding with its capture rulings (eager backbone, shallow-copy mutation guard, hoisted noise site).
Same structure declaration as the GR00T N1.6 binding. obs_encode ruled eager: SmolVLM vision embeddings use a boolean-mask scatter that is illegal on a capturing stream (same blocker class as SigLIP2 CPU indexing), so the tick form is eager prefill into static KV buffers plus a captured denoise loop; noise uses the host's native injectable window.
Consumption now mirrors kernels.get_kernel: a single call discovers structure seams in the host (pattern-matched from the module tree, no hand-written binding required), calibrates on the caller's real forward passes, runs per-layer parity gates and family-level accuracy plus net-win gates, transactionally activates only what earns, and returns a Plan with a receipt and exact detach. Calibration entries: calibration="auto" captures during the provided forward(s); a file path loads a prior capture or saves this one, so dataset-scale calibration is a path string. Multi-frame via a sequence of forwards or frames=N. The net-win gate requires a margin (min_speedup, default 1.02x) so measurement noise can never activate a family. The swap machinery module is renamed attach.py -> swap.py to free the attach name for the front door; no behavior change.
The single-site counterpart of attach, with get_kernel ergonomics: pull one structure, bind it to the module you point at, plug it in yourself. bind extracts weights from the module, calibrates on caller-provided real inputs, and self-checks the replacement against the original before handing it back; a part that misses the parity gate raises GateRefused instead of being returned. Pass residual= to gate at the declared structure boundary (residual included) — the same measurement attach uses; without it the check runs on the bare seam, which is strictly harsher. The returned module carries a certification record (worst cos, gate boundary, dims, m profile, variant).
structures.capture(fn, windows=..., reference=...) is the schedule- structure counterpart of the region doors: it warms and records the hot stage into a CUDA graph on a side stream and returns a replayable CapturedStage. Declared windows are the tensors the caller may rewrite between replays (noise, observations, condition buffers); replay reads their current contents in place. Parity against the host's eager path under the same window contents and a margin-gated net-win check run inside the call; a stage that fails either raises CaptureRefused. Replay timing records its events on the replay stream — default-stream events only measure enqueue time and overstate the win by orders of magnitude.
A calibrated single projection (x[M,K] -> y[M,N], optional bias) with epilogue variants (gelu+quant, residual add) and a negotiable input dtype: fp8_static marks a producer-negotiated seam where an upstream norm/adaln producer emits fp8+scale and this region skips its own input quantization — decided at plan level and re-certified at the composed boundary. Elementwise work exists only as epilogue variants here, never as standalone regions. Discovery is qualified (weight-size floor, sibling q/k/v/o grouped into one family), not a blind scan of every nn.Linear.
linear_proj gains its first implementation: the fused BF16-entry FP8 projection (FP8 weights, static per-tensor activation scale, fused input quantization), with work-based qualification measured from standalone preflight — projections whose GEMM cannot amortize the fixed quantization cost are refused at bind time and the host keeps its Linear. Per-calibrated-M buffers are pre-allocated so the hot path is allocation-free under compile and graph capture. Discovery matches sibling q/k/v/o(out) projections as one family per attention block behind a weight-size candidacy floor — never a blind scan of every nn.Linear. The front door records bind-time refusals per layer, and impls now share one process-wide hub loader: importing the same kernel repo twice re-registers its fake ops and raises.
A captured stage's declared windows become frt_model_runtime_v1 boundary windows directly; ports reference them by name. This closes the absorption edge: attach + capture + export takes a torch host to a runtime the FlashRT serving mechanisms (Nexus tick, capsule snapshot/restore) consume exactly like a whitebox-produced one.
Calling the hub loader from forward makes dynamo trace through kernels.get_kernel's version resolution (network calls, inspect.Signature) — 26 graph breaks that fragment the surrounding compiled region, and a fragment boundary can drop host-side constant construction into eager code that is illegal under CUDA graph capture. Binding the op function once at bind time keeps the module a single traceable custom-op call, which is why the FFN implementations (which already did this) never fragmented.
…ication lifecycle
The fourth family, the vision half of the ledger: LayerNorm→FP8 with the affine pair in the kernel, one merged-QKV FP8 GEMM with the bias in the epilogue, dense per-call attention (no cache, no mask — the patch sequence is full and unpadded; FA4 probed at the bound head shape, plain SDPA as the floor), and the output/down projections carry bias and residual in the GEMM epilogue — the native vision form, kernel for kernel. The chain returns in the dtype the host's own encoder returned on the probe: a tower that runs wider than its neighbours must hand back what its neighbours expect.
The earlier fusion knife silently missed — a patch whose match string never matched left the eager float GELU, its clamp, and the scatter quantize in the chain, and a full round of measurements judged an unapplied knife. Landed for real, the prefix pass sheds its entire elementwise middle: the whole-pipeline captured form drops from 102ms to 60ms on the target device, through the production receipt that stood at 75.85.
…ws only The norm producers accept a single style row, so the step tables shrink from one materialised row per token to one per norm — the per-step gather stops moving megabytes. The prefix chain computes on the used-token run alone: pad rows were garbage-in-garbage-out by construction, every consumer masks or discards them, and the padded tail of the output buffer stays zero.
The norm-fed GEMMs ride W4A4 NVFP4 with dynamic block scales (the native decoder's own band); the down projection stays static FP8 — its input already is. Measured on the target device the FP4 form loses to the FP8 chain at this stack's tiny row count (59.2 vs 57.3, the per-call quantise and scale-factor traffic outweighing the weight bytes saved), so the receipt keeps the FP8 winner and the candidate stays what it is: a qualified form the race can revisit when its overheads shrink.
A region family owns a structural identifier and an assembly recipe; the parts every recipe uses — weight packing, the rotate-half layout equivalence, the activation check, cache duck-typing, the attention ladder — had accumulated inside the first family and its siblings imported them across the boundary. They live in one elements module now: families stay thin enough to read as recipes, and two hosts' chains keep assembling the same certified parts instead of drifting copies. Pure move, behaviour identical.
- BANDS table + band element closures (stage_styles/norm_project/ out_project): the chain loop is one form, a precision band is a table row; region candidates generate from the table. - fp4 band: static packed/SF/gate out-buffers sized by the quantizer's own allocator (SF tile padding zeroed once, producers never write it), per-step batched style staging - per-call allocations and SF memsets leave the graph. - region receipts gain a host dimension: structural_signature (stack class, depth, head projection widths) scopes the decision key as region:<family>@<sig>; unscoped keys stay readable so existing receipts keep working. - pi05 denoise lowering: scalar_setitem_fill pin (a python-scalar store into a CUDA tensor becomes an in-graph fill) alongside the resident step schedule.
…chain layout Both chains assemble the same split kernel, so the tower's K rows already carry the adjacent-pair convention the stack's cache expects; writing them straight into the bound consumer's prefix rows is the identity of the consumer's own per-step gather, which therefore leaves the graph. Armed by autobuild only when both ends are bound with agreeing facts (rows, head width, depth), receipt-visible in notes, disarmed on revert.
…revision pin - patch_embed_band: the full-patch conv pair and the fp32 glue linears the host's fidelity policy left behind (modality projector, time/action MLPs) are carried to bf16 in place, with every module boundary cast back so all dtype contracts hold; originals retained for a bit-exact undo. Judged by the captured window's own parity against the stock reference. - hub_kernel FRT_KERNEL_REV_<REPO>: an exact revision outranks version resolution for one repo — the standard switch for bisecting a drift that arrives with a rebuilt artifact.
- adarms fp4_full: the complete native decoder form - all four projections ride NVFP4, the fused GEGLU emits packed FP4 for the down GEMM. A band is a table row plus its element closures; the loop gains a down_project element and stays one form. - prefill_tower gains the band table: the fp4 row is the native encoder preset (FFN pair and attention output on NVFP4 dynamic block scales, QKV stays static FP8). Region candidates generate from the table. - Both rows declare their factual prerequisites (the fused GEGLU producer's bf16 entry, the FP4 GEMM's row band probed at bind) and refuse cleanly until the packages land - receipts stay the judge.
Structural recognition only - a projection is a 2-D weight with a quant_method slot, an expert bank is a w13/w2 pair on whatever child holds it, the head is whatever answers through quant_method.apply. Three engine facts ride in the module so callers stop rediscovering them: seats install between weight load and first trace, band dispatch survives compilation only inside a custom op registered at import, and the head binds as row slabs past the quantize entry's row support. attach_engine returns the swap handle; one detach restores tree, experts and head method together.
The opaque op keeps only what tracing would freeze - the band branch and the bank walk. Softmax, top-k and renormalize move back where inductor can fuse them, and the fake signature follows the op's.
Same structural predicates, same seat machinery as the vLLM door; what is this engine's own rides in the module. The scheduler is a spawned subprocess, so install() writes a sitecustomize hook onto PYTHONPATH - the one carrier that reaches every child - gated on an env flag and inert elsewhere. Dense seating dequantizes FP8 block weights through their own scale in row slabs before packing: a raw-byte cast is finite, right-shaped and wrong, and no bind smoke can see it. The fused-MoE and head surfaces differ from vLLM's and are refused until profiled.
With release, seats attach in half-gigabyte slabs of original bytes and each slab's originals move to the weight store before the next binds - on a card holding a 29GB checkpoint the relief must land while binding continues, not after it. Detach survives as restore-from-store. The vLLM MoE seat declares its band host as serving so consume keeps what the prefill path still runs through.
A producer that accepts a single style row drops both the staging copy and the kernel's full-rows style read; an older producer falls back to the staged form. The judge stays the smoke and the captured parity gate.
The FFN pair rides NVFP4: the LayerNorm producer emits packed FP4, FC1 fuses bias+GELU and emits packed FP4 straight into FC2's bias+residual GEMM - two GEMMs, one producer, zero glue between them. The attention half stays FP8. Candidates generate from the band table.
…es (KeyError a_gu/a_dn)
… weight scales, band smoke floor The floor stays at the calibrated 0.95: an end-to-end judgment run showed deep-layer V at 0.92 compounds to 0.74 action parity through the decoder's ten attention passes - the FP8-era line was load-bearing. The band binds and races only when its smoke clears it.
The probe now carries all calibration thunks with sample boundaries exposed; the prefill chain keeps per-sample statistics and reduces them with the house percentile (accumulate_amax, p99.9) for both the scalar amax sites and the AWQ channel vectors. One sample degenerates to the previous raw max exactly. The layer-subset outlier threshold becomes tunable.
…data provenance, sample-count is recipe-selection not a remedy, controlled-experiment attribution, load-bearing floors
… weight, interleaved GeGLU epilogue GEMM, preset-table layer coverage The vision band packs FC1/FC2 through the padded weight pack (SigLIP's logical 4304 lives at the aligned physical width, so FC1's FP4 output feeds FC2 with zero glue), carries the up-projection AWQ inverse scale on the LayerNorm producer, and writes the residual GEMM into the stream explicitly. The prefill band gains the epilogue_hw P1 rows: gate/up pairwise row-interleaved with the down AWQ 1/s folded into the up rows at pack time, one GEMM writing the down input directly, plain RTN weights, and layer coverage as preset rows mirroring the reference constructor. The mid preset's smoke floor carries its own calibration: coverage-scaled, with the end-to-end parity gate as the judge.
The 17-layer il_hw band's tower smoke of 0.796 judged end-to-end captured-parity 0.99660 PASS - the same neighbourhood as the reference tier's own raw cosine. The old 0.95 floor's failing pair came from the retired merged-GEMM form and does not transfer across forms; the 0.99 end-to-end parity gate stays the judge.
skinny belongs to the decoder's narrow-N band; at the encoder shape the wide schedule element-benches 301us against 484us, and the swap is numerics-neutral (tower smoke and end-to-end parity unchanged).
Signature-probed compat across kernels 0.12 through 0.16: the trust gate is passed explicitly for first-party artifacts where the kwarg exists, an integer-version resolver receives the range's floor when the range string stops parsing, and the pre-semver band falls back to default-revision resolution. One call site, no version strings.
The explicit tier as a worked example: a device-free seat book (per region family, a band ladder in native-correspondence order), lowering pins, the cross-region prefix wire, residual seat assembly over whatever the bound regions leave behind, and the windowed capture protocol. Binding facts decide which rung answers on each box - the same file runs an SM110 board through its NVFP4 chains and an SM120 card through its seat population, with host/library pairing shims that probe signatures instead of version strings.
PI05_VIEWS=3 adds the third visual feature and feeds a real image/wrist/wrist_right observation set from an npz fixture - the native benchmark protocol, never synthetic views.
…iate flag A negotiated producer/consumer pair is one gate unit: the producer class maps into its consumer's judging group, so no arm can hand a host module an FP8 tensor the pair's seat was bound to consume. The vision norm pairing itself now honors negotiate_fp8, and attach forwards the flag.
The explicit seat book now walks the same ladder the receipts drive: the DiT stack binds the fused NVFP4 chain first and the per-seat declarations yield their claim only when the chain answers; the backbone attention interface rides the recorded band decision; and the linear projection family races the host on the calibrated shape before it seats - on one device the separate bias and quantize launches lose to the host's bias-fused GEMM, on another they win, and the same declaration gives each box its answer. The runner learns that a stock baseline is a receipt, not a rerun: FRT_SKIP_STOCK skips its compile and FRT_STOCK_MS carries the archived number. The pi05 vision-key shim casts to the parameter dtype it retargets.
The native correspondence the book was missing: every backbone block norms and quantizes in one step, because the norm output's only consumer is the FP8 FFN. The book now writes that pair out - an FP8-emitting norm producer seated with an FP8-input FFN twin as its direct consumer, seat-to-seat only, verified against the bf16 chain at bind and flipped only where the measured pair is faster. The race prints its verdict either way, so a receipt reader sees every seat the house rule kept as well as every one it flipped.
… whole The norm-to-FFN pairing assumed its premise structurally - the norm's sole consumer is the FFN - and one host disproved it at runtime: a modulation between the two meant the consumer saw bf16 on every call, self-detached after its strikes, and left the producer seated alone, feeding FP8 into a host that cannot read it (teacher-forced 0.974, thirty dead seats). Two mechanisms close both halves. The pairing now verifies the premise before it flips: one probe forward per build checks that each candidate norm's output tensor is the seat's input tensor, and a host that fails identity never pairs - no probe, no fact, no flip. And the guards of a seated pair know each other: one out-of-contract call demotes both seats as a unit, so a producer can never outlive its consumer.
A routed-MoE decode does not amortise with the batch - each token picks its own experts - so the packed bank keeps paying where dense seats stop. Measured against vLLM 0.26 on Thor: 2.10x at batch 1, 2.45x at 4, 2.51x at 8, and 1.65x at 16 when the seat is allowed to serve it. At the old threshold that batch fell back to the host and the whole arm measured 0.98x, since the dense seats alone are worth nothing there.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Structures as specs with references and gates, qualified implementations over Hub kernels, one-call assembly onto a host, and an export path that packages a qualified host as a standard model runtime. Tracks #149.
Usage
One call
forwardis any callable that runs the host once. Nothing else is asked of the host — no module paths, no hooks, no scale plumbing.attachgates before it commits, unit by unit, where a unit is a structure — except that a negotiated FP8 chain is one unit, since the producer emits under a scale the consumer was bound for. Each unit is judged on accuracy, on whether its seams actually ran, and on a net win timed by alternating both arms every round. The union is re-checked before commit; a refusal names the metric or the shape it was measured at, and a whole-host refusal leaves nothing behind.To build a plan without judging it — development, or when you own the gate:
plan.swapsis{module_path: replacement}.plan.notescarries the receipt: what was refused and why, which seams were negotiated as a chain, how calibration was obtained, and anything discovery had to assume.Runtime contract
A bound structure is calibrated for one executable form: a device, an input dtype, a width, and where buffers were preallocated, a row count. Called outside it, a seam runs the retained host module instead and records that it did.
The ledger exists because falling back is numerically exact, so a seam that quietly reverted is invisible to a parity check and to a passing test — it shows up only as latency that is not there. The first fallback per seam warns; a seam that falls back on 32 consecutive calls restores the host module permanently rather than staying on as a claim it no longer meets. Counts are eager-only: inside a compiled or captured region the kernel runs without re-entering Python, which is also why the check costs nothing there.
Refused rather than approximated: training mode, device or dtype migration while attached, loading a state dict into a model whose packed weights came from the old one, and a second thread entering one seam (its stash and scratch buffers are shared across that seam's calls by design).
state_dict()delegates to the retained host module, so saving while attached yields the same schema and the same bytes as the unattached model.Calibration
Three ways in, one axis.
forwardis always "run the host once".samplesis any iterable; with it,forwardtakes one sample.framesdefaults to one, and to the whole ofsampleswhen a sample source is given.The statistic is this repo's own. Within one sample: the max over every call a seam sees — one forward already covers every step of an iterative host. Across samples:
flash_rt/core/calibration.py'saccumulate_amaxat percentile 99.9, so one outlier sample cannot inflate every scale. What is held per sample is one float32 vector of per-point amaxes, never activations — 16 samples cost +0.11 GiB peak on Pi0.5 — so a larger sample source costs time rather than memory. The same helper's dispersion summary and scale-ceiling check run on every calibration and name the layers whose scales sit far above the median.plan.notes["calibration"]records{frames, source, stat, keep_samples}.Precision profiles
scheme=is the one precision entry, on both doors. A profile is aregistered quantisation scheme selected by name:
"auto"(the default)resolves to the fastest profile the device can execute — static FP8 on
FP8-capable hardware,
"none"elsewhere;"none"is the explicitoff-switch, under which fusion structures still attach and every
quantised seam stays at host precision;
"w8a16_decode"and"w4a16_decode"are weight-only INT8 / NVFP4 on the FFN decode band.Quantisation happens at attach time from the host's own weights —
scales and packed formats are derived at bind, activation statistics
come from running the host's forward — and detach restores the original
bit-exactly. Hardware support is not declared in this layer: the loader
reads the arch list each kernel package ships in its own metadata and
refuses a device outside it, by name, before the kernel can fail
somewhere less legible.
Single site, visible
bindextracts weights, calibrates, builds the replacement and self-checks it against the original on those samples. Missing the parity gate raisesGateRefusedrather than returning a bad part.Graph boundary
Every varying input must be a declared window; in-graph RNG never matches eager, so noise is a window rather than a call.
Development bench
Same-process A/B with a baseline re-time. A lever that does not both stay accurate and win latency is refused and recorded.
Catalog
decoder_ffnvision_ffnlinear_projbias/no_bias/fp8_in, each with its own measured work bandqkv_packadaln_producernorm_fusedattention_coredecoder_blockcadence_staticvla_tick_pipelineEach entry is
catalog/<name>/structure.yaml(boundary, weight slots, variants, qualification, gates, evidence) plus a plain-torch reference used as the gate's ground truth.Code
catalog/registry.pydiscover.pyautobuild.pyfrontdoor.pyguard.pyimpls/adapters/swap.pyhandle.pyget(name).bind(...)explicit doorstages.pycapture(...)graph doorrecipe.pyprovider.pyfrt_model_runtime_v1bindings/beta/Hosts
Four independent host stacks, no shared integration path between them:
lerobot/smolvla_basenvidia/GR00T-N1.6-3BtransformersResults
Two numbers per host: how much faster, and how close the output stays. The
baseline says which form of the host it is — eager, or
torch.compile.Output match is same input, compared output, against that host's own
unmodified form.
torch.compile+ CUDA graphgeneratetorch.compilegeneratetorch.compilehostgenerateOutput match is same input, compared output, against that host's own
unmodified form, on twelve inputs that were not in the calibration set.
Median first, then the worst of the twelve. The Pi0.5 latency is given as a
range because single-arm timing on this machine drifts several percent
between runs; a firm number needs the paired method.
Pi0.5 is in the band this repo's own native path measures against an FP32
reference — 0.9996-0.9998 (
docs/calibration.md§10). Getting there neededa defect fixed, and it is the kind this layer exists to surface. The
attention core keeps the prefix — the vision and language keys and values —
packed and rewrites only the suffix per denoise step. Correct within one
observation, and that is exactly what the bind-time check proves. Not
correct across observations: a new image means a new prefix while the packed
region still held whatever calibration captured, so later observations had
attention computed against the wrong frame. That seam is now offered only
when the caller declares it will run the refresh at the observation cadence
(
prefix_cadence=True), which the tick pipeline does; unbound is theaccurate default and costs about 0.09× of the speedup.
The Pi0.5 figure is the assembled form itself — structures, then
torch.compile, then one captured graph with observations enteringthrough static windows — measured on held-out frames. The graph matches
the compiled form bit-for-bit, and a frozen-window negative control
degrades to 0.73, so the windows are complete. The 0.59 worst frame is
not corruption: on that frame the host's own two samples (same frame,
two noise seeds) agree only to 0.78, and the frames where the quantised
arm strays are exactly the frames with the widest natural spread — a
quantisation error far below the mode distance can tip a flow sample
into another mode. Same-input cosine on a multimodal frame measures
which mode was sampled, not fidelity; the arbiter there is task-level
evaluation. Widening calibration noise coverage (one seed to four) does
not move it, and on unimodal frames the worst stays at 0.999.
SmolVLA and Qwen3-8B first measured far lower (0.979 worst 0.854; 82%
worst 63%), and that was the measurement's data, not the mechanism.
SmolVLA — trained on SO100-format robot data — was being calibrated and
judged on frames from a different robot suite through a hand-assembled
camera and state mapping; Qwen3 on short fragments repeated and padded
out to length. Re-measured on each host's real inference distribution —
training-domain SO100 episodes through the checkpoint's own preprocessing
pipeline, natural-corpus token windows with no padding — with a bit-exact
null check before attaching and a clean ledger, Qwen3-8B lands as
tabled and SmolVLA's quantised-seam arm at 0.9987 (worst 0.9935).
The
w4a16_decoderow is the gated form: the per-seam gate admits theseven middle FFN layers whose 4-bit parity clears the value band, and
the end-to-end grade is judged as a language model — teacher-forced
same-token rate, since free-running generations stop being comparable
at the first token that differs. The ledger for that row shows zero
fallbacks, 987 decode calls on the kernel path and 14 prefill calls
dispatched to the host by the declared band. Admitting all 36 layers is
a caller's choice via
floors=: it measures 1.34× at a 93.8%teacher-forced same-token rate, which is the warn band, and the receipt
records it as such.
Calibration and parity data must come from the host's real inference
distribution; out-of-distribution input alone cost 0.02 of cosine and 13
points of token agreement here. The remaining gap to Pi0.5's band is what
static per-tensor W8A8 costs over 180 seams; lifting it is a
quantisation-scheme question (per-channel or mixed width), not a
calibration-set one.
GR00T's figure is measured over held-out episodes with the row-locked
structures (packed QKV, block assembly) left unbound: its prompts are
natively variable-length, so on any length other than the calibrated one
those seams fall back by contract — numerically exact, and the ledger is
what separates that arm from a clean measurement. The full
assembled recipe measured the same way lands at 0.9994 (worst 0.9988) —
with the caveat that on this demo data two different frames' reference
actions already agree at about 0.9996, so the dataset itself sets the
measurement floor there.
The tabled SmolVLA figure is the tick form the latency column describes.
On this host the net-win gate refuses the FP8 region (0.93×), so the
tick form carries no quantised seams and its match is capture
numerics — measured over held-out episodes by refreshing the static
prefix buffers per observation and replaying, with a frozen-buffer
negative control (0.9937, worst 0.739) proving the check can see a stale
prefix. The same-observation replay is bit-identical, which is the replan row;
both latencies are from one run on the same held-out episode.
Figures in earlier revisions of this description were higher because they
were measured on the same input the scales were calibrated on, which reports
how well a fit fits itself.
GR00T's 5.10× is two levers multiplied: the schedule structure takes 57.4
to 15.42 ms, and the region structures take that to 11.26.
The GR00T row is also the reuse statement: the same
vision_ffndefinitionqualifies both a SigLIP encoder tower and a DiT diffusion action head, on a
host stack that is neither transformers nor lerobot.
Earlier region-only results on the LLM hosts, kept as recorded:
decoder_ffn×28decoder_ffn×36decoder_ffn+vision_ffnStructure vs standalone kernel swap
Same Hub kernels, same calibrated scales; the only variable is the composition.
torch.compilemax-autotuneChoosing the parity metric for a host
Whole-tensor cosine over logits is the wrong measure for a language host, and measuring it exposed why. Same bindings, same weights, two prompt lengths:
decoder_ffn×36decoder_ffn×36The two metrics move in opposite directions with length: the aggregate cosine falls while token agreement rises. Aggregated over every position it is dominated by positions that never drive a decision, so it tracks sequence length more than output fidelity. The generation-relevant quantities — the last position, top-1 agreement, per-token KL — hold across both lengths and across structure sets.
The gate therefore selects its metric from the host's output type, read off what the host returns: token agreement and last-position fidelity for a distribution, cosine for a value. Bands are per output kind: value outputs
passat cosine 0.999 /warnfrom 0.995; distribution outputs are judged on token agreement, where a clean static W8A8 with every per-seam gate passing measures 0.95–0.98 on real text — the grade of the quantisation, not damage — sopassfrom 0.95 /warnfrom 0.85. None refuses;floors={...}is the caller turning a number into a hard requirement. The older LLM rows above were scored the aggregate way and are therefore length-dependent; they are kept as recorded rather than restated.Parity belongs to a workload, and has to be measured held out
Static per-tensor scales are calibrated from data, so a parity figure belongs to a workload rather than to a host — and a figure measured on the frame it calibrated on is measuring its own fit. Pi0.5 on LIBERO, calibrated on 8 episodes and evaluated on 12 different ones (action cosine against the unmodified host on each frame, same noise, ledger clean):
The in-sample figure is optimistic by about 0.0026, so held out is the number that means anything. These rows were measured before the attention-prefix cadence fix; the current held-out figure is the Results table's. Parity figures elsewhere in this description that were measured on the calibration frame are labelled as such.
Calibration-set size is settled: redone with the repo's reducer and its stratified sampler at 1, 8 and 64 samples, the held-out figure moves by under 0.0001 cosine and the max deviation by about 2%. What does move it is the data's distribution — the SmolVLA and Qwen3-8B story under Results.
Validation
attach/detachis transactional in both directions: failed resolution leaves the model untouched, anddetachrestores the module tree, any routing an adapter patched, and bit-exact output.provider.pyand passes the same serving adoption/tick acceptance as the native pipelines.tests/test_structures_guard.py).Boundaries
eval()mode. Training, sharded parameters, concurrent use of one attachment, and migration while attached are refused rather than approximated.