feat: add InternVLA-N1-DualVLN vision-language navigation support - #193
Open
hungho77 wants to merge 48 commits into
Open
feat: add InternVLA-N1-DualVLN vision-language navigation support#193hungho77 wants to merge 48 commits into
hungho77 wants to merge 48 commits into
Conversation
InternVLA-N1-DualVLN pairs a Qwen2.5-VL planner with a diffusion trajectory head. The planner's token output is not what drives navigation: the head is conditioned solely on the hidden states at the n_query trajectory-query positions, normed and projected through cond_projector. Fold both steps into the graph so the engine emits z_latents directly, rather than raw hidden states plus a projector the runtime has to carry and a norm ordering it has to get right. The text decoder itself is stock Qwen2.5-VL, so the subclass adds nothing else. emit_hidden_states supplies the pre-norm residual, which is why the norm is applied explicitly. cond_projector hangs off self.model so the checkpoint keys model.cond_projector.* resolve without a remap; verified bit-exact against the checkpoint rather than trusted, since a missed key loads as random weights and is only logged at debug level. Two details that are not incidental: the projector is built with make_linear, not nn.Linear, so it inherits the backbone dtype and quantization policy; and onnx_export_spec raises the dummy sequence above n_query for the trace, because at the default single-token length the [-n_query:] slice is a no-op and risks being specialized away. n_query and latent_dim ride on ModelConfig, defaulting to 0 so no other model is affected. InternVLA-N1 records n_query at the config root and does not record latent_dim at all, so the latter falls back to the reference implementation's LatentEmbSize. Signed-off-by: hungho77 <[email protected]>
Exporting InternVLA-N1 previously meant first rewriting the checkpoint into a stock Qwen2.5-VL one -- a 15 GB intermediate copy whose only purpose was to make the exporter accept the config. It is not needed: the flat InternVLA config already carries vision_config with model_type qwen2_5_vl, and the tower weights sit under the bare "visual." prefix that build_qwen25_vl_visual already auto-detects. So the tower needs no shim and no new family, only registration: internvla_n1 joins the VLM model types, maps to the qwen2_5_vl visual family, takes the Qwen branch of _get_visual_config, and resolves to qwen2_5_vl for the C++ sidecar so Qwen25VLViTRunner serves it unchanged. Verified on the real checkpoint: export returns 0 for both components, and all 390 visual tensors are byte-identical to an ONNX initializer. That check is not ceremonial -- unmatched keys are dropped at debug level, so a tower that never loaded exports cleanly and emits random weights. Adds tmp/ to .gitignore for local export artifacts. Signed-off-by: hungho77 <[email protected]>
The System-1 diffusion head lives only in InternNav, which pins transformers 4.51 against this repo's 5.14.1 and builds on diffusers, a dependency this repo does not have. Reimplement it here instead, the way models/alpamayo does for its own flow-matching action expert. Only one denoising step is in the graph; the Euler loop stays outside, matching Alpamayo's split. Reimplementing also drops machinery the reference carries but never uses in this configuration: image_rotary_emb is None on every call so there is no RoPE, both attention masks are all-ones so attention is unmasked SDPA, and num_kv_heads equals num_attention_heads so the GQA repeat is the identity. patch_embedder is constructed by the reference and never called -- its two tensors are named and skipped explicitly rather than dropped silently. Two details are load-bearing. attn1 has no output projection: the reference sets it to Identity and reuses attn2.to_out[0] for the summed self+cross result, so there is one projection per block, not two. And the feed-forward gate activation runs in fp32, as FP32SiLU does upstream. The loader refuses a partial load rather than warning. Against the real checkpoint it accounts for all 330 tensors: 328 loaded, 2 dead, 0 unexpected, 0 missing, with ffn inner_dim resolving to 1024. Numerical parity against InternNav is still outstanding and is what decides whether this is correct -- a vendored model that is self-consistent but wrong would pass everything above. Signed-off-by: hungho77 <[email protected]>
…ically
Completes the System-1 port. The memory block turns a window of navigation
frames into the 32 tokens the trajectory expert cross-attends against:
DINOv2 ViT, temporal encoder, concat, QFormer resampler. Only the tower needed
real work -- the encoder and resampler are stock nn.Transformer{Encoder,Decoder}
in the reference.
Both vendored modules are now checked against InternNav on identical inputs in
fp32, which is what decides whether the port is right; matching tensor counts
would not.
traj_dit cosine 1.00000000, max abs diff 0.0 (bit-exact)
memory block cosine 1.00000000, max abs diff 3.1e-06 (fp32 round-off)
Both also export to ONNX cleanly with the fused-MHA fast path disabled, since
aten::_transformer_encoder_layer_fwd has no ONNX symbolic.
Three things about the reference that are easy to get wrong, all pinned in the
code: the tower is DepthAnythingV2's DINOv2 backbone and not its depth head,
which has no weights in this checkpoint at all; input arrives already
ResNet-normalized, so the block must not normalize again; and the positional
embedding is stored for a 37x37 grid and bicubically resampled to 16x16 with a
0.1 offset that looks like a rounding artifact and is load-bearing.
QFormer.visual_proj is in the checkpoint and never called, like patch_embedder
in the expert. Both loaders name what they skip and refuse a partial load
rather than warning, so a tensor that silently failed to land cannot reach an
engine.
Dropping the unused depth head also makes the exported graph smaller than the
reference path's: 129.5 MB against 200 MB, the difference being randomly
initialised DPT weights that never execute.
Signed-off-by: hungho77 <[email protected]>
One command now produces all three graphs from the raw checkpoint: thinker llm/model.onnx + embedding.safetensors visual visual/model.onnx action action/memory.onnx action/traj_dit.onnx System 1 ships two graphs rather than one expert, unlike Alpamayo. Fusing them would be wrong, not just inelegant: the memory block runs once per observation window while the trajectory expert runs once per denoising step, so a single graph would re-encode the frames ten times per trajectory. The trajectory expert's batch is 2 * num_sample_trajs because the reference sampler runs classifier-free guidance -- conditioning is [null, real] and the latents are duplicated. Also fixes the export summary, which assumed every component is a single model.onnx and so reported System 1 as a 0.0 MB path that does not exist. It now lists whatever graphs a component produced, and says so plainly when a component produced none. Signed-off-by: hungho77 <[email protected]>
First piece of the System-1 runtime. Lands under experimental_models/ for the same reason Cosmos3 does: the loop needs its own context memory and CUDA stream to run concurrently with System 2, and llmInferenceRuntime sizes one shared context buffer as max() over components, which assumes they are serialised. Doing that in core would cost every other model memory it does not need. The scheduler ports diffusers' FlowMatchEulerDiscreteScheduler as the reference drives it: set_timesteps(N, sigmas=linspace(1.0, 1/N, N)) with the default shift of 1.0. Under that configuration the schedule is closed-form -- sigma_i = 1 - i/N with a terminal zero, timestep_i = 1000 * sigma_i, and a uniform dt of -1/N. The general scheduler's resolution-dependent time shift and sigma-interpolation path are unreachable here, so they are deliberately not reproduced rather than carried along as code that cannot run. Sigmas are computed from the index rather than accumulated, so the terminal value is exactly zero instead of N roundings away from it. Checked against the diffusers schedule directly: sigmas agree to 3e-8, timesteps to 3e-5, terminal sigma is exactly 0.0, and dt is uniform at -0.1 for N=10. Compiles clean under -Wall -Wextra and through the cmake tree with -DBUILD_EXPERIMENTAL_MODELS=ON. Signed-off-by: hungho77 <[email protected]>
The exported expert took 384-wide features and returned 384-wide features, so anything driving it had to carry action_encoder, the sinusoidal waypoint encoding, and action_decoder itself. Fold all three in: the engine now takes and returns waypoints [batch, waypoints, 3]. That is what makes the C++ loop tractable. Everything folded in is a fixed linear map or a parameter-free encoding, so the runtime is left with control flow only -- duplicate for classifier-free guidance, blend, Euler update -- rather than two GEMMs and a positional encoding it would have to reproduce exactly. Verified bit-exact against the reference loop body on the real checkpoint: cosine 1.00000000, max abs diff 0.0. The engine also came out more accurate than before, 0.999087 -> 0.999964 against fp32, because the output is now in trajectory space after the decoder and that contraction averages out bf16 noise. Note the waypoint encoding is sine-then-cosine while the timestep embedding is cosine-then-sine. Both orderings really are in this model, and swapping either is silent. Signed-off-by: hungho77 <[email protected]>
Everything the System-1 denoising loop does outside the engine is now one
kernel. The engine runs on a doubled batch -- unconditional half first,
conditional second -- so the blend and the Euler update are fused into a single
pass rather than three, and the trajectory state is read and written once per
step instead of three times:
pred = uncond + guidanceScale * (cond - uncond)
latents += dt * pred
A second kernel writes a trajectory batch into both halves of the doubled
layout. The halves are identical by construction: they differ only in their
conditioning, which arrives through z_latents, never through the latents.
Checked against the same arithmetic on the host: max abs diff 1.19e-07 for the
fused step, and the duplicated halves are bit-identical to the source. The dt
comes from the scheduler rather than a literal, so the two cannot drift apart.
Signed-off-by: hungho77 <[email protected]>
Adds the runner and an inference example, closing the loop: load the two engines, encode a frame window, then denoise trajectories with the scheduler and the fused kernel already in place. The runner owns its context memory rather than joining the core runtime's shared pool. That is the whole point of landing here -- the shared pool is sized as a max() over components and so assumes they never overlap, which is exactly what System 1 has to do against System 2. One pool serves both System-1 engines, since within a plan they run one after the other. The example reads its tensors as raw float32 and writes the trajectories back the same way. Generating them internally would have been shorter and would have made the run impossible to compare against a reference bit for bit. Verified against the same loop in Python driving the same engine: cosine 1.00000000, max abs diff 4.5e-07. Getting there turned up a real defect worth naming. The reference truncates a float64 timestep, where 0.7 * 1000 lands just above 700. The same expression in float lands just below, so truncating a float32 sigma gives 699 and shifts one step of the schedule. Trajectories still looked reasonable -- cosine 0.9998, which reads as precision noise rather than an off-by-one. timestepIndexAt now rounds in double, and the loop takes its timestep from there. Signed-off-by: hungho77 <[email protected]>
InternVLA-N1 is an asynchronous dual system: the planner replans at a low rate while the trajectory head runs at a high one, and the head steers on the latest plan available rather than waiting for a fresh one. This adds the shared state that handoff needs, modelled on the reference agent's background thread and three locks. Three properties are the point, and each is checked: System 1 never blocks on System 2. With no plan published, latest() reports that and returns; it does not stall the control loop. Worst observed read under contention was 0.110 ms. A plan is published atomically. 2348 concurrent reads against a writer publishing 4000 distinct plans produced 0 torn reads. A half-written conditioning tensor would yield a plausible trajectory built from two different plans, which is worse than an honestly stale one. Staleness is observable. The head may run on an old plan, but a caller that cannot measure how old cannot enforce a safety bound. Plans replace rather than queue. A backlog is not merely wasteful here: it would steer the robot on plans already superseded. Forced replanning ignores both the cadence and the mode, because the reference routes look-down frames through System 2 unconditionally -- that frame is the one that establishes the pixel goal. This owns synchronization only. What actually lets the two overlap is System 1 holding its own context-memory pool, which landed with the runner. Measured on an idle GPU, 10 steps x 32 samples: the C++ trajectory loop runs in 46.43 ms (21.5 Hz), against 61.8 ms for the same loop in Python and 175.4 ms for the PyTorch reference. Signed-off-by: hungho77 <[email protected]>
Adds the VLA guide, links it from the workflow table and the supported-models list, and teaches the test config that an InternVLA checkpoint is a VLA. The guide records the parts that are specific enough to cost someone a day: System 1 ships two graphs because the memory block runs per observation window while the expert runs per denoising step; the trajectory batch is doubled for classifier-free guidance; the memory block expects frames that are already ResNet-normalized; and FP16 on Thor needs __LUNOWUD=-peep:fc_h_fusion=off, without which TensorRT 10.13 miscompiles the gate/up fusion at batch 1 and the engine emits fluent gibberish. It also states plainly that the planner's text output is not the signal that matters. A checkpoint can produce fluent replies and still be useless for navigation; z_latents is what decides. The VLA index gains a caveat rather than being left to imply otherwise: the other two workflows end in a single executable, and this one does not, because its two systems run at different rates. Signed-off-by: hungho77 <[email protected]>
The asynchronous handoff so far was synchronization only -- correct, but it still left the threading to whoever called it. The reference puts the planner on a Python thread inside the agent, so adopting that would have made this runtime depend on the agent. It belongs here instead. The planner is injected rather than owned. What a plan is -- which frames, which prompt, which engine -- is the deployment's business. What the runtime owes is the thread, the wake-up, and the guarantee that a slow planner cannot stall the trajectory loop. Requests coalesce: one arriving while a plan is in flight replaces the pending one instead of queueing. A backlog of plans is a backlog of stale plans. Measured with a planner 14x slower than the loop, matching the real ratio (646 ms against 46 ms): 60 ticks in 315 ms against an ideal 300 ms 5.1% overhead worst single tick 5.39 ms, planner taking 70 ms 15 replan requests -> 5 planner calls coalesced 46 ticks ran on a stale plan by design 0 torn reads Also corrects an overstatement in the previous commit, which said the separate context pool is what lets the two systems overlap. It is not. Two concurrent trajectory loops take 104 ms each against 47 ms alone -- this GPU has no headroom to overlap them. The pool exists so the two cannot corrupt each other's scratch, which is correctness, not speed. The gain from asynchrony here is latency hiding: System 1 does not wait 646 ms for System 2. The guide now records both, including that running on a stale plan is the design rather than a failure. Signed-off-by: hungho77 <[email protected]>
Renaming the emitted tensor to z_latents read better and broke the runtime. engineExecutor requires every engine I/O tensor to be registry-bound, and the registry knows binding_names::kOutputHiddenStates, so llm_inference and llm_bench refused to load the engine: engine binding 'z_latents' is neither registry-bound nor present in the TensorMap The role of the tensor never changed -- it is still what the next stage consumes -- so the name goes back and the contents are what differ. This went unnoticed because the bridge was verified through a hand-written TensorRT harness rather than the repo's own executables. AGENTS.md is explicit that export, build and inference must all be exercised, in that order; an export that loads under raw TensorRT is not evidence the model runs here. Verified the way it should have been the first time: llm_bench drives the rebuilt engine at 162.67 +/- 1.65 ms for a 1024-token prefill. Signed-off-by: hungho77 <[email protected]>
Replaces the placeholder figures with what llm_bench and the runtime actually report, and corrects a claim the measurements refuted. The guide said the separate context pool is what lets the two systems overlap. It is not: with System 2 running, the trajectory loop drops from 20.7 Hz to 8.2 Hz -- they contend for the GPU. Asynchrony still earns its place, for a different reason. Without it the head stalls outright for the ~646 ms System 2 takes, and 8.2 Hz throughout beats zero followed by a burst. The pool is a correctness property, not a speed one. Adds the NVFP4 build flag, which is easy to skip precisely because skipping it looks like a win: the miscompiled engine runs at 62.3 ms against 72.8 ms for the correct one, since a kernel doing the wrong work does less of it. No PyTorch row in the System-2 table. The figure to hand is a full multi-image VLN step, not a synthetic prefill plus decode, and the two are not comparable. Signed-off-by: hungho77 <[email protected]>
Measured the same way llm_bench measures the engines -- same decoder, same 1024-token input, same 1024-token past KV -- so the rows can be read against each other. The figure that was to hand before, 1631 ms, is a full multi-image agent step and would have compared two different things. PyTorch fp16 328.93 ms prefill 99.35 ms decode TensorRT FP16 150.80 ms 63.97 ms TensorRT FP8 90.17 ms 33.03 ms TensorRT NVFP4 72.83 ms 23.33 ms Stated as a speedup, 2.2x/1.6x for FP16 up to 4.5x/4.3x for NVFP4, with the caveat that this is decoder latency and not the agent's end-to-end step. Signed-off-by: hungho77 <[email protected]>
waitIdle() waited on a condition only the planner signals at the end of a plan. Once stop() had joined the thread, nothing would ever signal it again, so a caller waiting across shutdown blocked forever. stop() now notifies it and the predicate includes the stop flag; a waiter is released immediately. Also records why the timestep upload synchronizes once per denoising step. Two cheaper shapes were tried and both were wrong: binding an offset into a pre-uploaded schedule silently reuses step 0, because TensorRT resolves the address once rather than per enqueue, and it produced a plausible trajectory at cosine 0.614 against the reference rather than an error. A device-to-device copy into the bound buffer fails outright with an invalid argument. The synchronize is load-bearing: the host vector dies at the end of the iteration, and an async copy outliving its source is a use-after-free. It is also not the bottleneck -- the whole loop is 46 ms -- so the comment stays and the optimization does not. Signed-off-by: hungho77 <[email protected]>
System 1 produces the control output. If its rate collapses under a competing load the trajectories arrive too late to steer with, so it is the workload that must not be starved -- System 2 is allowed to take longer instead. Nothing in the runtime expressed that until now: both systems simply queued. Thor offers six stream priorities and supports compute preemption, so the lever exists. makeControlStream() takes the greatest priority available. Measured against a competing trajectory loop: alone 48.2 ms equal priority 94.0 ms competing loop 96.1 ms high priority 73.3 ms competing loop 91.3 ms Priority recovers roughly half of what contention costs, and the competing loop is unaffected within noise. It cannot recover all of it: CUDA preempts between kernels, not inside one, so a long-running kernel still holds the SMs. Also adds --concurrent and --prioritize to the example, which is how the above was measured. An earlier aggregate figure printed by that path was wrong -- the wall clock started before the worker threads loaded their engines -- so only the per-stream timings, taken after warmup inside each thread, are reported. Signed-off-by: hungho77 <[email protected]>
The priority stream was measured against a competing load inside one process, where it recovers 94.0 ms per trajectory down to 73.3 ms. Re-measuring with System 2 as a separate process shows no effect whatsoever -- 121 ms with and without. That is what should have been expected. Stream priorities order streams within a CUDA context; across processes the GPU time-slices between contexts and the priority is invisible. Without MPS there is no arrangement that changes this. The consequence is worth stating where someone deploying will read it: if System 2 runs as its own process, the control stream buys nothing, and the integration is what needs fixing rather than the scheduling. Adds --controlStream to the example so both arrangements can be reproduced. Signed-off-by: hungho77 <[email protected]>
Only z_latents had its shape set. That is enough for the engine we build, whose batch is static, but an engine built with a dynamic batch leaves latents and timestep unresolved as well and enqueue fails rather than falling back to the profile's optimum. Found while building a dynamic-batch engine for an experiment that did not pan out; the fix stands on its own and the experiment is not part of it. Verified unchanged against the static engine: cosine 1.00000012 on the same inputs. Signed-off-by: hungho77 <[email protected]>
Quantizing the planner produces a System-2-only InternVLA checkpoint, which is a legitimate thing to export. Until now the action stage tried anyway and died inside the weight loader with a traceback that pointed at the loader rather than at what the user should do. It now reports that the checkpoint carries no traj_dit weights and names the two ways forward: --skip-action, or --components thinker,visual with System 1 exported from the full checkpoint. Found while re-enabling the z_latents bridge on quantized engines. The quantized checkpoints were repackaged to plain qwen2_5_vl, so their engines emitted logits only and the bridge could not be measured at all. Restoring it does not need a fresh quantization pass -- the bridge is not quantized, so copying cond_projector and latent_queries across and restoring model_type and n_query is enough. Both FP8 and NVFP4 now export and build with hidden_states. Signed-off-by: hungho77 <[email protected]>
Folding final_norm and cond_projector into the graph made the engine emit
z_latents directly, which read well and was wrong. The runtime sizes its
hidden-states buffer {batch, maxInputLen, hiddenSize} and reshapes it per
request, so a graph emitting [batch, n_query, 768] violates that contract.
It does not fail loudly. getBaseModelHiddenStates returned a plausible
[1, 45, 3584] buffer from an engine whose actual output was [1, 4, 768] --
type-valid, contract-invalid, silent. That is the third defect of this shape in
this work, after the renamed binding and the reused timestep row, and the
pattern is the same each time: a cheap check passed and the real one was not
run.
The engine now emits the full-sequence, model-width hidden states, and the
projector runs on the host as the reference does. cond_projector and
latent_queries ship as bridge.safetensors next to the engine, since a consumer
that has neither the projector in the graph nor the weights on disk has nothing
to project with.
This gives up the self-contained engine that motivated the original design. The
runtime's contract is real and shared with llm_inference and Qwen3-Omni; the
convenience was mine.
Also adds the dump tool used to find this, including the loadEdgellmPluginLib
call whose absence reports a missing plugin creator rather than a missing
plugin library.
Signed-off-by: hungho77 <[email protected]>
The dump tool copied the runtime's hidden states out as float32. The runtime allocates them as __half, so the copy read twice the bytes it should and produced an array that was half zeros and half values around 1e15 -- shaped correctly, entirely wrong. It now reads getDataType() and converts to float32 on the way out, so consumers need not care what the engine used. With that fixed the bridge measures end to end for the first time on this branch, through the runtime rather than a hand-written harness: hidden states vs PyTorch 0.999908 over the full 45-token prefill z_latents FP16 0.999923 z_latents FP8 0.991430 z_latents NVFP4 0.974353 FP8 clears the 0.99 gate. NVFP4 does not, which is the number that decides it is unsuitable for navigation despite being the smallest and fastest engine. Restoring the bridge on the quantized checkpoints needed no fresh quantization pass: the bridge is never quantized, so copying cond_projector and latent_queries across and restoring model_type and n_query was enough. Signed-off-by: hungho77 <[email protected]>
Runs the repository's own hooks over this feature's files for the first time -- they could not be installed while the network was proxied. yapf rewrapped a few imports and signatures; nothing behavioural changed, and flake8 and the import check both still pass. Reformatting of files this feature does not touch was reverted. clang-format wanted to drop a blank line in nine pre-existing sources under cpp/ and unittests/; that is a real cleanup but it belongs to its own change, not to a model-support PR. Also exports InternVLAN1TrajDitStep and build_internvla_n1_traj_dit_step from the package. Both were added to the module's __all__ when the projections were folded in, but never to __init__, so the package exposed the inner expert and not the thing that is actually exported to ONNX. export_encoder imports from the module directly, which is why nothing failed. Adds .venv/ to .gitignore alongside tmp/. Signed-off-by: hungho77 <[email protected]>
All nine hooks now pass. The remaining formatting is what isort, yapf, clang-format and cmake-format wanted; none of it changes behaviour. codespell read the member mIdle as a misspelling of 'middle'. Renamed it and its counterpart to mIdleCv and mWakeCv rather than adding a word to the shared --ignore-words-list: the suppression would be permanent and repo-wide to accommodate one local name, and the Cv suffix says what the members are anyway. Rechecked after the rename -- the experimental target builds, the handoff test still reports no torn reads, and waitIdle is still released on stop(). Signed-off-by: hungho77 <[email protected]>
.venv/ matters: uv builds it in the repo and it is several gigabytes.
Note for anyone running the hooks here: pre-commit run --all-files fails on
clang-format against nine pre-existing sources under cpp/ and unittests/ that
this branch does not touch. That is the repo's own formatting drift, not this
feature's, and reformatting them here would bury a model-support change under
unrelated churn. Run the hooks over the changed files instead:
pre-commit run --files $(git diff --name-only main..HEAD)
All nine hooks pass that way.
Signed-off-by: hungho77 <[email protected]>
uv writes .venv/.gitignore containing '*', so the venv excludes itself and no entry is needed. tmp/ went with the local artifacts when they moved outside the repo. Worth recording since it looks contradictory: git check-ignore reports .venv as not ignored -- venv*/ on line 30 does not match a leading dot -- while git status stays clean, because the exclusion happens inside the directory rather than through a pattern here. Signed-off-by: hungho77 <[email protected]>
Until now System 1 read its conditioning from a file and System 2 was only reachable through a separate dump tool, so nothing exercised the asynchronous design end to end. This adds the binary that does: System 2 plans on the driver's thread while System 1 keeps sampling from the newest plan available. One process is not incidental. CUDA orders streams within a context, so across processes the priority stream is invisible and the GPU simply time-slices -- System 1's priority only means anything when the two share a context. The bridge runs on the host, as the reference does: the engine emits model-width hidden states and the final norm plus cond_projector turn the trajectory-query rows into z_latents. Four rows of arithmetic, so a plain loop costs nothing against a 623 ms plan and stays easy to check. Measured, 40 ticks at a replan every 4: first plan 623 ms ticks run 40, none stalled mean tick 166.75 ms (6.0 Hz) plans completed 11 final staleness 3 observations Two things this shook out. The loop must wait for the first plan: without it, 40 ticks complete in microseconds while System 2 is still on its first plan, and the run measures the empty case. And LLMGenerationRequest leaves temperature, topP and topK without default initializers -- uninitialized topK made the sampler size its workspace from garbage, surfacing as a size_t underflow reported as an 18-exabyte allocation. Signed-off-by: hungho77 <[email protected]>
The System-2 bridge runs outside the graph, so its weights have to travel with the engine -- a consumer holding only the engine directory cannot reconstruct them, and telling users to copy a file by hand is a step that will be missed. llm_build now copies bridge.safetensors when the export produced one, next to where it already copies embedding.safetensors, and ignores it when absent. It is keyed on the file existing rather than on a model type, so the builder needs no per-model knowledge and no other model sees a warning. Verified on a real build: the engine directory now contains the sidecar without any manual step. Also drops the bf16 suffix from the System-1 engine names. It described the build rather than the contents, and would quietly become a lie if the precision ever changed; the runner and the guide now agree on memory.engine and traj_dit.engine. Signed-off-by: hungho77 <[email protected]>
The loop copied the plan out of the shared state, allocated a device tensor and transferred it on every tick, to move bytes that change only when a new plan lands -- at most once every cadence ticks. It now checks the staleness scalar under the lock, fetches the plan only when it is one it has not uploaded, and reuses the device tensor. Uploads fall from 40 to 9 over 40 ticks. Worst tick improves 328 to 248 ms. The mean barely moves, 166.75 to 164.84 ms, which is the useful part of the measurement: at 221 KB the transfer was never the bottleneck, and the cost of splitting the systems across host and device is about 2 ms against a 165 ms tick. What limits the rate is GPU contention between the two systems, as the standalone measurements already showed. Signed-off-by: hungho77 <[email protected]>
…l signal The example reads the last four prompt-token positions as the trajectory queries. The model appends latent_queries -- four learned embeddings -- and reads the hidden states there instead. Measured on the same prompt, the substitution gives cosine 0.5877 and rel-L2 0.96 against the reference: a different signal, not a degraded one, so the trajectories are not navigation output. It went unnoticed because nothing about it looks wrong. The shapes match, the projection is exact, the loop runs, and the trajectories are plausible. handleRequest takes messages and does the embedding lookup internally, so the queries cannot be injected through it; fixing this needs an entry point that accepts inputs_embeds, which is a runtime API change and out of scope here. The example still earns its place: it establishes the plumbing this feature is about -- both engines in one process, the planner on its own thread, the priority stream, and a handoff that never blocks the control loop. The warning now says which of those two things a reader is looking at. Signed-off-by: hungho77 <[email protected]>
maxGenerateLength has no default initializer, so the planner ran on whatever was on the stack and generated a full reply. The bridge reads the prefill hidden states; the tokens after that are discarded. Setting it to 1 makes the planner do only the work the bridge needs: first plan 623 ms -> 152 ms mean tick 164.84 ms -> 67.20 ms (6.1 Hz -> 14.9 Hz) 14.9 Hz lines up with the 14.4 Hz measured earlier for an in-process System 1 on a priority stream against a competing load, which is the cross-check that the number is real rather than an artifact. Third uninitialised field in this struct after topK, topP and temperature. Reading it uninitialised is undefined behaviour that happens to look like a slow planner rather than a fault. Signed-off-by: hungho77 <[email protected]>
…graph The dual-system example was producing its conditioning from the wrong positions: the reference appends four learned latent_queries to the prompt and reads the hidden states there, but handleRequest performs its own embedding lookup, so the example fell back to the last four prompt tokens. Measured against the reference that substitution is cosine 0.5877 -- a different signal, not a degraded one. The way in is the route Alpamayo already uses for trajectory tokens: make the queries real tokens. The export writes them into the embedding table's trailing padding rows -- Qwen2.5 pads its table well past the last used ID (151664 used, 152064 rows) and the padding rows are zero -- registers <|latent_q0..3|> as special tokens, and records the IDs in config.json. A prompt ending with those tokens puts the queries into the sequence through the runtime's ordinary lookup. No runtime change, no API change. With the positions now correct, the final norm and cond_projector return to the graph, so the engine emits z_latents directly and the bridge sidecar and host-side projection are gone -- including the copy hook llm_build carried for the sidecar. Consumers read the first n_query * latent_dim elements of the hidden-states buffer; the buffer's reported shape is the runtime's model-width copy convention, documented in the guide and both examples. The queries sit after the assistant generation prompt, where the model was trained to find them, so the examples assemble the ChatML themselves and turn the request's template off. Verified end to end on the rebuilt FP16 engine, engine against PyTorch reference on the same prompt: z_latents cosine 0.999978 (rel-L2 0.66%) prefill 45 -> 49 tokens (the four latent tokens, via tokenizer) dual-system loop first plan 144 ms, 14.4 Hz over 40 ticks, none stalled The dump tool also gains the sampling-field initializers the dual example already needed; uninitialized topK sizes the sampler workspace from stack garbage. Signed-off-by: hungho77 <[email protected]>
The example writes trajectories now (--output, last tick), and comparing them against the reference exposed a silent parameter bug: it left the runner's Config defaults in place, and the Config default for guidanceScale is the reference generate_traj default of 1.0 -- but deployment, the standalone example, and every benchmark in this repo run 1.5. Nothing about 1.0 looks wrong from the outside; the trajectories are plausible and the loop runs at full rate. Only the reference comparison shows it: cosine 0.9715 and a mean waypoint deviation of 61% of a waypoint's reach. The example now exposes --numTrajs/--steps/--guidance like the standalone one, defaulting to deployment values. Isolation that found it, same conditioning fed to both samplers: engine memory vs reference 0.999980 python sampler, engine vs ref cond 0.999396 (conditioning precision is fine) python sampler vs C++ chain 0.972728 (the gap was here) After the fix, end to end -- engine z_latents, engine memory, engine denoise loop against the fp32 InternNav reference on identical inputs: trajectory cosine 0.997001 mean waypoint deviation 7.6% of a waypoint's reach (was 61%) Signed-off-by: hungho77 <[email protected]>
The previous commit set the dual-system example's default guidance to 1.5 and claimed that is the deployment value. Checking the call sites shows it is not: every generate_traj call in InternNav -- the realworld agent, the policy, and both habitat evaluator paths -- leaves guidance_scale at its default of 1.0. The 1.5 came from this repo's own earlier benchmarks and had propagated into the claim unexamined. Both sides of the reference comparison re-run at 1.0 (the guidance value does not affect latency; those numbers stand): FP16 trajectory 0.997769 deviation 9.6% of reach mean-traj 0.999900 FP8 trajectory 0.909049 deviation 92.8% mean-traj 0.989103 NVFP4 trajectory 0.954183 deviation 53.7% mean-traj 0.997287 The qualitative picture is unchanged -- FP16 faithful, FP8 worst despite the better bridge cosine, NVFP4 in between -- so the conclusion that z-cosine is not a sufficient acceptance metric stands. At 1.0 the classifier-free blend reduces to the conditioned branch, so the null half of the conditioning costs compute but cannot change the output; the guide notes this. Signed-off-by: hungho77 <[email protected]>
The driver already counts completed plans; the example kept its own atomic doing the same job. One counter, owned by the component that does the work. Signed-off-by: hungho77 <[email protected]>
Follows the shape of nemotron3_5_asr/README.md: what lives in the main package versus this directory, why the runtime is separate, layout, and a quickstart covering export, both builders, the two trtexec graphs, and both inference CLIs. The experimental_models index gains the InternVLA-N1 row. The two conventions a consumer cannot discover from the code -- the hidden-states prefix read and the one-process requirement for stream priority -- are stated here as well as in the guide. Signed-off-by: hungho77 <[email protected]>
With the projection folded into the graph the tool's old output -- the whole model-width hidden buffer -- was 3072 valid floats followed by 172,544 of garbage tail, and every consumer had to know to trim it. It now writes exactly z_latents [4, 768] as float32 and says so. Verified against the reference: cosine 0.999978, unchanged. Both examples stay, with distinct jobs: dump_bridge verifies the System-2 bridge without loading System 1 at all, and system1_inference drives the trajectory head from a conditioning file without loading the 14 GB LLM -- reproducible parity checks for each half, where the dual-system binary needs everything loaded and computes its conditioning live. Signed-off-by: hungho77 <[email protected]>
NVFP4LinearMethod registers weight, weight_scale, weight_scale_2 and input_scale, but no pre_quant_scale. ModelOpt's NVFP4 AWQ configs emit one per input channel, so the loader finds no buffer to write it into and discards it without a warning: the GEMM then runs on unsmoothed activations against weights that were quantized assuming the smoothing had been applied. The result is not a degraded layer, it is a wrong one. On Qwen2.5-VL-7B the scale spans [0.1553, 6.4375] across the 3584 input channels -- a 40x spread -- and an NVFP4 AWQ engine built from such a checkpoint returns a bridge tensor at cosine 0.2505 against the PyTorch reference, i.e. very nearly orthogonal. The multiply now happens in both apply() and apply_linear_allreduce(), gated on a flag that a new repacking pass sets by inspecting the loaded buffer. A checkpoint without a pre_quant_scale leaves the buffer at ones and the flag at False, so the multiply is not traced into its graph and the ONNX for a plain NVFP4 model is byte-identical to before. Detection is by value rather than by config because the quantization config records the algorithm but not which layers were smoothed, and excluded layers legitimately carry none. Verified both directions at the unit level: no scale in the checkpoint leaves the flag off; a non-unit scale turns it on and is cast to fp16. Signed-off-by: hungho77 <[email protected]>
The note described a TensorRT 10.13 Myelin miscompile that llm_build already works around, so it told a reader to check for a flag they never have to set, about a defect this branch does not touch and does not carry code for. The NVFP4 CASK note stays -- that workaround is not applied anywhere in the tree and still has to be exported by hand -- and now says plainly that FP16 and FP8 need nothing, which is the only part of the removed note a reader wanted. Signed-off-by: hungho77 <[email protected]>
The note is about NVFP4; saying which other precisions are unaffected invites the reader to wonder why that needed saying. The flag is documented where it applies and nowhere else. Signed-off-by: hungho77 <[email protected]>
… model width A model may project its hidden states before emitting them. InternVLA-N1 folds its final norm and cond_projector into the graph and emits a 768-wide bridge tensor, but the runtime allocated and copied `hidden_states` at model width (3584) in every case, taking the dimension from config rather than from what the engine actually produces. Nothing failed loudly. `getBaseModelHiddenStates` returned a plausible [1, seq, 3584] buffer whose first n_query * latent_dim elements were the real values and whose remaining 98% was whatever sat next in memory, and every consumer had to know to read only the prefix. The copy also moved 322 KB per request where 6 KB was valid, reading past the engine's output. The width now comes from an optional `output_hidden_size` config key that defaults to `hidden_size`, so a model emitting hidden states at model width is unaffected -- the concept already existed for spec-decode, where EAGLE-3 emits `hidden_size * 3`; this extends it to the plain path. Input embeddings keep their own width, since only the output side is projected. The InternVLA-N1 export writes the key from `latent_dim`. Verified on the same engine, config the only difference: the reported shape becomes [1, 49, 768] instead of [1, 49, 3584], and the values are bit-identical (cosine 0.999978 against the PyTorch reference, unchanged). Also registers internvla_n1_s2_server as a CMake target: a long-lived System-2 generation server, one JSON request per stdin line. Closed-loop navigation asks System 2 for a decision at every simulator step, so per-step process spawning would reload a 14 GB engine tens of thousands of times over a benchmark run. Signed-off-by: hungho77 <[email protected]>
internvla_n1_dump_bridge wrote z_latents, not a bridge. The engine folds the norm and cond_projector, so what comes out is the projected tensor itself, and internvla_n1_dump_z_latents says that without a reader having to know which of the two the word 'bridge' meant. Also drops the README's prefix-read convention, which the output_hidden_size change made obsolete: the runtime now sizes the buffer from what the engine emits, so the reported shape is the real one. Signed-off-by: hungho77 <[email protected]>
An InternVLA-N1 checkpoint declares model_type "internvla_n1" and ships neither modeling code nor an auto_map, so every AutoModel* factory fails on it and tensorrt-edgellm-quantize could not reach it at all. Underneath the declaration System 2 is a stock Qwen2.5-VL -- visual.blocks.*, visual.merger.*, model.layers.* in the standard layout, with vision_config.model_type already reading qwen2_5_vl -- so the loader presents it as one through a directory of symlinks whose config.json says so, the same move _prepare_alpamayo_visual_params makes for Alpamayo's tower. This follows qwen3_asr_loader, which handles the identical situation for a checkpoint that declares a model_type nothing can load. The remaining tensors are System 1 and the System-2 -> System-1 bridge. They are not part of the graph being quantized, so the loader ignores them and restore_system1_tensors copies them into the export afterwards, along with the model_type the exporter dispatches on. Both halves of that restore matter and both fail silently: without the tensors, tensorrt-edgellm-export matches no keys for those modules, leaves them default-initialised and still exits 0; without the model_type, the export is dispatched as a plain VLM and loses System 1 again. The bridge is kept at the source dtype rather than quantized. It is four rows through a Linear/GELU/Linear, so quantizing it saves nothing measurable and would put error directly on the tensor System 1 steers by. Verified end to end on the released 15 GB checkpoint: 1323 quantizers placed, NVFP4 export written, 609 System-1 tensors restored, and the exported config.json reads model_type internvla_n1 with cond_projector and latent_queries present. Signed-off-by: hungho77 <[email protected]>
Six binaries had accumulated, several of them overlapping. What ships now is two, matching the scale of the other experimental models: internvla_n1_dual_system_inference run and time both systems on file tensors internvla_n1_dual_system_server the same, resident, for an agent's loop internvla_n1_dual_system_server is new. dual_system_inference encodes its observation window once at startup, which demonstrates the asynchronous contract but cannot drive a simulator; the server takes the fresh frames an agent produces at every step. Its three request kinds cover what the removed separate servers did -- text for a System-2 decision, replan to wake the planner, trajectory for System 1 against whatever plan is current -- and it reports how many observations old that plan is, which a two-process split cannot know. One process is also what makes System 1's priority stream effective at all: CUDA orders streams within a context, so across processes the GPU time-slices and the priority is invisible. Tensors travel as raw float32 behind the request header rather than as JSON arrays. A 2x3x224x224 frame window is 301,056 values, and parsing that as text cost 545 ms per step -- more than every engine in the pipeline put together. The reply is binary for the same reason: serializing 3,072 floats back cost 37.7 ms of a 90 ms request. A trajectory request now measures 50.7 ms, of which 1.1 ms is protocol. Removed: internvla_n1_z_latents_dump duplicated the server's z_latents mode internvla_n1_s2_server superseded by the dual-system server internvla_n1_system1_inference its latency role is covered better in the loop That last one is worth stating plainly. Measured against the trajectory head on its own at 48.1 ms, dual_system_inference reports 50.2 ms with replanning effectively off and 55.5 ms at cadence 4. The gap is System 2 competing for the GPU, which is what a deployed agent actually pays -- so the in-loop number is not a degraded proxy for the standalone one, it is the more honest figure. Bit-exact parity against a reference implementation, the tool's other role, belongs in tests/defs rather than in a shipped CLI. Signed-off-by: hungho77 <[email protected]>
internvla_n1_system1_inference is gone, so the example that invoked it goes with it. In its place the README carries the measurement that replaced it: first-plan latency and sustained System-1 rate per System-2 precision, taken from dual_system_inference at cadence 4. Worth stating because it is not obvious: quantizing System 2 raises the control rate even though System 1 is untouched by it. The trajectory head runs at 48 ms whichever planner shares the GPU; what changes is how much the planner crowds it, 14.6 Hz at FP16 against 17.9 Hz at NVFP4. Signed-off-by: hungho77 <[email protected]>
The doc gained a Run example for internvla_n1_dual_system_server but nothing showing how to actually drive it or where the tensors it needs come from. Habitat-sim has no C++ binding, so any closed-loop agent using this model is Python on the caller side regardless of how the runtime itself is built; this section is the client contract that side needs. Includes where frames.bin / noise.bin come from -- a random draw for a latency check versus real ResNet-normalized frames for an actual decision -- since feeding un-normalized pixels produces plausible-looking but wrong trajectories with no error at all. Signed-off-by: hungho77 <[email protected]>
…ndation Every latency figure re-taken on one build with an idle GPU, so the tables are internally comparable rather than assembled across sittings: llm_bench for System 2 at 100 iterations, dual_system_inference at cadence 4 for the sustained System-1 rate. The recommendation changes. This page previously recommended FP8 and ruled NVFP4 out for navigation because its bridge cosine sits at 0.962, below a 0.99 gate. Closed-loop success rate over 199 episodes says otherwise: NVFP4 reaches 67.8% against PyTorch's 69.8% (McNemar p = 0.572), while FP16 -- highest on every cosine -- scores lowest of the three at 66.8%. The gate was measuring something that does not predict the outcome, so it is replaced with the SR result and a note to use cosine only for catching a broken export. Also records what llm_bench can and cannot reach. It links edgellmCore, so it has no route to InternVLAN1System1Runner; System-1 and end-to-end numbers come from dual_system_inference, which additionally measures them under the GPU contention a deployed agent actually sees. Signed-off-by: hungho77 <[email protected]>
vla/index.md still named internvla_n1_system1_inference as the entry point; that binary was removed when the CLIs consolidated onto internvla_n1_dual_system_inference. Its accompanying note also described the two systems as "driven separately rather than by a single executable", which was true of the removed binary but not of the one that replaced it. supported-models.md linked InternRobotics/InternVLA-N1, which 404s. The released checkpoint is InternRobotics/InternVLA-N1-DualVLN, used consistently everywhere else this PR references it. CHANGELOG.md and experimental_models/README.md were checked and need no change -- neither names a specific CLI or checkpoint URL. Signed-off-by: hungho77 <[email protected]>
hungho77
added a commit
to VinRobotics/model-quantization-recipes
that referenced
this pull request
Aug 24, 2026
NVIDIA/TensorRT-Edge-LLM#193 exports InternVLA-N1-DualVLN directly, no repackage step, bridge folded into the graph, plus an async C++ runtime and 199-episode closed-loop SR this recipe never measured. Also flags that the recipe's central claim -- z_latents cosine as the acceptance metric -- does not hold. Closed-loop SR showed neither z_latents cosine nor trajectory cosine predicts navigation success for this model, in either direction; they rank this recipe's own FP8 vs NVFP4 call the wrong way. The FP8-on-System-1 measurement and the NVFP4 root-cause analysis are unaffected and still hold.
hungho77
added a commit
to VinRobotics/model-quantization-recipes
that referenced
this pull request
Aug 24, 2026
…port NVIDIA/TensorRT-Edge-LLM#193 exports InternVLA-N1-DualVLN directly, no repackage step, bridge folded into the graph, plus an async C++ runtime and 199-episode closed-loop SR this recipe never measured. Also flags that the recipe's central claim -- z_latents cosine as the acceptance metric -- does not hold. Closed-loop SR showed neither z_latents cosine nor trajectory cosine predicts navigation success for this model, in either direction; they rank this recipe's own FP8 vs NVFP4 call the wrong way. The FP8-on-System-1 measurement and the NVFP4 root-cause analysis are unaffected and still hold.
hungho77
added a commit
to VinRobotics/model-quantization-recipes
that referenced
this pull request
Aug 24, 2026
…structions (PR #193) NVIDIA/TensorRT-Edge-LLM#193 (open, not yet merged) adds native InternVLA-N1-DualVLN export -- direct checkpoint export, the z_latents bridge (final_norm + cond_projector) folded into the graph -- so once building from that branch, the repackage pass, host-side bridge computation, and custom export/quantize scripts this recipe used to carry are no longer needed. What is left as this recipe's job is the one piece that stays outside TensorRT-Edge-LLM regardless: building a navigation-domain calibration set, since calibrating on the deployment prompt's own domain rather than generic news text measurably improves quantization quality (FP8 trajectory cosine 0.909 -> 0.978 in earlier testing). Replaces the old repackage/quantize/export/verify script tree (45 files) with a single build_calib_jsonl.py plus a two-target Makefile, and rewrites the README with quantize + export + build instructions for PR #193's branch, plus the 199-episode closed-loop SR results, which supersede the recipe's earlier z_latents-cosine-based FP8-over-NVFP4 recommendation.
Collaborator
|
Thanks. May I ask what is this for? I think this is pretty useful but want to understand what is this model for, and your effort to support this model. To take this model officially I will ask @linc-nv on the decision. |
Author
Thanks for asking. Here is the context:
Thanks for considering it for official support. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Closes #190.
Type of change: New feature
Overview: Adds support for InternVLA-N1-DualVLN, a vision-language navigation model
built from two systems that run at different rates:
They are joined by
z_latents: the hidden states at the trajectory-query positions, normed andprojected. The planner's text output is not what drives navigation — a checkpoint can produce
fluent replies and still be useless here, so
z_latentsis the signal this PR is validated on.The four learned latent queries travel as real tokens, the route Alpamayo uses for its
trajectory tokens: the export writes them into the embedding table's trailing padding rows
(Qwen2.5 pads its table well past the last used ID and those rows are zero), registers
<|latent_q0..3|>as special tokens, and records the IDs inconfig.json. A prompt endingwith those tokens puts the queries into the sequence through the runtime's ordinary embedding
lookup, so the final norm and
cond_projectorfold into the graph and the engine emitsz_latentsdirectly — no runtime API change anywhere.Three things are worth calling out for review:
No repackaging step. The exporter reads the released InternVLA config directly. Its
vision_configalready declaresmodel_type: qwen2_5_vland the tower weights sit under thebare
visual.prefix thatbuild_qwen25_vl_visualauto-detects, so the tower needs no shim —only registration.
One core change came out of this: the runtime sized and copied its hidden-states buffer at
model width regardless of what the engine actually emits, so a graph that projects its
hidden states handed consumers a buffer whose tail was unrelated memory — silently, since
the shape it reports looks reasonable either way. The width now comes from an optional
output_hidden_sizeconfig key defaulting tohidden_size, which extends to the plain patha notion the spec-decode path already had (EAGLE-3 emits
hidden_size * 3). Verifiedbit-identical output on the same engine with only the config changed.
System 1 is vendored, not imported. Its reference implementation targets transformers 4.x
and builds on
diffusers; this repo takes neither. Reimplementing follows the precedent set bymodels/alpamayo, and also drops machinery the reference carries but never reaches in thisconfiguration (no RoPE, all-ones attention masks, an identity GQA repeat).
The runtime lands under
experimental_models/, following Cosmos3. System 1 keeps its owncontext-memory pool: the core runtime sizes one shared pool as
max()over components, whichassumes they never overlap — and overlapping with System 2 is exactly what System 1 must do.
No core file is modified beyond registration. Nothing under
cpp/,unittests/, or3rdParty/is touched.
Try it
Jetson Thor, JetPack 7.1 (TRT 10.13.3.9, CUDA 13). ~2 h end to end, most of it the engine
builds. Every number in Validation below is reproduced by these steps.
Setup
Quantize, export, build
trtexeclives at/usr/src/tensorrt/bin/trtexecand is not onPATH.Check the quantized checkpoint kept System 1 — both halves fail silently if the loader is
broken, and the symptom only appears as garbled trajectories much later:
Reproduce the latency and control-rate table
frames.bin/noise.binare plainnumpy.tofile()dumps — nothing generates them for you.Random ones are fine here: only the timing is being measured, the trajectories are meaningless.
Expect from the last one, with an NVFP4 System 2 (±4 % run-to-run drift is normal):
0 before the first plan landedis the line that matters — it means System 1 waited for a planrather than steering on nothing.
llm_benchcovers System 2 only. It linksedgellmCoreand has no route toInternVLAN1System1Runner, so System-1 and end-to-end numbers come fromdual_system_inference, which additionally measures them under the GPU contention a deployedagent actually sees.
Reproduce the closed-loop SR numbers
SR needs a simulator, and habitat-sim is Python-only, so the loop stays in Python and calls the
engines through
internvla_n1_dual_system_server— one JSON object per stdin line, tensors asraw float32 behind the header:
$EX/internvla_n1_dual_system_server --llmEngineDir engines/llm --actionEngineDir engines/actionThe client is a thin shim that swaps InternNav's
model.generate/generate_latents/generate_trajfor calls into that server; the evaluator itself is unchanged.docs/source/user_guide/examples/vla/internvla_n1.mdcarries a ~40-line reference client and theexact request shapes, including how to build
frames.binfrom real camera frames (they must beResNet-normalized — the memory block does not normalize again, and raw
[0,255]pixels giveplausible-looking but wrong tokens with no error).
Reproducing the table needs the InternNav habitat harness, its
mp3d_cescenes, and about fourhours per variant at 199 episodes.
Validation
Jetson Thor (sm_110, JetPack 7.1, TRT 10.13.3.9), batch 1, idle GPU unless noted. All figures
reproduced by the commands under Try it.
System 1 port is exact
Vendored
traj_dit/ memory block against the reference implementation, same weights, sameinputs, fp32 — the check a self-consistent-but-wrong port fails:
traj_ditEngines against PyTorch: memory block 0.999929, one denoising step 0.999964, full System-1
chain (real conditioning from a file) 0.999670 — mean waypoint deviation 1.6% of a waypoint's
own reach.
Latency and control rate
Per-stage cost of one planning step and the sustained control rate from
internvla_n1_dual_system_inferenceat--ticks 40 --cadence 4. System 1 is BF16 in everyrow — only System 2 is quantized:
Quantizing System 2 raises the control rate even though System 1 is untouched by it — the
trajectory head costs ~52 ms whichever planner shares the GPU, and what changes is how much the
planner crowds it. Part of the 4.8 → 18.0 Hz jump is the engines, part is the asynchronous
handoff itself: the reference agent blocks while System 2 plans, so its worst tick is 331 ms
against 78 ms here.
llm_benchcovers System 2 only (see Try it); System-1 and end-to-endnumbers come from
dual_system_inference, measured under real GPU contention rather than idle.Run-to-run drift across sittings is ~4%, the same order as the gap between repeated
measurements of one engine — do not read a winner out of a few milliseconds.
One process is what makes the priority stream effective — CUDA orders streams within a
context, so across processes the GPU merely time-slices:
System 1 never waits for a plan — it samples from the newest one available and reports its
staleness. Measured against blocking until the planner finishes, that costs nothing per tick
(55.9 ms vs. 57.9 ms) and caps the worst case at 78 ms instead of 331 ms.
Calibration data matters more than the algorithm
The quantized rows above are calibrated on navigation prompts (InternData-N1's R2R training
instructions, deployment template, ending in the trajectory-query tokens) rather than
cnn_dailymail. Same FP8 recipe: trajectory cosine 0.909 → 0.978, mean waypoint deviation 93%→ 29% of a waypoint's reach. NVFP4 barely moves — it quantizes activations dynamically and has
no static scale to mis-fit.
No offline metric predicts SR — measure it
Bridge cosine (z-cosine) and trajectory cosine each rank a different variant wrong, in
opposite directions. z-cosine puts three NVFP4 recipes within 0.002 of each other while their
trajectories span 0.949 / 0.927 / 0.811. Trajectory cosine ranks FP16 above FP8 (0.998 vs.
0.978); closed-loop SR below ranks them the other way. Accept a quantized checkpoint on
closed-loop SR or not at all — use cosine only to catch a broken export (0.25 means broken).
QAD (distilling z against the bf16 teacher on navigation prompts) makes the same point from
the other side: it raises bridge cosine as intended (0.9621 → 0.9775) and lowers both
trajectory cosine (0.9489 → 0.9383) and SR (67.8% → 66.3%). Optimizing the visible metric moved
the model away from the outcome it stands in for. Not recommended, not part of this PR.
Closed-loop navigation success
199 R2R val_unseen episodes, habitat-sim on Thor, identical episodes per variant,
VLN_TRAJ_SEED=100.pis an exact McNemar test on the discordant pairs:No variant differs from PyTorch significantly (lowest p = 0.296), so the choice is size and
speed. Recommendation: NVFP4 — fastest, smallest, highest control rate, SR indistinguishable
from PyTorch. FP8 is the conservative choice for 2.6 GB more.
Scope: these SR runs put System 2 on TensorRT and left System 1 on PyTorch, because the
habitat harness is the reference agent's. The full TensorRT pipeline (both systems, async) has
since been run closed-loop and works — 10 episodes, SR 90%, no protocol errors — but not yet at
the 199 episodes this table needs.
A retracted claim, kept as a warning (click to expand)
An earlier revision of this PR reported NVFP4 at 63.3% (23 lost / 10 won, p = 0.035) and called
it a real regression. That run drew the trajectory sampler's noise unseeded. Re-running the
identical engine over the identical episodes with the seed fixed gives 67.8%, 16/12, p = 0.572 —
the 0.035 was sampler noise landing one-sided. The noise floor is real: the same checkpoint over
the same 50 episodes returned 72.0% and 68.0% on two unseeded runs, and even seeded, 6 of 50
episodes flip between two different seeds. Never publish a paired SR result from an unseeded
run, and prefer ~200 episodes — at n=50 this same comparison gives p = 0.375 for an effect that
199 episodes resolves.
🚀 Pull Request Checklist
✅ Pre-commit Checks
pre-commitby runningpip install pre-commit.pre-commit install.pre-commit run --all-filesand fixed any reported issues.🧪 Tests
📄 Documentation
⚙️ Compatibility
Additional Information
On
pre-commit run --all-files. Every hook passes on this PR's files. The run also reportsclang-format changes in nine pre-existing sources under
cpp/andunittests/that this PR doesnot touch — they are byte-identical to
main, and the pinned clang-format still wants toreformat them, so the drift predates this branch. They are left alone here, per the single-concern
guidance in CONTRIBUTING.md; happy to send them as a separate formatting PR if you would like.
Scoped to this PR's files, all nine hooks pass:
pre-commit run --files $(git diff --name-only main..HEAD)On the test checkboxes.
tests/defs/config.pynow classifies an InternVLA checkpoint asModelType.VLA, anddocs/,CHANGELOG.mdand the supported-models list are updated.The VLA build test was run rather than skipped. It builds the LLM and visual engines
successfully and then stops at step 3,
action_build, which documents its input as"model.onnx and config.json" — one graph. InternVLA's System 1 is deliberately two, since a
fused graph would re-encode the frames on every denoising step, so it does not fit that shape
and the guide uses
trtexecinstead. Extending the harness to a multi-graph action componenttouches shared test infrastructure, so it seemed better to agree on the approach first than to
land it here.
For context on the third checkbox:
tests/test_lists/contains no VLA node IDs for any model,Alpamayo included, so the VLA suite is manual-only today and there is no passing baseline to
match. "All tests are passing" is left unticked rather than claimed.
On backward compatibility. Every change to an existing file is additive. The two new
ModelConfigfields (n_query,latent_dim) default to0, so no other model's behaviourchanges.
On the guidance default. Both examples expose
--guidanceand default it to 1.0, which iswhat InternNav deploys: every
generate_trajcall site in the reference (realworld agent,policy, habitat evaluator) leaves
guidance_scaleat its default. The flag matters because asilent mismatch between the two sides of a comparison is invisible — the trajectories stay
plausible and only the reference comparison shows the drift.
One note for anyone reproducing the NVFP4 numbers. NVFP4 at
maxBatchSize 1needs__LUNOWUD=-cask_fusion:max_num_epilogues=1exported beforellm_build. Without it the engineproduces garbage and is faster — 62.3 ms against 74.0 ms — because a miscompiled kernel does
less work, so a suspiciously good number from an unpatched build is the symptom rather than a win.
Building at
--maxBatchSize 2avoids it at no cost and needs no flag.