Skip to content

flashnext: a shard in a folder inside the model folder (#1220) - #1221

Closed
paragontasx wants to merge 2 commits into
ashhart:mainfrom
paragontasx:flashnext-shard-folder
Closed

paragontasx wants to merge 2 commits into
ashhart:mainfrom
paragontasx:flashnext-shard-folder

Conversation

@paragontasx

@paragontasx paragontasx commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

nvidia/Qwen3.8-Flash-Next-FP4-Local maps every tensor to tensors/model-000NN-of-00021.safetensors.
#1179 lets the core index accept a shard in a folder inside the model folder, but Flash Next validates the weight map
a second time before any header is read, and that gate refused any / with UnsafeFlashShard — so serve stopped
before CUDA init. This is the family half of #1187; filed as #1220.

  • families/flashnext/index.zig: shardName/plainPart now take the same rule as core/checkpoint_index.zig
    (plain names joined by /, no empty, hidden, dotted-tail or reserved part, and the file must be .safetensors),
    so tensors/model-…safetensors is admitted while ../, absolute paths, drive names and device stems stay refused.
    It is kept local because this file imports no core by design. Tests: the committed 30-shard fixture is admitted
    flat, admitted identically under tensors/, and refused under ../; plus an accept/refuse table.
  • core/checkpoint.zig: a checkpoint whose index names shards in a folder opens and reads a tensor
    (shardFiles + openModel + get), so the loader itself is covered, not only the index.
  • cli/pull_test.zig: the fake hub serves tensors/model-…safetensors; the main pull test asserts the snapshot's
    subfolder link resolves to the verified blob and that a second pull reports cached; a security: test pins that
    the nested path stays inside the snapshot. The new sha hex is freed in stopServer.
  • cli/hub.zig: sizeOf counts shards in a subfolder (this is the info memory floor a nested checkpoint would
    otherwise undercount); the registry test no longer hardcodes gemma4_unified, which only the Metal registry serves
    — it panicked when the module was tested with the CUDA registry.
  • build: pull_test.zig previously ran in no passing no-GPU step. zig/build/pull_tests.zig adds
    test-pull-host (the whole pull suite, part of test-cuda-host, skipped on Windows where the fake hub’s POSIX socket cannot compile), and golden.step now takes the platform's engine
    registry so zig build test-pull-security compiles again — it had been failing the no-GPU CI job since hub.zig
    began reading native_engines. The CUDA host passes native_modules.engines, the Metal host its own engines.

No kernel changes.

Receipt

Environment: Linux x86_64, Zig 0.17.0, no GPU; main @ 8e49cf8 plus this branch.

  • zig build test-pull-host → 21 tests pass
  • zig build test-cuda-host --summary all → 176/176 steps succeeded
  • zig test zig/src/families/flashnext/index.zig → 2/2
  • zig fmt --check → clean

Not run, and why

The GPU path: this machine has no NVIDIA device, and the checkpoint-level proof here is the publisher's own index
layout (30 shards, 3,747 tensors) rather than a load. I have a GB10 available and will send a served receipt
(serve nvidia/Qwen3.8-Flash-Next-FP4-Local, --prompt-cache-gib 0) against this branch next, unless you would
rather take it from here.

Fixes #1220. Remaining half of #1187.

nvidia/Qwen3.8-Flash-Next-FP4-Local maps every tensor to
tensors/model-000NN-of-00021.safetensors. Core's index check accepts that
since ashhart#1179, but the family's own admission refused any '/' with
UnsafeFlashShard, so serve stopped before CUDA init.

- families/flashnext/index.zig: shardName/plainPart take the same rule as
  core/checkpoint_index.zig, kept local so this file still imports no core;
  escapes (../, absolute, a drive, a hidden or reserved part) stay refused.
  Tests: the 30-shard fixture admitted flat, under tensors/, and refused
  under ../.
- core/checkpoint.zig: a checkpoint whose index names shards in a folder
  opens and reads them (shardFiles + openModel + get).
- cli/pull_test.zig: the fake hub serves tensors/model-…safetensors; the
  main pull test asserts the snapshot's subfolder link, its blob and the
  second pull's "cached", and a security test pins the nested path.
- cli/hub.zig: sizeOf counts shards in a subfolder (the info memory floor);
  the registry test no longer hardcodes a Metal-only family.
- build: test-pull-host runs pull's tests with the engine registry; the
  older test-pull-security step lacks native_engines and does not compile.

Evidence: zig build test-pull-host (21 tests) and zig build test-cuda-host
(176/176 steps) pass on Linux x86_64 with Zig 0.17.0; zig fmt --check clean.
@paragontasx

Copy link
Copy Markdown
Contributor Author

Served receipt (one GB10, this branch)

Environment: NVIDIA GB10 (DGX Spark), aarch64, driver 580.173.02, CUDA 13.0, nvcc 13.0.88, Zig 0.17.0,
zig build native -Dnvcc=/usr/local/cuda/bin/nvcc -Dsm=121. Source: this PR's head 8e20fd0 on main 8e49cf8.

tensorfold-native serve <nvidia/Qwen3.8-Flash-Next-FP4-Local@b9ce18a6> \
  --name p0-flashnext --host 127.0.0.1 --port 8099 \
  --context 32768 --parallel 4 --temperature 0 --prompt-cache-gib 0

Startup, unedited:

[tensorfold] CUDA sm_121 device 0 (NVIDIA GB10, memory shared with the host): model 73.24 GiB; 4 streams at once,
1.21 GiB each at a 32768-token window, of 14.8 GiB left after a 12.2 GiB reserve; prompts in passes of up to
1536 rows, no prompt cache; own glue kernels
[tensorfold] serving p0-flashnext at http://127.0.0.1:8099/v1 (sampling: greedy; drafts: on; context: 32768;
loaded in 163.1s)

The publisher's tensors/model-000NN-of-00021.safetensors layout loaded as it ships — no symlink folder and no
rewritten index. The same checkpoint on v1.1.2 stops before CUDA init.

Requests (greedy, reasoning_effort: none, 256 tokens):

measured
drafted, one stream, code 60.5 / 61.4 / 61.1 tok/s, TTFT 143–149 ms, draft acceptance 0.84
"draft": false, same prompt 24.2 tok/s, token_sha identical to the drafted runs
4 streams, drafted, code 59.7 tok/s aggregate (15.4–15.9 each), every reply the solo run's token_sha
~8k-token cold prompt TTFT 5.93 s

So, on this box and this branch: drafted == "draft": false, and concurrent == solo. Drafted throughput does not
grow with streams because each drafted stream verifies its window in its own pass, as the recipe documents.

Not run here: vision (server started without --vision), requests past the 32,768 window, the kernel-contract
receipts, and two ranks. The box's other work was stopped for this run and restored afterwards.

`pull_test.zig` ran in no passing no-GPU step: `test-pull-security` had not
compiled since `cli/hub.zig` began reading `native_engines` for `family()`,
and `cli/tests.zig` is built with two name filters only.

- new `zig/build/pull_tests.zig` owns the pull test module — the engine
  registry, EXLR8's import, the `test-pull-host` step and its dependency on
  `test-cuda-host`; it returns early on Windows, where the fake hub's POSIX
  socket (`sockaddr.in`) cannot compile.
- `golden.step` takes the platform's engine registry, so
  `test-pull-security` compiles again: the CUDA host passes
  `native_modules.engines`, the Metal host its own `engines`.
- `cuda.zig` stays within the 600-line lean limit; the call replaces a blank
  line beside the other Windows-guarded host test.

Verified: `zig build test-pull-host test-pull-security` passes; lean check
reports 0 new problems against 8e49cf8; `zig fmt --check` clean.
@paragontasx
paragontasx force-pushed the flashnext-shard-folder branch from 0d6c7b0 to 9ec3f52 Compare October 23, 2026 19:42
@ashhart

ashhart commented Oct 24, 2026

Copy link
Copy Markdown
Owner

Thanks @paragontasx. This is on main now (3aaf727): both commits, with your name and your address in GitHub's numeric noreply form. Flash Next's index admission now takes a shard in a folder inside the model folder by the same rule as the core checkpoint index: plain parts joined by /, no hidden, empty, drive or device names, .safetensors only. ../, absolute paths and the rest are still refused. The pull host tests pass, 21 of 21, along with both new index tests. Closing as landed, and #1220 with it.

- Synthema, Ash's AI Assistant.

@ashhart ashhart closed this Oct 24, 2026
@ashhart

ashhart commented Oct 24, 2026

Copy link
Copy Markdown
Owner

This shipped in TensorFold 1.1.4: https://github.com/ashhart/TensorFold/releases/tag/v1.1.4

Thank you @paragontasx for the contribution.

- Synthema, Ash's AI Assistant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flash Next CUDA admission refuses a shard in a folder inside the model folder (the family half of #1187)

2 participants