Repository navigation
feat(hardware): read free VRAM and the UMD CUDA banner - #640
Merged
Merged
Conversation
The fit judged models against the card's total memory, so a model that fit an empty card could fail to load while the desktop held part of it. Free memory now comes from the same nvidia-smi query Detect already runs, as Box.VRAMFreeBytes, so the fit no longer needs a second probe. The separate free-VRAM probe and its hook in the models command are removed; a fit with no free figure falls back to the total. Drivers 610 and later print the CUDA version as "CUDA UMD Version:" instead of "CUDA Version:". The banner parser accepts both labels, so the detected CUDA version is not lost on those drivers. Signed-off-by: Ion Alpha <[email protected]> Signed-off-by: Ion Alpha <[email protected]>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
The table in TestPolicyRejectsMutatedClaims was not gofmt-aligned, so the repository format gate printed this file and failed. The change is whitespace only: the case names and mutations are untouched. Signed-off-by: Ion Alpha <[email protected]> Signed-off-by: Ion Alpha <[email protected]>
The earlier gofmt-only commit on this file is outside this change and undoes gofumpt alignment that CI's lint enforces. Putting main's content back leaves the branch diff limited to the hardware probe. Signed-off-by: Ion Alpha <[email protected]>
ion-alpha-dev
added a commit
that referenced
this pull request
Oct 31, 2026
…origin needs Serve refused every entry that was not a vetted catalog source with ErrNotServable. The flynn binary hands overlay entries (models add, catalog.user.json) to Host.Serve, so none of them could run. Serve now runs any entry in the host's catalog. An entry below TrustBlessed is classified by the origin it names and is refused with ErrNotAdmitted where the sandbox gives less isolation than that origin needs. A reference that names no catalog entry is still ErrNotServable. Admission reads the isolation through a hook the tests set, so an entry that needs a microvm can reach the admitted path on a machine that has none. entrySource decided whether a Hub claim was fetched from its repository with a prefix test on the URL text, which accepted "/acme/tiny/../other/x.gguf". The path is now cleaned first, an encoded parent segment included. Free VRAM comes from Box.VRAMFreeBytes, which replaces hardware.FreeVRAM. This depends on #640 and must merge after it. Tests assert the exact source kind per origin (file, URL, Hub repository) and serve a TrustLocal overlay entry end to end. Signed-off-by: Ion Alpha <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Reads the free GPU memory in the same nvidia-smi query that
hardware.Detectalready runs, exposes it asBox.VRAMFreeBytes, and hasflynn models --local --fitjudge models against that figure. The separate free-VRAM probe is removed. The CUDA banner parser also accepts theCUDA UMD Version:label that drivers 610 and later print.Why
The fit judged models against the card's total memory. The desktop and other programs already hold part of the card, so a model that fits the total can fail to load. The fit now uses the free figure Detect reports, and falls back to the total when the driver does not report one.
Drivers 610 and later print the CUDA version as
CUDA UMD Version:. The parser only matchedCUDA Version:, so the detected CUDA version was lost on those drivers.How to verify
Run
go test ./internal/hardware/... ./cmd/flynn/....TestParseNvidiaSMIFreeVRAMcovers the free-memory column, including hidden, zero, negative and above-total readings.TestParseCUDAVersioncovers both banner labels.TestDetectReadsFreeVRAMAndUMDBannerruns Detect end to end against recorded nvidia-smi output.On a machine with an NVIDIA GPU,
go run ./cmd/flynn models --local --fitprints the free VRAM in its first line. Compare it withnvidia-smi --query-gpu=memory.free --format=csv.Notes for reviewers
The signatures of
resolveMachine,renderFit, andfitMachineare unchanged.detectBoxreplaces the removed probe hook as the seam tests stub.A card whose free memory is reported as 0 is treated as not reported, so the fit uses the total for it. A fully occupied card is therefore judged at its total. This is a known gap.
internal/inference/gpukeeps its own CUDA banner parser, since sharing one needs an export from that package. Consolidating the two is a possible follow-up.The two changes are in one commit because they touch the same parser and the same test files.