From f38f26782ef5c46962bd7c8da051c5c736c1f4dc Mon Sep 17 00:00:00 2001 From: Ivan Zatevakhin Date: Sat, 24 Oct 2026 23:56:37 +0100 Subject: [PATCH 1/3] refactor: simplify bindings lifecycle and examples --- CHANGELOG.md | 10 +- Justfile | 8 +- README.md | 135 +++++------------------ RELEASING.md | 70 ++---------- vllm-cpp/examples/chat.rs | 188 +++++++++++++------------------- vllm-cpp/src/engine.rs | 42 ++----- vllm-cpp/src/params.rs | 6 +- vllm-cpp/src/request.rs | 224 ++++++++++++++++++-------------------- 8 files changed, 240 insertions(+), 443 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f8f7300..c17e915 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,8 +6,6 @@ All notable changes to this project will be documented in this file. ## [0.0.2] - 2026-08-27 -This entry describes the release candidate; it does not authorize or attest publication. - ### Changed - Pinned bundled native vllm.cpp to tag `v0.0.2`, commit `7020de93652ca920424a10ac5255b34810dd2f24`, moving the stable C contract from ABI 10 with 19 functions to ABI 17 with 35 functions; system libraries must match the new ABI, layouts, signatures, and exports. @@ -16,9 +14,9 @@ This entry describes the release candidate; it does not authorize or attest publ - Removed arbitrary package size and file-count caps for both crates. File counts, compressed sizes, regular-file payload sums, and SHA-256 hashes remain diagnostics, not release limits; archive inventories must match Cargo's package lists, and content, provenance, extracted-build, and downstream compatibility checks remain required. - Contained the entire Rust logits-processor trampoline lifetime, including final capture destruction after concurrent registration teardown, while preserving callback panic reporting and panic-payload disposal. - Pinned online GGUF and Safetensors metadata and body retrieval to a validated immutable commit; explicit full-SHA requests reject different metadata or offline snapshots. Validate downloads before explicitly updating requested refs, and repair hf-hub's implicit pinned-SHA ref writes on rejection without rolling back mutable aliases. Offline resolution still requires a cache ref; mutable `main` remains the default requested revision. -- Native C API fixture failures now retain the default scheduler's failing status; an alternate scheduler run cannot make the gate pass. Callback counts are not token counts, so the upstream callback-count assertion alone does not establish token loss. -- Optional ASan/UBSan and native-only TSan recipes select exact package/target test executables from Cargo JSON, reject empty suites, and require each exact TSan filter to pass one named test. Removed the stale release number from the CUDA/Vulkan conflict diagnostic. -- Simplified duplicated packaging policy and release documentation, and replaced the video downloader/mux application with a minimal local-checkpoint PPM/WAV example. Narrowed the committed WAV fixture reader, repeated link-mode tests, native-gate status plumbing, and redundant build/sampling checks. Bundled builds now watch the complete native include directory. Public library APIs and centralized native-buffer safety checks are unchanged. +- The native C API fixture preserves the default scheduler's result. Callback counts are not token counts and do not alone prove token loss. +- Hardened optional ASan/UBSan and native-only TSan recipes to select exact Cargo test targets and require nonempty passing suites; each exact TSan filter must run one named test. Also corrected the CUDA/Vulkan conflict diagnostic. +- Replaced the video downloader/mux application with a minimal local-checkpoint PPM/WAV example; narrowed its fixture reader and simplified duplicate maintainer checks. Bundled builds now watch the complete native include directory; public APIs and centralized native-buffer safety checks are unchanged. ### Added @@ -26,7 +24,7 @@ This entry describes the release candidate; it does not authorize or attest publ - Pre-tokenized `Engine::complete_tokens` with owned token output, optional copied completion metadata, and explicit truncation reporting. - Thread-local, exclusive `TranscriptionEngine` and `EmbeddingEngine` owners with borrowed inputs and Rust-owned transcription and row-major embedding results. - Separate `VideoEngine` checkpoint-set ownership, blocking generation parameters/results, and standalone `VideoMuxParams`/`VideoMuxArgv` composition that never executes ffmpeg. -- Model-free acceptance gates for docs, tests, ABI/layout/signature/exports, native fixtures, sanitizers, link modes, packages, downstream consumers, publish dry-run, and exact MSRV. Focused Rust regressions cover callback destruction and loopback Hub branch movement/cache identity. Native-gate status, exact artifact selection, nonzero test counts and named filters, and regular-file payload accounting remain enforced by the gate implementations. These gate definitions are not evidence of an exact-candidate pass. +- Added model-free gates for docs, workspace tests, ABI/layout/signature/exports, native fixtures, sanitizers, link modes, packages, downstream consumers, publish dry-run, and exact MSRV. Rust regressions cover callback destruction and Hub branch/cache identity; gates retain exact artifact selection, nonzero/named test assertions, and diagnostic-only payload accounting. ### Compatibility diff --git a/Justfile b/Justfile index 7d701b5..e5b055f 100644 --- a/Justfile +++ b/Justfile @@ -61,7 +61,7 @@ header-check: "${CXX:-c++}" -std=c++20 -fsyntax-only -x c++ -I'{{ root }}/vllm-cpp-sys' \ '{{ root }}/vllm-cpp-sys/wrapper.h' -# Run focused tests for private build-script artifact selection helpers. +# Run focused no-CMake tests; the workspace all-target test also covers this suite. build-support-test: #!/usr/bin/env bash set -euo pipefail @@ -72,7 +72,7 @@ build-support-test: -o "$temp/build-support-tests" "$temp/build-support-tests" -# Test the pure Linux backend build planner without configuring CMake. +# Run the pure planner tests without CMake; the workspace all-target test covers this suite. backend-config: #!/usr/bin/env bash set -euo pipefail @@ -92,8 +92,8 @@ backend-integrity: layout-test: CARGO_TARGET_DIR="${CARGO_TARGET_DIR:-{{ root }}/target/layout-test}" cargo test --locked -p vllm-cpp-sys --release --test layout -# Run generated-binding, header, layout, and backend configuration conformance checks. -sys: bindings-check header-check build-support-test backend-config backend-integrity layout-test +# Run sys conformance checks not already covered by workspace all-target tests. +sys: bindings-check header-check backend-integrity layout-test # Test all Linux CPU link modes and the exact shared-library exports. link-modes: diff --git a/README.md b/README.md index fc72b78..e8760cf 100644 --- a/README.md +++ b/README.md @@ -1,146 +1,63 @@ # vllm.cpp Rust Bindings -Rust bindings for [vllm.cpp](https://github.com/mudler/vllm.cpp), organized as: +Rust bindings for [vllm.cpp](https://github.com/mudler/vllm.cpp), split into a safe application API and a low-level C ABI crate. -- `vllm-cpp-sys`: raw C API bindings and the pinned native source build. -- `vllm-cpp`: the application-facing safe inference API. +## Status and identity -## Status +> Work in progress: this project follows a pinned native release and may lag later upstream APIs or backend behavior. -> **Work in progress:** these bindings track a pinned vllm.cpp release and may lag later upstream APIs or backend behavior. Check the exact native identity and support boundary before adoption. +The bundled source is vllm.cpp `v0.0.2`, commit `7020de93652ca920424a10ac5255b34810dd2f24`. The stable C boundary is ABI 17 with 35 functions. `vllm-cpp-sys` contains the checked-in raw bindings; `vllm-cpp` provides RAII-based completion, chat, requests, transcription, embeddings, and video APIs. -The safe crate covers text completion, streaming, raw-JSON and optional serde chat, concurrent requests, structured output, custom logits processing, pre-tokenized completion, transcription, embeddings, video generation, and standalone video-mux argument composition. It also includes a synchronous Hugging Face resolver, text examples, and a minimal local-checkpoint video example. The sys crate exposes the complete 35-function stable C boundary at ABI 17 with checked-in generated bindings and C/Rust conformance checks. +Native Linux x86_64 CPU is the only supported runtime family, with bundled or system libraries and static or dynamic linking. Other platforms and CUDA, CUTLASS, Triton, Vulkan, Metal, and MLX are build or optional-validation surfaces, not runtime-support claims. Compilation and model-free tests do not establish model inference or backend correctness. -The native source is independently pinned to vllm.cpp tag `v0.0.2`, commit `7020de93652ca920424a10ac5255b34810dd2f24`. Native Linux x86_64 CPU is the supported runtime target, including bundled/system and static/dynamic linking. Linux ARM64, Apple ARM64, CUDA, external CUTLASS, Triton AOT, Vulkan, Metal, and external MLX are configured build or optional validation surfaces; they are not accelerator runtime-support claims. +## Crates and guides -## Prerequisites +- [`vllm-cpp`](vllm-cpp/README.md) is the safe crate guide, including ownership, callbacks, Hub resolution, features, and deployment. +- [`vllm-cpp-sys`](vllm-cpp-sys/README.md) documents the unsafe ABI, native build, conformance checks, and link modes. +- [`vllm-cpp/examples`](vllm-cpp/examples/README.md) is a standalone guide to the text and experimental local-checkpoint video examples. -Native builds require: +These package-local guides are the detailed API references; this root page covers workspace setup and maintainer checks. -- Rust and Cargo. -- CMake 3.24 or newer. -- Ninja or another CMake build tool. -- A C11 and C++20 compiler. -- A system linker and C++ standard library. -- Just 1.40 or newer for maintainer workflows, plus Git, `jq`, and GNU tar. +## Requirements and checkout -This repository provides a Nix development shell with the pinned development tools. Linux also has minimal CUDA and Vulkan shells: +Native builds require Rust/Cargo, CMake 3.24+, a C11 and C++20 compiler, a linker, and Ninja or another CMake generator. Just 1.40+, Git, `jq`, and GNU tar are maintainer tools. The optional Nix shells provide pinned development tools: ```console nix develop +nix develop .#msrv nix develop .#cuda nix develop .#vulkan -nix develop .#msrv ``` -## Checkout - -vllm.cpp is a Git submodule. Clone recursively, or initialize it after cloning: +The native source is a Git submodule. Clone recursively or initialize it after checkout: ```console git submodule update --init --recursive ``` -## Safe API - -The packaged [`vllm-cpp` guide](vllm-cpp/README.md) covers the full API, ownership, callbacks, filesystem effects, link modes, and deployment. The [`vllm-cpp-sys` guide](vllm-cpp-sys/README.md) documents the unsafe ABI and native build boundary. - -`Engine::load` accepts a native-compatible model directory or standalone GGUF. `HuggingFaceModel` synchronously resolves a GGUF or runtime-complete sparse Safetensors snapshot into the normal cache; retrieval does not prove native model, task, or backend compatibility. - -Text `Engine` is a cloneable `Send + Sync` RAII owner. It provides blocking completion, streaming, chat, structured output, custom logits processing, and `complete_tokens`, plus non-blocking `Request` submission. Requests retain an `Arc` to the engine, are `Send` but not `Sync`, and expose completion probes, cancellation, waiting, and copied diagnostics. Callback panics are contained before crossing C; asynchronous callbacks run on a native delivery thread, and callback-thread wait/free is rejected or deferred under the stable lifecycle contract. - -`EngineBuilder` starts from `vllm_model_params_default()` and overlays only explicitly selected settings, preserving native helper defaults such as `block_size=32`, `max_num_seqs=32`, and `gpu_memory_utilization=0.92`. Text/task [`Device`](vllm-cpp/src/params.rs) numbering is `Auto=0`, `Cpu=1`, and `Cuda=2`; explicit CUDA never silently falls back. KV sizing precedence is `num_blocks` over `kv_cache_memory_bytes` over `gpu_memory_utilization` and its native profile/fallback path. - -`complete_tokens` borrows caller-provided token IDs for one blocking call. Its output capacity limits only reported/copied IDs, not native generation; `truncated` compares the copied count with native completion metadata. `include_completion = false` suppresses the Rust metadata copy, while the hidden native metadata request still occurs so truncation remains accurate. - -`TranscriptionEngine` and `EmbeddingEngine` are separate, non-cloneable, conservatively thread-local RAII owners. Their operations take `&mut self`, block, borrow call inputs, and copy results into Rust-owned values; embeddings are row-major and preserve input order. ABI 17 has no task-introspection function, so none of the task-specific owners can prove a checkpoint's task at load time. Native wrong-task `InvalidArgument` diagnostics remain authoritative. - -`VideoEngine` separately owns a MiniMax-H3 checkpoint set and performs exclusive blocking generation. Video device numbering is `Cpu=0` and `Cuda=1`, with no `Auto`; explicit CUDA never falls back. Generation writes frame/audio artifacts, may leave stale or partial output, and trusts caller paths without sandboxing. `VideoMuxParams` and `VideoMuxArgv` only compose ordered `OsString` argument boundaries. This crate never executes ffmpeg, an HTTP server, or another process; `vllm_server_main` remains available only through the raw crate. - -The high-level crate intentionally adds no tokenizer, task-query, raw-handle, process-execution, or HTTP-server wrapper. See [the examples guide](vllm-cpp/examples/README.md) for text binaries and the experimental local-checkpoint video example. Release-facing changes are recorded in the [changelog](CHANGELOG.md), and maintainers use the manual [release process](RELEASING.md). - -## Build and Test - -Use the root `Justfile` (requires Just 1.40 or newer) for maintainer workflows, optionally inside the Nix shell: +## Build and test ```console CMAKE_GENERATOR=Ninja cargo build --locked --release -CMAKE_GENERATOR=Ninja cargo test --locked --workspace --release -cargo test --locked -p vllm-cpp --release --features serde +just test just ci +nix develop .#msrv -c just msrv ``` -Set `CMAKE_BUILD_PARALLEL_LEVEL` to control native parallelism. The default bundled build remains deterministic and CPU-only: native tests, examples, the HTTP server, CUDA, Metal, MLX, Vulkan, Triton, and CUTLASS fetching are disabled explicitly. Use `nix develop .#msrv -c just msrv` for the exact local Rust 1.85.0 policy check; the manual `platforms` workflow runs the same exact toolchain policy. - -`build.rs` is consumer-only native build/link integration; it does not download dependencies or compile/execute the maintainer layout probe. Ordinary consumers do not need Just, bindgen, or libclang. Normal first-time Cargo dependency resolution may access crates.io; use Cargo's standard `--offline` mode after dependencies are cached. The high-level crate's required `hf-hub` 0.5 dependency uses only its synchronous `ureq` feature, without Tokio or another async runtime. Library download progress is disabled by default. - -## Experimental Backend Builds +`just test` runs locked workspace all-target tests with `serde`; `just ci` adds formatting, lint/docs, ABI and layout conformance, backend integrity, CPU link modes, native fixtures, sanitizers, package/downstream checks, and a publish dry-run. The exact Rust 1.85.0 check is separate. Fast `just build-support-test` and `just backend-config` recipes remain available without configuring CMake. -Backend features are bundled-only and mutually exclusive with `system`; CUDA and Vulkan also conflict. CUDA/CUTLASS/Triton/Vulkan target Linux x86_64/aarch64, while Metal/MLX require exact `aarch64-apple-darwin`. Features do not imply runtime support and do not enable `bundled` for `--no-default-features` callers. Use a fresh `CARGO_TARGET_DIR` for each backend/link combination. - -- `cuda` requires `VLLM_CPP_CUDA_ARCHITECTURES` equal to `80`, `86`, `87`, `89`, `90a`, `100a`, `103a`, `110`, `120a`, `121a`, or `120a;121a`. -- `cuda-cutlass` requires caller-provided CUTLASS >=4.5.0, disables fetching, and rejects `103a` and `110`. -- `triton-aot` packages and embeds all six checked-in AOT trees: `sm_80`, `sm_86`, `sm_89`, `sm_90a`, `sm_100a`, and `sm_121a`. Runtime dispatch selects only an exact SM match; other accepted CUDA targets, including `87`, `103a`, `110`, and `120a`, retain the portable C++/CUDA fallback. Regeneration remains disabled for consumer builds. -- `vulkan` uses packaged headers and SPIR-V and opens the runtime loader dynamically. -- `metal` links the Apple Metal and Foundation frameworks on Apple ARM64. -- `mlx` implies `metal` and requires an external `MLX_ROOT`; Cargo neither fetches nor packages MLX and emits no machine-local rpath. - -A configuration example, not candidate runtime evidence: - -```console -nix develop .#cuda -VLLM_CPP_CUDA_ARCHITECTURES=80 \ - CARGO_TARGET_DIR=target/cuda-static \ - cargo build --locked --release --features cuda -``` - -Static CUDA links toolkit libraries selected by CMake. Static Apple builds link `libc++` and the selected frameworks/providers. Dynamic builds rely on the native shared library's transitive dependencies. Deploy `libvllm.so`/`libvllm.dylib` and optional toolkit/MLX libraries through normal loader paths. - -Compilation does not establish runtime correctness. CUDA/CUTLASS/Triton, Vulkan, Metal, and MLX remain experimental build surfaces; no accelerator runtime support is claimed. - -## Optional Model and Sanitizer Recipes - -The required Linux x86_64 CPU candidate gate is model-free and uses committed native fixtures. An optional prepared-model lane uses Apache-2.0 `Qwen/Qwen3-0.6B` at revision `c1899de289a04d12100db370d81485cdf75e47ca`; the model is never included in repository or crate packages. No ordinary test or instrumentation recipe downloads a model implicitly. - -```console -model=$(just setup-test-model) -VLLM_CPP_TEST_MODEL="$model" \ - cargo test --locked -p vllm-cpp --release --test qwen3 -- --test-threads=1 -just sanitizers "$model" -just tsan "$model" -``` - -These commands are optional evidence and are recorded only when rerun against the exact candidate. Sanitizer recipes use exact package/target executables from Cargo JSON, require nonzero complete suites, and reject an exact TSan filter unless one named test passes. The TSan recipe instruments native C++ only and does not claim Rust standard-library race coverage. `VLLM_CPP_SANITIZE` is a bundled-build test input; system mode rejects it. - -## Link Modes - -- The default `bundled` feature builds and statically links the pinned CPU-only source. -- `bundled,dynamic-link` builds and dynamically links the pinned shared library. -- `system` requires `--no-default-features` and links a prefix selected by `VLLM_CPP_ROOT`. -- `system,dynamic-link` dynamically links the selected system library. - -System-mode consumer builds validate that `VLLM_CPP_ROOT/include/vllm.h` exists; they do not compare its layout. The maintainer `layout-test` recipe compiles `tests/layout.c` at test runtime against the bundled header and compares its C layouts with the generated Rust declarations. The `link-modes` recipe repeats layout conformance against bundled and system fixture headers and also runs the runtime ABI test. `VLLM_CPP_LIB_DIR` selects a nonstandard vllm library directory. Upstream's normal CMake install does not install `libblake3_vendored.a`, so a stock install works directly with `system,dynamic-link`; system static users must provision that archive separately and set `VLLM_CPP_BLAKE3_LIB_DIR` (or place it in the vllm library directory). Dynamic tests and applications must make `libvllm.so` loader-visible with `LD_LIBRARY_PATH`, rpath, or an installed loader path; Cargo neither deploys the library nor configures rpath. - -## Packaging - -Inspect and test both packaged crates with: - -```console -cargo package -p vllm-cpp-sys --locked --list -cargo package -p vllm-cpp --locked --list -just package-test -``` +The default bundled build is deterministic and CPU-only. Native tests, examples, server, accelerator backends, and native dependency fetching are disabled. `build.rs` performs consumer build/link integration without network access; ordinary consumers do not need Just, bindgen, or libclang. Cargo may access crates.io during normal dependency resolution. The safe crate's synchronous `hf-hub` dependency does not add an async runtime; ordinary tests and instrumentation use a model only when one is explicitly prepared and supplied. -Cargo manifests define package contents. The gate compares archives with `cargo package --list`, checks metadata, essential native inputs, licenses and forbidden payloads, then runs offline extracted tests, four extracted sys link modes, and independent sys and safe downstream consumers. File counts, compressed sizes, regular-file payload bytes and SHA-256 hashes are diagnostics only, with no arbitrary caps. +## Backend and packaging boundaries -## Support +Backend features are bundled-only, have explicit platform and prerequisite rules, and do not imply runtime support. See the safe and sys crate guides for exact feature contracts and deployment requirements. Use a fresh `CARGO_TARGET_DIR` for each backend/link combination. -Native Linux x86_64 CPU is the supported runtime family. `just ci` runs the mandatory model-free gates; the exact MSRV check is separate. See [RELEASING.md](RELEASING.md) for the complete checklist and sys-first publication process, including full registry-resolved safe dry-run verification. +Cargo manifests define package contents. `just package-test` compares archives with `cargo package --list`, checks metadata, native inputs, licenses and forbidden payloads, then tests extracted crates and independent downstream consumers offline. File counts, compressed sizes, payload bytes, and SHA-256 values are diagnostics, not caps. -Model-free tests do not establish prepared-model inference or successful MiniMax-H3 generation. Accelerator runtime, other platforms, prepared-model sanitizers, TSan and Miri remain optional or deferred. Configured workflows and successful compilation alone are not runtime-support evidence. +`just publish-dry-run` performs package verification and a workspace Cargo dry-run with `--no-verify` to avoid the registry dependency cycle. It does not upload crates or create tags, and it does not replace the full registry-resolved safe-crate dry-run after the sys crate is available. See [RELEASING.md](RELEASING.md) for the sys-first release sequence and exact gates. -## Licensing and Affiliation +## Licensing and affiliation -The Rust workspace is dual-licensed under MIT or Apache-2.0. The root `LICENSE` contains the MIT terms, including the original QueryMate copyright, and `LICENSE-APACHE` contains the Apache-2.0 terms. Bundled components retain their own licenses and notices under `vllm-cpp-sys`. +The Rust workspace is dual-licensed under MIT or Apache-2.0. Bundled components retain their own licenses and notices under `vllm-cpp-sys`. -vllm.cpp and these bindings are independent community projects. They are not affiliated with, endorsed by, or sponsored by the vLLM project, the PyTorch Foundation, or the Linux Foundation. +vllm.cpp and these bindings are independent community projects, not affiliated with, endorsed by, or sponsored by the vLLM project, the PyTorch Foundation, or the Linux Foundation. diff --git a/RELEASING.md b/RELEASING.md index 8a3bb0b..9435f1c 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -1,34 +1,17 @@ # Releasing -Releases are manual. Review the candidate from a clean checkout before publishing; -validation and dry-runs do not upload crates or create tags. +Releases are manual. Validation and dry-runs do not upload crates or create tags; those actions are separate, explicit release steps. ## Prepare -- Check `git status --short` and `git submodule status --recursive`, including a - clean native worktree. Confirm the gitlink, native HEAD, and peeled `v0.0.2` tag - all name `7020de93652ca920424a10ac5255b34810dd2f24`: - - ```console - git rev-parse HEAD:vllm-cpp-sys/vllm.cpp - git -C vllm-cpp-sys/vllm.cpp rev-parse HEAD 'refs/tags/v0.0.2^{}' - ``` - -- Confirm both Rust crates are `0.0.2`, with exactly `vllm-cpp-sys =0.0.2` in the - safe crate, and native CMake reports `0.0.2`. Check manifests and lockfiles. -- Preserve ABI 17 and all 35 stable C functions: generated bindings, C11/C++20 - headers, layouts, signatures, runtime ABI, linking, and dynamic exports must - agree. ABI-10 system libraries are incompatible. -- Keep `Unreleased` empty above the dated changelog entry. Describe only tested - support: Linux x86_64 CPU is supported; accelerator backends remain experimental. -- Review packaged licenses/notices and `vllm-cpp-sys/THIRD_PARTY.md`, including - imported license texts. Packages must not contain models, media fixtures, build - output, caches, SDKs, external CUTLASS trees, internal records, or local paths. - Cargo manifests define package inclusion and exclusion. +- Start from a clean checkout and recursive submodule. Confirm the gitlink, native HEAD, and peeled `v0.0.2` tag all resolve to `7020de93652ca920424a10ac5255b34810dd2f24`. +- Keep both Rust crates at `0.0.2`, `vllm-cpp` dependent on exactly `vllm-cpp-sys =0.0.2`, and native CMake at `0.0.2`. +- Preserve ABI 17 and all 35 stable C functions across bindings, headers, layouts, signatures, runtime checks, linking, and dynamic exports. ABI-10 system libraries are incompatible. +- Claim only tested support: native Linux x86_64 CPU is supported; accelerator backends remain experimental. Keep licenses/notices and Cargo's package boundaries accurate; packages must exclude models, fixtures, caches, build output, external CUTLASS, and local paths. ## Validate -Use the normal Rust/native build tools and Just 1.40 or newer: +Run the maintained gates with Just 1.40 or newer: ```console just ci @@ -36,36 +19,17 @@ RUSTUP_TOOLCHAIN=1.85.0 just msrv git diff --check ``` -`just ci` runs formatting, warnings-denied lint/docs, model-free workspace tests, -bindings/header/layout/signature/ABI/export -checks, four workspace CPU link modes, native C API fixtures, ASan/UBSan/leak -checks, package verification, and a no-upload publish dry-run. The MSRV gate -separately requires Rust and Cargo exactly 1.85.0. +`just ci` runs formatting, warnings-denied lint/docs, locked model-free workspace tests, binding/header/layout/ABI/symbol checks, backend integrity, four CPU link modes, native C API fixtures, ASan/UBSan/leak checks, package/downstream verification, and a publish dry-run. Workspace all-target tests include the build-support and backend-config suites. The separate MSRV gate requires Rust and Cargo exactly 1.85.0. -Run the package gate with a fresh target directory (the CI package run may serve -as this run). It compares archives with `cargo package --list`, checks metadata, -source inputs, licenses and leaks, then runs offline extracted tests, four -extracted sys link modes, and independent sys and safe downstream consumers. -File counts, payload/compressed sizes and SHA-256 hashes are diagnostics, not -caps. A second run is optional; claim byte reproducibility only if compared -archive hashes actually match. +`just package-test` compares each archive with `cargo package --list`, checks metadata, native inputs, licenses and leaks, then runs offline extracted tests, four extracted sys link modes, and independent sys and safe downstream consumers. File counts, payload/compressed sizes, and SHA-256 values are diagnostics, not caps. Claim byte reproducibility only when two archive hashes actually match. -The native fixture gate unsets `VT_ASYNC_SCHED` and `VT_ASYNC_RUNNER`. A default -failure remains a failure; alternate-scheduler diagnostics cannot satisfy it. -Record any failed gate rather than treating later individual passes as a CI pass. +The native fixture gate clears `VT_ASYNC_SCHED` and `VT_ASYNC_RUNNER` and validates the default-scheduler result. -Nix is optional: use `nix develop --no-write-lock-file -c just ci` and -`nix develop .#msrv --no-write-lock-file -c just msrv` instead if desired. -Prepared-Qwen inference/sanitizers, native-only TSan, Rust MiniMax-H3 generation, -Miri, other platforms, and accelerator runtime are optional or deferred lanes. -Report only what actually ran; model-free passes do not establish model inference. +Nix is optional. Prepared-Qwen inference/sanitizers, native-only TSan, Rust MiniMax-H3 generation, Miri, other platforms, and accelerator runtime are optional or deferred. Report only evidence actually collected: model-free passes and successful compilation do not establish prepared-model inference or backend correctness. ## Publish -Check crates.io for the exact versions before uploading. An unavailable or -ambiguous registry state needs diagnosis, not a trial upload. Publish sys first, -wait for registry availability, then run the **full** safe dry-run against registry -sys `0.0.2` before uploading the safe crate: +`just publish-dry-run` performs package verification followed by a workspace Cargo dry-run with `--no-verify` to avoid the registry dependency cycle. It does not upload crates or tag a release, and it is not the full registry-resolved safe-crate dry-run. Publish sys first, wait until it is available, then fully verify the safe crate against registry sys `0.0.2` before its upload: ```console cargo publish -p vllm-cpp-sys --locked @@ -73,14 +37,4 @@ cargo publish -p vllm-cpp --locked --dry-run cargo publish -p vllm-cpp --locked ``` -The pre-publication `just publish-dry-run` uses `--no-verify` to avoid the registry -cycle; it does not replace that full safe dry-run. After publication, verify -registry metadata, docs.rs, licenses, and a clean downstream build before tagging -the published commit and creating a release. - -Accepted versions are immutable: never republish or replace them with different -bytes. If sys succeeds but safe fails, check registry state before retrying; -retry unchanged safe contents only for a transient failure, otherwise prepare a -new coordinated version. Stop on identity drift, failed gates, dirty release -state, or unexpected package content. Yanking does not erase a crate or make its -version reusable and is not an ordinary abort mechanism. +After explicit publication, verify registry metadata, docs.rs, licenses, and a clean downstream build before tagging the published commit. Published versions are immutable: never replace a version with different bytes. Check registry state before retrying a failed safe upload; yanking does not make a version reusable. diff --git a/vllm-cpp/examples/chat.rs b/vllm-cpp/examples/chat.rs index 6c41069..f4049fe 100644 --- a/vllm-cpp/examples/chat.rs +++ b/vllm-cpp/examples/chat.rs @@ -262,12 +262,14 @@ fn run(args: Args) -> ChatResult<()> { write_stdout_line("Interactive chat: use /clear to reset the conversation or /quit to exit.")?; if let Some(prompt) = initial_prompt { - handle_turn( - &mut history, - prompt, - |history, prompt| run_turn(&engine, history, &config, !no_stream, prompt), - report_turn_error, - )?; + if let Err(error) = run_history_turn(&mut history, prompt, |history| { + generate_assistant(&engine, history, &config, !no_stream) + }) { + if error.is_fatal() { + return Err(error); + } + eprintln!("chat: {error}"); + } } interactive_loop(&engine, &mut history, &config, !no_stream) } @@ -330,51 +332,50 @@ fn interactive_loop( } Input::Empty => {} Input::Message(message) => { - handle_turn( - history, - message.to_owned(), - |history, message| run_turn(engine, history, config, stream, message), - report_turn_error, - )?; + if let Err(error) = run_history_turn(history, message.to_owned(), |history| { + generate_assistant(engine, history, config, stream) + }) { + if error.is_fatal() { + return Err(error); + } + eprintln!("chat: {error}"); + } } } } } -fn run_turn( +fn generate_assistant( engine: &Engine, - history: &mut Vec, + history: &[Message], config: &GenerationConfig, stream: bool, - user_message: String, -) -> ChatResult<()> { - run_history_turn(history, user_message, |history| { - let request = build_request(history, config, stream)?; - if stream { - write_assistant_prefix()?; - let result = stream_assistant(engine, &request); - let content = result.as_ref().map_or("", |assistant| assistant.as_str()); - finish_assistant_output(content)?; - result - } else { - engine - .chat_json(&request) - .map_err(|error| ChatError::turn(format!("blocking chat request failed: {error}"))) - .and_then(|response| extract_blocking_content(&response)) - .and_then(|assistant| { - write_assistant_prefix()?; - { - let stdout = io::stdout(); - let mut stdout = stdout.lock(); - stdout.write_all(assistant.as_bytes()).map_err(|error| { - ChatError::fatal(format!("failed to write assistant response: {error}")) - })?; - } - finish_assistant_output(&assistant)?; - Ok(assistant) - }) - } - }) +) -> ChatResult { + let request = build_request(history, config, stream)?; + if stream { + write_assistant_prefix()?; + let result = stream_assistant(engine, &request); + let content = result.as_ref().map_or("", |assistant| assistant.as_str()); + finish_assistant_output(content)?; + result + } else { + engine + .chat_json(&request) + .map_err(|error| ChatError::turn(format!("blocking chat request failed: {error}"))) + .and_then(|response| extract_blocking_content(&response)) + .and_then(|assistant| { + write_assistant_prefix()?; + { + let stdout = io::stdout(); + let mut stdout = stdout.lock(); + stdout.write_all(assistant.as_bytes()).map_err(|error| { + ChatError::fatal(format!("failed to write assistant response: {error}")) + })?; + } + finish_assistant_output(&assistant)?; + Ok(assistant) + }) + } } fn run_history_turn( @@ -393,40 +394,12 @@ where Ok(()) } Err(error) => { - rollback_turn(history, checkpoint); + history.truncate(checkpoint); Err(error) } } } -fn handle_turn( - history: &mut Vec, - user_message: String, - run: F, - mut report_error: R, -) -> ChatResult<()> -where - F: FnOnce(&mut Vec, String) -> ChatResult<()>, - R: FnMut(&ChatError), -{ - match run(history, user_message) { - Ok(()) => Ok(()), - Err(error) if error.is_fatal() => Err(error), - Err(error) => { - report_error(&error); - Ok(()) - } - } -} - -fn report_turn_error(error: &ChatError) { - eprintln!("chat: {error}"); -} - -fn rollback_turn(history: &mut Vec, checkpoint: usize) { - history.truncate(checkpoint); -} - fn stream_assistant(engine: &Engine, request: &str) -> ChatResult { let stdout = io::stdout(); let mut stdout = stdout.lock(); @@ -834,39 +807,33 @@ mod tests { #[test] fn rolls_back_failed_turn_and_continues_with_the_next_input() { - let mut history = vec![Message::new(Role::System, "system")]; - let mut errors = Vec::new(); - let mut attempted_messages = Vec::new(); - - handle_turn( - &mut history, - "failed question".to_owned(), - |history, message| { - attempted_messages.push(message.clone()); - run_history_turn(history, message, |_history| { - Err(ChatError::turn("request failed")) - }) - }, - |error| errors.push(error.to_string()), - ) - .unwrap(); - assert_eq!(history, vec![Message::new(Role::System, "system")]); - assert_eq!(attempted_messages, ["failed question"]); - assert_eq!(errors, ["request failed"]); - - handle_turn( - &mut history, - "next question".to_owned(), - |history, message| { - run_history_turn(history, message, |_history| Ok("answer".to_owned())) - }, - |_| panic!("successful turn must not report an error"), - ) + let system = Message::new(Role::System, "system"); + let mut history = vec![system.clone()]; + + let error = run_history_turn(&mut history, "failed question".to_owned(), |history| { + history.extend([ + Message::new(Role::Assistant, "junk assistant entry"), + Message::new(Role::Assistant, "partial streamed response"), + ]); + Err(ChatError::turn("request failed")) + }) + .unwrap_err(); + assert_eq!(error.to_string(), "request failed"); + assert!(!error.is_fatal()); + assert_eq!(history, [system.clone()]); + + run_history_turn(&mut history, "next question".to_owned(), |history| { + assert_eq!( + history.as_slice(), + &[system.clone(), Message::new(Role::User, "next question")][..] + ); + Ok("answer".to_owned()) + }) .unwrap(); assert_eq!( history, vec![ - Message::new(Role::System, "system"), + system, Message::new(Role::User, "next question"), Message::new(Role::Assistant, "answer"), ] @@ -874,18 +841,17 @@ mod tests { } #[test] - fn propagates_fatal_errors_instead_of_recovering() { - let mut history = Vec::new(); - let mut reported = false; - let error = handle_turn( - &mut history, - "question".to_owned(), - |_history, _message| Err(ChatError::fatal("standard output failed")), - |_| reported = true, - ) + fn propagates_fatal_errors_after_rolling_back_the_turn() { + let system = Message::new(Role::System, "system"); + let mut history = vec![system.clone()]; + let error = run_history_turn(&mut history, "question".to_owned(), |history| { + history.push(Message::new(Role::Assistant, "partial output")); + Err(ChatError::fatal("standard output failed")) + }) .unwrap_err(); + assert!(error.is_fatal()); - assert!(!reported); + assert_eq!(history, [system]); } #[test] diff --git a/vllm-cpp/src/engine.rs b/vllm-cpp/src/engine.rs index 96ee849..e64f57e 100644 --- a/vllm-cpp/src/engine.rs +++ b/vllm-cpp/src/engine.rs @@ -14,7 +14,7 @@ use crate::callback::{ callback_trampoline, CallbackState, StreamControl, StreamEvent, StreamOutcome, }; use crate::error::{invalid_configuration, status_result, Error}; -use crate::params::{Device, SamplingParams, SchedulerPolicy, Toggle}; +use crate::params::{to_cstring, u32_to_i32, Device, SamplingParams, SchedulerPolicy, Toggle}; /// A cloneable, `Send + Sync` vllm.cpp text-generation engine. #[derive(Clone)] @@ -2346,10 +2346,6 @@ fn optional_cstring(value: Option<&str>, field: &'static str) -> Result Result { - CString::new(value).map_err(|_| Error::InteriorNul { field }) -} - #[cfg(unix)] fn path_to_cstring(path: &Path, field: &'static str) -> Result { use std::os::unix::ffi::OsStrExt; @@ -2387,11 +2383,6 @@ fn required_optional_path_to_cstring( .and_then(|path| nonempty_path_to_cstring(path, field)) } -fn u32_to_i32(value: u32, field: &'static str) -> Result { - i32::try_from(value) - .map_err(|_| invalid_configuration(format!("{field} exceeds native i32 range"))) -} - fn validate_video_frame_path_length(path: &CString, field: &'static str) -> Result<(), Error> { // Native appends "/frame_%06d.ppm" in a fixed 512-byte buffer. Keeping the // encoded directory at 481 bytes or fewer prevents unchecked truncation. @@ -2513,12 +2504,11 @@ mod tests { embedding_from_raw, generate_video_with, load_engine_with, load_video_engine_with, token_completion_from_raw, transcribe_with, transcription_from_raw, validate_embedding_count, validate_pcm_input, validate_pointer_count, validate_token_input, - video_mux_argv_from_raw, Device, EmbeddingEngineBuilder, EmbeddingTask, - MarshaledEmbeddingInput, MarshaledModelParams, MarshaledTranscriptionInput, - MarshaledVideoGenerationParams, MarshaledVideoModelParams, ModelConfig, SchedulerPolicy, - TextTask, Toggle, TranscriptionEngineBuilder, TranscriptionInput, TranscriptionTask, - VideoDevice, VideoEngine, VideoGenerationParams, VideoModelConfig, VideoMuxParams, - VideoPartition, + video_mux_argv_from_raw, Device, EmbeddingEngineBuilder, MarshaledEmbeddingInput, + MarshaledModelParams, MarshaledTranscriptionInput, MarshaledVideoGenerationParams, + MarshaledVideoModelParams, ModelConfig, SchedulerPolicy, TextTask, Toggle, + TranscriptionEngineBuilder, TranscriptionInput, VideoDevice, VideoEngine, + VideoGenerationParams, VideoModelConfig, VideoMuxParams, VideoPartition, }; use crate::abi::Compatibility; use crate::Error; @@ -2634,21 +2624,6 @@ mod tests { assert_eq!(raw.kv_cache_memory_bytes, 4096); } - #[test] - fn forwards_all_memory_settings_without_changing_native_precedence() { - let mut config = ModelConfig::new("model-dir"); - config.num_blocks = Some(7); - config.gpu_memory_utilization = Some(2.0); - config.kv_cache_memory_bytes = Some(8192); - - let mut params = MarshaledModelParams::new(config).expect("marshal memory settings"); - params.apply_defaults(native_defaults()); - let raw = params.raw(); - assert_eq!(raw.num_blocks, 7); - assert_eq!(raw.kv_cache_memory_bytes, 8192); - assert_eq!(raw.gpu_memory_utilization, 2.0); - } - #[test] fn restricted_task_builders_map_only_their_exposed_configuration() { let transcription = TranscriptionEngineBuilder::new("transcription-model") @@ -2764,10 +2739,8 @@ mod tests { } #[test] - fn shared_loader_checks_abi_before_defaults_for_every_task_marker() { + fn shared_loader_checks_abi_before_defaults() { assert_shared_load_order::(); - assert_shared_load_order::(); - assert_shared_load_order::(); } #[test] @@ -3276,7 +3249,6 @@ mod tests { )); } assert!(checked_product(usize::MAX, 2, "test product").is_err()); - assert!(i32::MAX as usize * i32::MAX as usize > isize::MAX as usize / size_of::()); } #[test] diff --git a/vllm-cpp/src/params.rs b/vllm-cpp/src/params.rs index a63daa6..2265a8c 100644 --- a/vllm-cpp/src/params.rs +++ b/vllm-cpp/src/params.rs @@ -605,13 +605,13 @@ fn valid_slice_layout(pointer: *const T, length: usize) -> bool { && length <= (isize::MAX as usize) / size_of::() } -fn discard_panic_payload(payload: Box) { +pub(crate) fn discard_panic_payload(payload: Box) { if let Err(payload) = catch_unwind(AssertUnwindSafe(|| drop(payload))) { std::mem::forget(payload); } } -fn lock_unpoisoned(mutex: &Mutex) -> std::sync::MutexGuard<'_, T> { +pub(crate) fn lock_unpoisoned(mutex: &Mutex) -> std::sync::MutexGuard<'_, T> { mutex .lock() .unwrap_or_else(std::sync::PoisonError::into_inner) @@ -646,7 +646,7 @@ fn optional_u32_to_i32(value: Option, field: &'static str) -> Result Result { +pub(crate) fn u32_to_i32(value: u32, field: &'static str) -> Result { i32::try_from(value) .map_err(|_| invalid_configuration(format!("{field} exceeds native i32 range"))) } diff --git a/vllm-cpp/src/request.rs b/vllm-cpp/src/request.rs index aac005e..cf3acc1 100644 --- a/vllm-cpp/src/request.rs +++ b/vllm-cpp/src/request.rs @@ -13,7 +13,9 @@ use vllm_cpp_sys as ffi; use crate::callback::{StreamControl, StreamEvent}; use crate::engine::{Engine, EngineInner}; use crate::error::{status_result, Error}; -use crate::params::{to_cstring, LogitsProcessorRegistration, SamplingParams}; +use crate::params::{ + discard_panic_payload, lock_unpoisoned, to_cstring, LogitsProcessorRegistration, SamplingParams, +}; /// How a successfully waited non-blocking request ended. /// @@ -377,30 +379,11 @@ unsafe extern "C" fn async_callback_trampoline( state.record_result(result, finished) } -fn lock_unpoisoned(mutex: &Mutex) -> std::sync::MutexGuard<'_, T> { - mutex - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner) -} - struct CleanupJob { - state: CleanupState, - context: CleanupContext, -} - -enum CleanupContext { - Caller, - Reaper, -} - -enum CleanupState { - Armed { - raw: NonNull, - callback: Box, - logits_processor: Option, - engine: Arc, - }, - Disarmed, + raw: NonNull, + callback: Box, + logits_processor: Option, + engine: Arc, } impl CleanupJob { @@ -411,122 +394,85 @@ impl CleanupJob { engine: Arc, ) -> Self { Self { - state: CleanupState::Armed { - raw, - callback, - logits_processor, - engine, - }, - context: CleanupContext::Caller, - } - } - - fn run(mut self) { - self.finish(); - } - - fn finish(&mut self) { - if matches!(self.state, CleanupState::Disarmed) { - return; - } - let needs_deferral = match self.context { - CleanupContext::Caller => match &self.state { - CleanupState::Armed { - callback, - logits_processor, - .. - } => { - callback.is_delivery_thread() - || logits_processor - .as_ref() - .is_some_and(LogitsProcessorRegistration::is_active_on_current_thread) - } - CleanupState::Disarmed => return, - }, - // A successfully sent job is owned only by the prestarted Rust reaper, - // so it cannot be running in the native request callback. - CleanupContext::Reaper => false, - }; - if needs_deferral { - self.defer_to_reaper(); - } else if let Err(payload) = catch_unwind(AssertUnwindSafe(|| self.cleanup_now())) { - // Retrying free after a Rust unwind could double-free an opaque void - // native operation. Disarm and leak the unknown remainder instead. - self.leak_armed(); - std::mem::forget(payload); - } - } - - fn cleanup_now(&mut self) { - let state = std::mem::replace(&mut self.state, CleanupState::Disarmed); - let CleanupState::Armed { raw, callback, logits_processor, engine, - } = state - else { - return; - }; - // If this function unwinds, forget every owner before CleanupJob::Drop can - // run. Repeating an opaque void free could double-free, while releasing - // callback/engine without a known join would be unsafe. - let mut owners = std::mem::ManuallyDrop::new((callback, logits_processor, engine)); - // ABI coupling: output delivery records its permanent thread ID, while - // logits calls record their active thread. This path cannot free from either - // callback context. New native user_data entrypoints must join this tracking. - // - // SAFETY: this job owns the request once and retains its callback and engine. - // Native free cancels the request and joins output delivery before the logits - // processor registration is removed. - unsafe { ffi::vllm_request_free(raw.as_ptr()) }; - // SAFETY: output delivery and native request teardown are complete. - let (callback, logits_processor, engine) = - unsafe { std::mem::ManuallyDrop::take(&mut owners) }; - // User callback captures and a stored panic payload can have arbitrary - // destructors. Never let them unwind out of cleanup. - if let Err(payload) = catch_unwind(AssertUnwindSafe(|| drop(callback))) { - std::mem::forget(payload); } - if let Err(payload) = catch_unwind(AssertUnwindSafe(|| drop(logits_processor))) { - std::mem::forget(payload); + } + + fn run(self) { + if self.needs_deferral() { + self.defer(); + } else { + self.free(); } - drop(engine); } - fn leak_armed(&mut self) { - let state = std::mem::replace(&mut self.state, CleanupState::Disarmed); - std::mem::forget(state); + // request_free joins output delivery only; avoid both its callback thread and + // the currently active logits thread before transferring ownership to reaper. + fn needs_deferral(&self) -> bool { + self.callback.is_delivery_thread() + || self + .logits_processor + .as_ref() + .is_some_and(LogitsProcessorRegistration::is_active_on_current_thread) } - fn defer_to_reaper(&mut self) { + fn defer(self) { let sender = match CLEANUP_REAPER.get() { Some(Ok(sender)) => sender, // Submission starts the process-lifetime reaper before native code // can create a request. Without it, self-thread cleanup is impossible. _ => std::process::abort(), }; - let job = Self { - state: std::mem::replace(&mut self.state, CleanupState::Disarmed), - context: CleanupContext::Reaper, - }; - if let Err(error) = sender.send(job) { - // SendError owns the still-live native handle. Its Drop would recurse - // here on the callback thread and eventually release callback/engine - // before native join, so leak it and terminate instead. + if let Err(error) = sender.send(self) { + // Dropping this error would drop the callback, logits processor, and + // engine while the raw handle is live, before native free joins delivery. + // Forget it and abort rather than run those destructors in this terminal state. std::mem::forget(error); std::process::abort(); } } + + fn free(self) { + let Self { + raw, + callback, + logits_processor, + engine, + } = self; + // Keep all Rust owners alive until request_free has returned. If it unwinds, + // the native handle's state is unknown, so retain them without retrying cleanup. + let mut owners = std::mem::ManuallyDrop::new((callback, logits_processor, engine)); + // SAFETY: the unique job owner calls free once, off both protected callback + // contexts, and retains the callback, processor registration, and engine. + let free_result = catch_unwind(AssertUnwindSafe(|| unsafe { + ffi::vllm_request_free(raw.as_ptr()) + })); + if let Err(payload) = free_result { + discard_panic_payload(payload); + return; + } + + // SAFETY: free returned after joining output delivery; the owners were kept + // in ManuallyDrop until that point. + let (callback, logits_processor, engine) = + unsafe { std::mem::ManuallyDrop::take(&mut owners) }; + drop_cleanup_owners(callback, logits_processor, engine); + } } -impl Drop for CleanupJob { - fn drop(&mut self) { - // This is the ownership backstop: every caller-side armed drop either - // frees and joins or transfers all owners to the reaper; a reaper-owned - // drop always completes cleanup locally, so send failure cannot recurse. - self.finish(); +fn drop_cleanup_owners(callback: C, logits_processor: L, engine: E) { + // User captures and stored panic payloads can have arbitrary destructors. + // Contain each panic so later owners are still released, with engine last. + if let Err(payload) = catch_unwind(AssertUnwindSafe(|| drop(callback))) { + discard_panic_payload(payload); } + if let Err(payload) = catch_unwind(AssertUnwindSafe(|| drop(logits_processor))) { + discard_panic_payload(payload); + } + drop(engine); } // The job transfers unique native-handle ownership to the reaper. Its callback @@ -555,3 +501,47 @@ fn cleanup_sender() -> Result<&'static mpsc::Sender, Error> { }), } } + +#[cfg(test)] +mod tests { + use std::sync::{Arc, Mutex}; + + use super::drop_cleanup_owners; + + struct DropProbe { + name: &'static str, + drops: Arc>>, + panic: bool, + } + + impl Drop for DropProbe { + fn drop(&mut self) { + self.drops.lock().unwrap().push(self.name); + if self.panic { + panic!("probe destructor panic"); + } + } + } + + #[test] + fn owner_drop_panics_do_not_skip_later_owners() { + for (callback_panics, logits_panics) in + [(false, false), (true, false), (false, true), (true, true)] + { + let drops = Arc::new(Mutex::new(Vec::new())); + let probe = |name, panic| DropProbe { + name, + drops: Arc::clone(&drops), + panic, + }; + + drop_cleanup_owners( + probe("callback", callback_panics), + Some(probe("logits", logits_panics)), + probe("engine", false), + ); + + assert_eq!(*drops.lock().unwrap(), ["callback", "logits", "engine"]); + } + } +} From 355ecd090e83b3e3abb0b4db1716758da07032e0 Mon Sep 17 00:00:00 2001 From: Ivan Zatevakhin Date: Sun, 25 Oct 2026 01:09:13 +0100 Subject: [PATCH 2/3] ci: deduplicate conformance and example tests --- Justfile | 5 +- vllm-cpp/examples/common/mod.rs | 103 ------------------------------ vllm-cpp/tests/example_common.rs | 104 +++++++++++++++++++++++++++++++ vllm-cpp/tests/safe_api.rs | 6 -- 4 files changed, 106 insertions(+), 112 deletions(-) create mode 100644 vllm-cpp/tests/example_common.rs diff --git a/Justfile b/Justfile index e5b055f..21d45f8 100644 --- a/Justfile +++ b/Justfile @@ -92,8 +92,8 @@ backend-integrity: layout-test: CARGO_TARGET_DIR="${CARGO_TARGET_DIR:-{{ root }}/target/layout-test}" cargo test --locked -p vllm-cpp-sys --release --test layout -# Run sys conformance checks not already covered by workspace all-target tests. -sys: bindings-check header-check backend-integrity layout-test +# Static sys checks; release layout is in the matrix or standalone `layout-test`. +sys: bindings-check header-check backend-integrity # Test all Linux CPU link modes and the exact shared-library exports. link-modes: @@ -721,7 +721,6 @@ native-capi: ctest --test-dir "$build" -N --tests-regex '^test_capi$' | tee "$listing" test_count=$(grep -Ec '^[[:space:]]*Test #[0-9]+: test_capi$' "$listing") [[ $test_count -eq 1 ]] - grep -Fxq 'Total Tests: 1' "$listing" env -u VT_ASYNC_SCHED -u VT_ASYNC_RUNNER \ ctest --test-dir "$build" --output-on-failure --tests-regex '^test_capi$' \ diff --git a/vllm-cpp/examples/common/mod.rs b/vllm-cpp/examples/common/mod.rs index 2fd79d3..d0e8d17 100644 --- a/vllm-cpp/examples/common/mod.rs +++ b/vllm-cpp/examples/common/mod.rs @@ -149,106 +149,3 @@ fn usage(example: &str) -> UsageError { "usage:\n {example} \n {example} local \n {example} hf-gguf [--revision ]\n {example} hf-safetensors [--revision ]" )) } - -#[cfg(test)] -mod tests { - use super::*; - - fn os(values: &[&str]) -> Vec { - values.iter().map(OsString::from).collect() - } - - #[test] - fn parses_local_aliases() { - assert_eq!( - parse_model_source("example", os(&["model.gguf"])).unwrap(), - ModelSource::Local("model.gguf".into()) - ); - assert_eq!( - parse_model_source("example", os(&["local", "model"])).unwrap(), - ModelSource::Local("model".into()) - ); - } - - #[test] - fn parses_hugging_face_sources_with_optional_revisions() { - assert_eq!( - parse_model_source("example", os(&["hf-gguf", "owner/model", "model.gguf"])).unwrap(), - ModelSource::Gguf { - repo: "owner/model".to_owned(), - filename: "model.gguf".to_owned(), - revision: None, - } - ); - assert_eq!( - parse_model_source("example", os(&["hf-safetensors", "owner/model"])).unwrap(), - ModelSource::Safetensors { - repo: "owner/model".to_owned(), - revision: None, - } - ); - assert_eq!( - parse_model_source( - "example", - os(&[ - "hf-gguf", - "owner/model", - "model.gguf", - "--revision", - "release", - ]), - ) - .unwrap(), - ModelSource::Gguf { - repo: "owner/model".to_owned(), - filename: "model.gguf".to_owned(), - revision: Some("release".to_owned()), - } - ); - assert_eq!( - parse_model_source( - "example", - os(&["hf-safetensors", "owner/model", "--revision", "release",]), - ) - .unwrap(), - ModelSource::Safetensors { - repo: "owner/model".to_owned(), - revision: Some("release".to_owned()), - } - ); - } - - #[test] - fn rejects_missing_extra_duplicate_and_misordered_arguments() { - for args in [ - os(&[]), - os(&[""]), - os(&["local"]), - os(&["local", ""]), - os(&["model", "extra"]), - os(&["hf-gguf", "owner/model"]), - os(&["hf-gguf", "owner/model", ""]), - os(&["hf-safetensors", ""]), - os(&["hf-safetensors", "owner/model", "release"]), - os(&["hf-safetensors", "owner/model", "--revision"]), - os(&["hf-safetensors", "owner/model", "--revision", ""]), - os(&[ - "hf-safetensors", - "owner/model", - "--revision", - "one", - "--revision", - "two", - ]), - os(&[ - "hf-gguf", - "--revision", - "release", - "owner/model", - "model.gguf", - ]), - ] { - assert!(parse_model_source("example", args).is_err()); - } - } -} diff --git a/vllm-cpp/tests/example_common.rs b/vllm-cpp/tests/example_common.rs new file mode 100644 index 0000000..1dffff4 --- /dev/null +++ b/vllm-cpp/tests/example_common.rs @@ -0,0 +1,104 @@ +include!("../examples/common/mod.rs"); + +#[cfg(test)] +mod tests { + use super::*; + + fn os(values: &[&str]) -> Vec { + values.iter().map(OsString::from).collect() + } + + #[test] + fn parses_local_aliases() { + assert_eq!( + parse_model_source("example", os(&["model.gguf"])).unwrap(), + ModelSource::Local("model.gguf".into()) + ); + assert_eq!( + parse_model_source("example", os(&["local", "model"])).unwrap(), + ModelSource::Local("model".into()) + ); + } + + #[test] + fn parses_hugging_face_sources_with_optional_revisions() { + assert_eq!( + parse_model_source("example", os(&["hf-gguf", "owner/model", "model.gguf"])).unwrap(), + ModelSource::Gguf { + repo: "owner/model".to_owned(), + filename: "model.gguf".to_owned(), + revision: None, + } + ); + assert_eq!( + parse_model_source("example", os(&["hf-safetensors", "owner/model"])).unwrap(), + ModelSource::Safetensors { + repo: "owner/model".to_owned(), + revision: None, + } + ); + assert_eq!( + parse_model_source( + "example", + os(&[ + "hf-gguf", + "owner/model", + "model.gguf", + "--revision", + "release", + ]), + ) + .unwrap(), + ModelSource::Gguf { + repo: "owner/model".to_owned(), + filename: "model.gguf".to_owned(), + revision: Some("release".to_owned()), + } + ); + assert_eq!( + parse_model_source( + "example", + os(&["hf-safetensors", "owner/model", "--revision", "release",]), + ) + .unwrap(), + ModelSource::Safetensors { + repo: "owner/model".to_owned(), + revision: Some("release".to_owned()), + } + ); + } + + #[test] + fn rejects_missing_extra_duplicate_and_misordered_arguments() { + for args in [ + os(&[]), + os(&[""]), + os(&["local"]), + os(&["local", ""]), + os(&["model", "extra"]), + os(&["hf-gguf", "owner/model"]), + os(&["hf-gguf", "owner/model", ""]), + os(&["hf-safetensors", ""]), + os(&["hf-safetensors", "owner/model", "release"]), + os(&["hf-safetensors", "owner/model", "--revision"]), + os(&["hf-safetensors", "owner/model", "--revision", ""]), + os(&[ + "hf-safetensors", + "owner/model", + "--revision", + "one", + "--revision", + "two", + ]), + os(&[ + "hf-gguf", + "--revision", + "release", + "owner/model", + "model.gguf", + ]), + ] { + assert!(parse_model_source("example", args).is_err()); + } + } +} diff --git a/vllm-cpp/tests/safe_api.rs b/vllm-cpp/tests/safe_api.rs index 23a5762..50abd94 100644 --- a/vllm-cpp/tests/safe_api.rs +++ b/vllm-cpp/tests/safe_api.rs @@ -114,7 +114,6 @@ fn missing_model_is_typed_for_every_task_owner_and_builder() { ]; for error in errors { assert!(matches!(error, Error::ModelLoad { .. }), "{error:?}"); - assert!(!error.to_string().is_empty()); } } @@ -150,11 +149,6 @@ fn interior_nul_fails_before_ffi() { } } -#[test] -fn device_defaults_to_native_auto_selection() { - assert_eq!(Device::default(), Device::Auto); -} - #[test] fn engine_builder_accepts_all_safe_options() { let error = Engine::builder(missing_model()) From 63e0e27a853990f8def80f6d855acdf8b0f36ea9 Mon Sep 17 00:00:00 2001 From: Ivan Zatevakhin Date: Mon, 26 Oct 2026 22:27:23 +0000 Subject: [PATCH 3/3] docs: clarify release dry-run and upload retry guidance --- README.md | 2 +- RELEASING.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index e8760cf..85543cc 100644 --- a/README.md +++ b/README.md @@ -54,7 +54,7 @@ Backend features are bundled-only, have explicit platform and prerequisite rules Cargo manifests define package contents. `just package-test` compares archives with `cargo package --list`, checks metadata, native inputs, licenses and forbidden payloads, then tests extracted crates and independent downstream consumers offline. File counts, compressed sizes, payload bytes, and SHA-256 values are diagnostics, not caps. -`just publish-dry-run` performs package verification and a workspace Cargo dry-run with `--no-verify` to avoid the registry dependency cycle. It does not upload crates or create tags, and it does not replace the full registry-resolved safe-crate dry-run after the sys crate is available. See [RELEASING.md](RELEASING.md) for the sys-first release sequence and exact gates. +`just publish-dry-run` runs only a workspace Cargo publish dry-run with `--no-verify` to avoid the registry dependency cycle. It does not upload crates or create tags. `just ci` runs package verification separately through `just package-test`. This dry-run does not replace the full registry-resolved safe-crate dry-run after the sys crate is available. See [RELEASING.md](RELEASING.md) for the sys-first release sequence and exact gates. ## Licensing and affiliation diff --git a/RELEASING.md b/RELEASING.md index 9435f1c..3499155 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -29,7 +29,7 @@ Nix is optional. Prepared-Qwen inference/sanitizers, native-only TSan, Rust Mini ## Publish -`just publish-dry-run` performs package verification followed by a workspace Cargo dry-run with `--no-verify` to avoid the registry dependency cycle. It does not upload crates or tag a release, and it is not the full registry-resolved safe-crate dry-run. Publish sys first, wait until it is available, then fully verify the safe crate against registry sys `0.0.2` before its upload: +`just publish-dry-run` runs only a workspace Cargo publish dry-run with `--no-verify` to avoid the registry dependency cycle; it does not verify packages, upload crates, or tag a release. `just ci` runs package verification separately through `just package-test`. This is not the full registry-resolved safe-crate dry-run. Publish sys first, wait until it is available, then fully verify the safe crate against registry sys `0.0.2` before its upload: ```console cargo publish -p vllm-cpp-sys --locked @@ -37,4 +37,4 @@ cargo publish -p vllm-cpp --locked --dry-run cargo publish -p vllm-cpp --locked ``` -After explicit publication, verify registry metadata, docs.rs, licenses, and a clean downstream build before tagging the published commit. Published versions are immutable: never replace a version with different bytes. Check registry state before retrying a failed safe upload; yanking does not make a version reusable. +After explicit publication, verify registry metadata, docs.rs, licenses, and a clean downstream build before tagging the published commit. Published versions are immutable: never replace a version with different bytes, and do not retry once a version is accepted. If an upload result is unknown, unavailable, or ambiguous, diagnose registry state until acceptance is known. Retry only a transient failure confirmed absent from the registry, using the unchanged previously validated contents; otherwise prepare a new coordinated version. Yanking does not make a version reusable.