From 0521de36cf46941fce93bba157cd7fdd8423e6d9 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 15 Sep 2026 08:05:14 +0000 Subject: [PATCH 1/6] feat: upgrade llama.cpp from b10948 to b10969 First of two reviewable chunks toward the b10976 release (the full b10948..b10976 diff is 121 KiB, over the runbook's 100 KiB per-step threshold; this step is 99 KiB / 21 commits). Upstream reverses the precompiled-header experiment (#28892, #28882). That is a correctness fix this project shipped, not housekeeping: PCH arrived at b10917, so every pin from b10919 through b10948 carried a heap-buffer-overflow in the rope work buffer (upstream #28858). The PCH force-included ggml-impl.h ahead of ops.h, defining __cpp_lib_hardware_interference_size, so the C++ kernels resolved CACHE_LINE_SIZE to std::hardware_destructive_interference_size while ggml-cpu.c -- a C TU, where that macro can never be defined -- kept the 64 fallback. The buffer is sized by the C side and written by the C++ side. The deleted ggml-cpu guard skipped PCH for GCC on non-x86, so the s390x cross and aarch64-native GCC jobs were never exposed; the clang, AppleClang, clang-cl, MSVC and GCC-on-x86 jobs were. No project source change. No priority-list header moved; common/common.cpp is the only priority-list file touched and only its implementation (#28749 hoists the llama_n_rs_seq check above the probe decode in common_context_can_seq_rm -- same enum, same signature, one fewer decode). Zero tools/server/*.{cpp,h}: every server contract file is byte-identical by blob hash across the whole b10948..b10976 walk, so the three mechanical server-contract greps have no input to compare. All four CMake removals that reach this build are removals, so nothing had to be added here; tools/server/CMakeLists.txt is not processed at all (LLAMA_BUILD_TOOLS/SERVER are forced OFF, the server TUs compile straight into jllama). All nine patches apply, verified against a pristine b10976 worktree up front and independently against pristine b10969 so this intermediate commit is a valid bisect point. #28776 is the sequel to the previous bump's 0013 drop: upstream added a non-VXE s390x build to their own CI, so the scalar configuration build-linux-s390x builds is now guarded upstream too. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Cft7guQngfyKycdJfBEfJP --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 737302a0..c911271a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10948** +Current llama.cpp pinned version: **b10969** ## Upgrading CUDA Version @@ -502,7 +502,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network for the asset build; the embed step is plain cmake -P -git clone --depth 1 --branch b10948 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10969 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build ) mkdir -p webui-generated /tmp/ui-gen cmake -DUI_SOURCE_DIR=/tmp/lc/tools/ui -DUI_BINARY_DIR=/tmp/ui-gen \ @@ -542,7 +542,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10948`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10969`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1562,7 +1562,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10948`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10969`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index f995a214..a3287720 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10948](https://img.shields.io/badge/llama.cpp-%23b10948-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10948) +[![llama.cpp b10969](https://img.shields.io/badge/llama.cpp-%23b10969-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10969) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index d1bc2215..7a16b56b 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -728,3 +728,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10934–b10938 | patches + upstream verification | End of the six-chunk walk. **All ten patches apply at pristine b10938 and all four standing drop-checks say "still required"**, run against the pristine tag because the fail-loud applier detects "does not apply" but never "upstream already fixed this": `0001` (`common_params_parse_main` 0 occurrences in `common/arg.h`; the WIN32 override still at `common/arg.cpp:1282` — unmoved despite three edits to that file in this range), `0010` (`{"vocab_type", …}` still uncast at `server-context.cpp:4554`), `0012` (bare `splits[i] /= split_sum;` still at `src/llama-model.cpp:1491`), `0013` (the three s390x helpers still unguarded at 73/77/83). **`0013` remains filed-but-unmerged upstream** as [ggml-org/llama.cpp#28775](https://github.com/ggml-org/llama.cpp/pull/28775); when it merges, the first tag carrying it aborts the configure and the response is to **delete** the patch, not refresh it. | | b10938–b10948 | 10 commits, **33 KiB** — identical excluding `tools/ui`, because the range does not touch the WebUI at all. `ggml/src/ggml-sycl/**` (**#28227**, a Level-Zero memory-query fix, a new `GGML_SYCL_DEV_DEBUG` macro in `base.hpp`, and a `LEVEL_ZERO_INCLUDE_DIR` → `LEVEL_ZERO_DEV_INCLUDE_DIR` rename plus a longer warning in `ggml-sycl/CMakeLists.txt`), `ggml/src/ggml-cpu/arch/s390/repack.cpp` (**#28775** — *this project's own patch*, see the paired row), `src/models/nemotron-h.cpp` (**#28779**, the expert-FFN-size fallback now throws instead of dividing by zero when a layer declares neither `expert_feed_forward_length` nor `expert_used_count`), plus `tests/**`, `.github/**`, `ci/run.sh`, `docs/backend/SYCL.md` and `.pi/gg/SYSTEM.md`. 16 files, 151 insertions, 95 deletions. | **No project-source change, and not one file on the priority review list moved.** Zero files under `common/`, `include/`, `tools/server/` or `tools/mtmd/`, so every row of the API-compatibility table is vacuously satisfied and the three mechanical server-contract greps have **no input to compare** — the request-field set, its `set_hard_limits` bounds and the emitted response keys cannot have moved. The two upstream *code* changes that are neither tests nor CI are both internal TUs upstream compiles itself: the nemotron-h loader guard (a new `throw` on a malformed GGUF; no signature moved) and the SYCL backend, which only the `sycl-*` classifier jobs build — `base.hpp`'s new `extern int g_ggml_sycl_dev_debug` and the `ggml-sycl/CMakeLists.txt` variable rename are local to `ggml-sycl` and add no required flag or dependency, so those classifiers build unchanged. What *is* ours in this range is a **patch deletion**, not a source edit — see the paired row. | | b10938–b10948 | patches + upstream verification | **Nine patches now, not ten: `0013` was dropped here.** Upstream merged this project's own PR [ggml-org/llama.cpp#28775](https://github.com/ggml-org/llama.cpp/pull/28775) ("ggml-cpu(s390x): guard VXE-only repack helpers", commit `6978052`, first tagged at **b10948**), so `ggml/src/ggml-cpu/arch/s390/repack.cpp` now wraps `vxe_dot_acc` / `vxe_splat_granule` / `vxe_fold` in the `#if defined(__VXE__) || defined(__VXE2__)` guard itself — byte-identical to the patch apart from the trailing `// __VXE__ || __VXE2__` comment the patch put on its `#endif`. **That is the designed signal, and the response is delete, not refresh** (the `0009` precedent at b10280): `git apply -p1` of `0013` against pristine b10948 fails with `patch failed: ggml/src/ggml-cpu/arch/s390/repack.cpp:70 … does not apply`, because the `#if` line it wants to insert is already there. The rationale that outlives the patch — why `build-linux-s390x` is a *scalar* (non-VXE) cross build, and why `-DGGML_VXE=ON` is the wrong response to a future VXE compile error — moved into the "`0013` was dropped at the b10948 bump" note in `CLAUDE.md`, which the job's own comment in `publish.yml` now points at. **The other three standing drop-checks still say "still required"**, run against the pristine tag because the fail-loud applier detects "does not apply" but never "upstream already fixed this": `0001` (`common_params_parse_main` 0 occurrences in `b10948:common/arg.h`; the WIN32 `argv = utf8.ptrs.data()` override still in `common/arg.cpp`), `0010` (`{"vocab_type", meta.model_vocab_type}` still uncast at `b10948:tools/server/server-context.cpp:4554`), `0012` (bare `splits[i] /= split_sum;` still at `b10948:src/llama-model.cpp:1491`, no zero-sum guard). **The s390x half was re-verified with the real cross toolchain rather than by inspection**, because that path has no runnable guard beyond the CI job: `s390x-linux-gnu-g++` (13.2.0) compiles pristine `b10948:ggml/src/ggml-cpu/arch/s390/repack.cpp` clean **both** with the job's own scalar flags (the `-DGGML_NATIVE=OFF` configuration, no `-mvx -mzvector`) **and** with `-mvx -mzvector -march=z15` — upstream's guard covers exactly what the patch covered and forecloses nothing. Verified end-to-end for real: `rm -rf build` then `cmake -B build -DBUILD_TESTING=ON` through the real `FetchContent` path, configure clean, stamp at head `5f436dddb440a288ee5611d7d1eca564a6aca9f4` (= `b10948`) with **nine** SHA-256 lines; extraction unchanged at **138 CLI / 57 request / 15 trainer** names; full `cmake --build --config Release` clean; `ctest` **537/537**; `nm -D` **40** `Java_*` exports, **0** mangled; `mvn -pl llama clean test -Dtest=NativeLibraryLoadSmokeTest` **4/4, 0 skipped** — which is the check that cross-validates the bumped `LLAMA_CPP_VERSION` constant against the linked `build-info`, and needs the `clean` because the constant is inlined into the already-compiled test class. | +| b10948–b10969 | 21 commits, **99 KiB** — identical excluding `tools/ui`, because the range does not touch the WebUI at all. The headline is a **reversal**: **#28892** ("cmake : remove precompiled headers") deletes every `target_precompile_headers` call from `common/`, `src/`, `tests/`, `tools/mtmd/` and `tools/server/`, and **#28882** additionally disables the ggml-cpu PCH and makes `CACHE_LINE_SIZE` include-order independent. Also here: **#28749** `common/common.cpp` (hoist the `llama_n_rs_seq` check above the probe decode in `common_context_can_seq_rm`), **#27000** a new `LLM_ARCH_MAPLE` (Maple 20B-A1B ternary MoE, CPU) with `src/models/maple.cpp`, `conversion/maple.py`, `gguf-py` constants, a `llama-graph.cpp` swiglu-clamp arm and NEOX rope; model-loader correctness (**#28868** `get_key_or_arr` misuse across gemma4 / gemma4-assistant / mimo2 / step35 / qwen4exp, **#28865** mimo2 SWA pattern, **#28896** qwen4exp rms_norm+mul fusion); **#28846** ggml-cuda F32 fallback on devices without BF16 hardware acceleration; **#28670** a SYCL radix `top_k` (new `ggml-sycl/topk-radix.{cpp,hpp}`, +555); **#26885** a `llama_grammar_advance_stack` find+insert coalesce; **#28776** s390x (see the paired row); a new upstream `scripts/check-apiabi-compat.sh` (**#28579**, not consumed here); plus upstream CI/release workflow churn, `ci/run.sh`, and the llama.cpp 0.4.1 / ggml 0.24.0 version bumps. 46 files, 1409 insertions, 273 deletions. | **No project source change, and no priority-list *header* moved** — `common/common.cpp` is the one priority-list file touched and only its implementation. Zero `tools/server/*.{cpp,h}`: `server-schema.cpp`, `server-task.cpp`, `server-context.cpp`, `server-common.h`, `server-task.h` and `server-schema.h` are **byte-identical by blob hash** across the whole b10948→b10976 walk, so the three mechanical server-contract greps have **no input to compare** — the request-field set, its `set_hard_limits` bounds and the emitted response keys cannot have moved. **The PCH removal is a heap-buffer-overflow fix this project shipped**, not housekeeping: PCH arrived with #28091 at **b10917**, so every pin from b10919 through b10948 carried it. Mechanism (upstream #28858) — the PCH force-includes `ggml-impl.h` before `ops.h`, which pulls `` and defines `__cpp_lib_hardware_interference_size`, so the **C++** kernels resolved `CACHE_LINE_SIZE` to `std::hardware_destructive_interference_size` while `ggml-cpu.c`, a **C** TU where that feature macro can never be defined, kept the 64 fallback; the rope work buffer is sized by the C side and written by the C++ side, overflowing it by `(CACHE_LINE_SIZE/4 - 16) * n_threads * 4` bytes and crashing later in `ggml_compute_forward_rope_flt`. Exposure was per-job, not universal: the deleted ggml-cpu guard skipped the PCH for **GCC on non-x86**, so `build-linux-s390x` (cross g++) and the aarch64-native GCC 14 job were never affected, while the clang / AppleClang / clang-cl / MSVC and GCC-on-x86 jobs were — and only where that toolchain's `hardware_destructive_interference_size` actually differs from 64. **On the build-system side all four changes that reach this build are pure removals**, so nothing had to be added here: `tools/server/CMakeLists.txt` is *not* processed at all (`LLAMA_BUILD_TOOLS`/`LLAMA_BUILD_SERVER` are forced OFF and the server TUs are compiled straight into `jllama`), while `common/`, `src/`, `tools/mtmd/` and `ggml/src/ggml-cpu/` are — and dropping PCH also removes the one thing that could have interfered with the sccache-over-Depot compiler launcher. #28749 changes only *when* `common_context_can_seq_rm` answers: for a recurrent/hybrid context it now returns `COMMON_CONTEXT_SEQ_RM_TYPE_RS` before the `llama_memory_clear` + throwaway `llama_decode` probe instead of after it — same enum, same signature, one fewer decode; the consumer is `server-context.cpp`'s `ctx_tgt_seq_rm_type`, compiled into `libjllama`. `LLM_ARCH_MAPLE` is a **mid-enum insert**, but `llm_arch` lives in `src/llama-arch.h` and is never exposed through public `llama.h`, so it carries no ABI consequence for a consumer. The SYCL `top_k` files are picked up by ggml-sycl's own glob (`ggml-sycl/CMakeLists.txt` did not move), so the three `sycl-*` classifiers build unchanged, and the CUDA BF16 fallback only widens device support for `cuda13-*`. | +| b10948–b10969 | patches + upstream verification | **All nine patches apply, and none is droppable.** Following the b10933–b10934 precedent the whole set was applied against the **target** tag up front — `git apply -p1` of all nine, in filename order, against a pristine `b10976` worktree, every one clean — and then independently against pristine `b10969` so this intermediate commit is a valid bisect point rather than an untested waypoint. Two patch targets move in this chunk and both are harmless: `src/llama-model.cpp` (`0012`) gains three additive lines at the `LLM_ARCH_MAPLE` dispatch and rope-type switch, far from the `load_tensors` split arithmetic the patch rewrites, and `tests/CMakeLists.txt` (`0012`) *loses* the `test-peg-parser` PCH line near the peg-parser block, nowhere near the patch's `llama_build_and_test(test-model-split.cpp)` registration. Everything else the nine patches touch — `common/arg.{cpp,h}`, `common/peg-parser.cpp`, all of `tools/server/*.cpp`, and the ~34 standalone `main()` call sites — is **untouched by the entire b10948→b10976 range**. **#28776 is the direct sequel to the previous bump's `0013` drop**: having merged this project's #28775, upstream reverted their temporary in-tree carry of it and added a **non-VXE s390x build to their own CI** (`.github/workflows/build-ibm.yml`), plus the one `UNUSED(nb)` in `ggml/src/ggml-cpu/arch/s390/quants.c` that the scalar path exposed — so the scalar, `-DGGML_NATIVE=OFF` configuration `build-linux-s390x` builds is now guarded upstream as well as here, which is the outcome the `0013` note in `CLAUDE.md` was written to anticipate. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index bf0f1a1f..06d611ef 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10948 + GIT_TAG b10969 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index c5c1e493..38f45486 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10948"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10969"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10948-"} — call + * plus the resolved upstream commit, e.g. {@code "b10969-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10948"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10969"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10948"; + public static final String LLAMA_CPP_VERSION = "b10969"; // Constants holder — not instantiable. private LlamaCppVersion() {} From a15dbb3158dc5ed9c5c070d58d83efc4b3370370 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 15 Sep 2026 08:12:57 +0000 Subject: [PATCH 2/6] feat: upgrade llama.cpp from b10969 to b10976 Final chunk of the two-step walk, reaching the target release (20 KiB, 7 commits). Nothing under common/, include/, src/ or tools/server/*.{cpp,h}. #28771 retargets five CMake path references from CMAKE_SOURCE_DIR to PROJECT_SOURCE_DIR and extends examples/test-cmake to cover subproject consumption. That is exactly the bug class that bites a FetchContent subproject like this one -- CMAKE_SOURCE_DIR resolves to the top-level project, here llama/, not llama.cpp's own root -- but none of the five repaired files is processed by this build (LLAMA_BUILD_TOOLS/SERVER/APP are forced OFF, tests/ and examples/ are never added, and tools/mtmd was already correct). Inert today, one less latent trap tomorrow. Backend work is classifier-scoped and additive in effect: #28897 widens CUDA GGML_OP_DUP to i16/i32 for cuda13-*, and #28576 switches HIP flash-attention MMA to fp32 accumulation on AMD MFMA hardware, which no GitHub-hosted runner has -- the rocm-* jobs are build-only by design. #28646 is WebUI, auto-followed by build-webui from the pinned GIT_TAG. The patch set is unchanged at nine: nothing dropped, nothing refreshed. All nine apply into a pristine b10976 worktree, and every standing drop-check still reports "still required" against the pristine tag, which the fail-loud applier cannot do for itself (it detects "does not apply", never "upstream already fixed this"). Verified end to end: fresh configure through the real FetchContent path, stamp at head 987498f4592a76897863cf53711dce38380c082b with nine SHA-256 lines; wire-name extraction unchanged at 138 CLI / 57 request / 15 trainer; full Release build clean with zero errors; ctest 537/537; nm -D shows 40 Java_* exports and 0 mangled; NativeLibraryLoadSmokeTest 4/4 with 0 skipped, cross-validating the bumped LLAMA_CPP_VERSION constant against the linked build-info. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Cft7guQngfyKycdJfBEfJP --- CLAUDE.md | 8 ++++---- README.md | 2 +- docs/history/llama-cpp-breaking-changes.md | 2 ++ llama/CMakeLists.txt | 2 +- .../java/net/ladenthin/llama/value/LlamaCppVersion.java | 8 ++++---- 5 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index c911271a..929e8480 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co Java bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI. -Current llama.cpp pinned version: **b10969** +Current llama.cpp pinned version: **b10976** ## Upgrading CUDA Version @@ -502,7 +502,7 @@ needs no extra step here, `build-webui` re-reads the tag and rebuilds the matchi ships no UI): ```bash # needs node/npm + network for the asset build; the embed step is plain cmake -P -git clone --depth 1 --branch b10969 https://github.com/ggml-org/llama.cpp /tmp/lc +git clone --depth 1 --branch b10976 https://github.com/ggml-org/llama.cpp /tmp/lc ( cd /tmp/lc/tools/ui && npm ci && npm run build ) mkdir -p webui-generated /tmp/ui-gen cmake -DUI_SOURCE_DIR=/tmp/lc/tools/ui -DUI_BINARY_DIR=/tmp/ui-gen \ @@ -542,7 +542,7 @@ cache lives in **Depot Cache** over sccache's **WebDAV** backend: - `SCCACHE_WEBDAV_TOKEN: ${{ secrets.DEPOT_TOKEN }}` — a Depot **organization** token, stored as the repo secret **`DEPOT_TOKEN`**. -Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10969`), the +Because `sccache` is **content-addressed** and llama.cpp is pinned (`GIT_TAG b10976`), the ~280 upstream object files are byte-identical every run, so a warm cache recompiles only the *changed* files. Depot's cache is **shared across all branches** (unlike GitHub's per-branch `actions/cache`), so every branch builds incrementally; a `b` version bump @@ -1562,7 +1562,7 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" #### Upstream source location (in CMake build tree) -llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10969`. +llama.cpp is fetched via CMake FetchContent, pinned to `GIT_TAG b10976`. **GoogleTest** is a separate `BUILD_TESTING`-only FetchContent (`GIT_TAG v1.17.0`), used solely by the `jllama_test` C++ unit-test binary — not by the shipped library, and not coupled to the diff --git a/README.md b/README.md index a3287720..9f336ef4 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ **Build:** ![Java 8+](https://img.shields.io/badge/Java-8%2B-informational) ![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20macOS%20%7C%20Windows%20%7C%20Android-lightgrey) -[![llama.cpp b10969](https://img.shields.io/badge/llama.cpp-%23b10969-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10969) +[![llama.cpp b10976](https://img.shields.io/badge/llama.cpp-%23b10976-informational)](https://github.com/ggml-org/llama.cpp/releases/tag/b10976) [![JPMS](https://img.shields.io/badge/JPMS-modular%20JAR-25A162)](https://openjdk.org/projects/jigsaw/) ![JUnit](https://img.shields.io/badge/tested%20with-JUnit6-25A162) [![JSpecify](https://img.shields.io/badge/JSpecify-1.0.0%20%40NullMarked-25A162)](https://jspecify.dev) diff --git a/docs/history/llama-cpp-breaking-changes.md b/docs/history/llama-cpp-breaking-changes.md index 7a16b56b..72c88810 100644 --- a/docs/history/llama-cpp-breaking-changes.md +++ b/docs/history/llama-cpp-breaking-changes.md @@ -730,3 +730,5 @@ Used during `llama.cpp` version bumps: when upgrading, scan this file from the r | b10938–b10948 | patches + upstream verification | **Nine patches now, not ten: `0013` was dropped here.** Upstream merged this project's own PR [ggml-org/llama.cpp#28775](https://github.com/ggml-org/llama.cpp/pull/28775) ("ggml-cpu(s390x): guard VXE-only repack helpers", commit `6978052`, first tagged at **b10948**), so `ggml/src/ggml-cpu/arch/s390/repack.cpp` now wraps `vxe_dot_acc` / `vxe_splat_granule` / `vxe_fold` in the `#if defined(__VXE__) || defined(__VXE2__)` guard itself — byte-identical to the patch apart from the trailing `// __VXE__ || __VXE2__` comment the patch put on its `#endif`. **That is the designed signal, and the response is delete, not refresh** (the `0009` precedent at b10280): `git apply -p1` of `0013` against pristine b10948 fails with `patch failed: ggml/src/ggml-cpu/arch/s390/repack.cpp:70 … does not apply`, because the `#if` line it wants to insert is already there. The rationale that outlives the patch — why `build-linux-s390x` is a *scalar* (non-VXE) cross build, and why `-DGGML_VXE=ON` is the wrong response to a future VXE compile error — moved into the "`0013` was dropped at the b10948 bump" note in `CLAUDE.md`, which the job's own comment in `publish.yml` now points at. **The other three standing drop-checks still say "still required"**, run against the pristine tag because the fail-loud applier detects "does not apply" but never "upstream already fixed this": `0001` (`common_params_parse_main` 0 occurrences in `b10948:common/arg.h`; the WIN32 `argv = utf8.ptrs.data()` override still in `common/arg.cpp`), `0010` (`{"vocab_type", meta.model_vocab_type}` still uncast at `b10948:tools/server/server-context.cpp:4554`), `0012` (bare `splits[i] /= split_sum;` still at `b10948:src/llama-model.cpp:1491`, no zero-sum guard). **The s390x half was re-verified with the real cross toolchain rather than by inspection**, because that path has no runnable guard beyond the CI job: `s390x-linux-gnu-g++` (13.2.0) compiles pristine `b10948:ggml/src/ggml-cpu/arch/s390/repack.cpp` clean **both** with the job's own scalar flags (the `-DGGML_NATIVE=OFF` configuration, no `-mvx -mzvector`) **and** with `-mvx -mzvector -march=z15` — upstream's guard covers exactly what the patch covered and forecloses nothing. Verified end-to-end for real: `rm -rf build` then `cmake -B build -DBUILD_TESTING=ON` through the real `FetchContent` path, configure clean, stamp at head `5f436dddb440a288ee5611d7d1eca564a6aca9f4` (= `b10948`) with **nine** SHA-256 lines; extraction unchanged at **138 CLI / 57 request / 15 trainer** names; full `cmake --build --config Release` clean; `ctest` **537/537**; `nm -D` **40** `Java_*` exports, **0** mangled; `mvn -pl llama clean test -Dtest=NativeLibraryLoadSmokeTest` **4/4, 0 skipped** — which is the check that cross-validates the bumped `LLAMA_CPP_VERSION` constant against the linked `build-info`, and needs the `clean` because the constant is inlined into the already-compiled test class. | | b10948–b10969 | 21 commits, **99 KiB** — identical excluding `tools/ui`, because the range does not touch the WebUI at all. The headline is a **reversal**: **#28892** ("cmake : remove precompiled headers") deletes every `target_precompile_headers` call from `common/`, `src/`, `tests/`, `tools/mtmd/` and `tools/server/`, and **#28882** additionally disables the ggml-cpu PCH and makes `CACHE_LINE_SIZE` include-order independent. Also here: **#28749** `common/common.cpp` (hoist the `llama_n_rs_seq` check above the probe decode in `common_context_can_seq_rm`), **#27000** a new `LLM_ARCH_MAPLE` (Maple 20B-A1B ternary MoE, CPU) with `src/models/maple.cpp`, `conversion/maple.py`, `gguf-py` constants, a `llama-graph.cpp` swiglu-clamp arm and NEOX rope; model-loader correctness (**#28868** `get_key_or_arr` misuse across gemma4 / gemma4-assistant / mimo2 / step35 / qwen4exp, **#28865** mimo2 SWA pattern, **#28896** qwen4exp rms_norm+mul fusion); **#28846** ggml-cuda F32 fallback on devices without BF16 hardware acceleration; **#28670** a SYCL radix `top_k` (new `ggml-sycl/topk-radix.{cpp,hpp}`, +555); **#26885** a `llama_grammar_advance_stack` find+insert coalesce; **#28776** s390x (see the paired row); a new upstream `scripts/check-apiabi-compat.sh` (**#28579**, not consumed here); plus upstream CI/release workflow churn, `ci/run.sh`, and the llama.cpp 0.4.1 / ggml 0.24.0 version bumps. 46 files, 1409 insertions, 273 deletions. | **No project source change, and no priority-list *header* moved** — `common/common.cpp` is the one priority-list file touched and only its implementation. Zero `tools/server/*.{cpp,h}`: `server-schema.cpp`, `server-task.cpp`, `server-context.cpp`, `server-common.h`, `server-task.h` and `server-schema.h` are **byte-identical by blob hash** across the whole b10948→b10976 walk, so the three mechanical server-contract greps have **no input to compare** — the request-field set, its `set_hard_limits` bounds and the emitted response keys cannot have moved. **The PCH removal is a heap-buffer-overflow fix this project shipped**, not housekeeping: PCH arrived with #28091 at **b10917**, so every pin from b10919 through b10948 carried it. Mechanism (upstream #28858) — the PCH force-includes `ggml-impl.h` before `ops.h`, which pulls `` and defines `__cpp_lib_hardware_interference_size`, so the **C++** kernels resolved `CACHE_LINE_SIZE` to `std::hardware_destructive_interference_size` while `ggml-cpu.c`, a **C** TU where that feature macro can never be defined, kept the 64 fallback; the rope work buffer is sized by the C side and written by the C++ side, overflowing it by `(CACHE_LINE_SIZE/4 - 16) * n_threads * 4` bytes and crashing later in `ggml_compute_forward_rope_flt`. Exposure was per-job, not universal: the deleted ggml-cpu guard skipped the PCH for **GCC on non-x86**, so `build-linux-s390x` (cross g++) and the aarch64-native GCC 14 job were never affected, while the clang / AppleClang / clang-cl / MSVC and GCC-on-x86 jobs were — and only where that toolchain's `hardware_destructive_interference_size` actually differs from 64. **On the build-system side all four changes that reach this build are pure removals**, so nothing had to be added here: `tools/server/CMakeLists.txt` is *not* processed at all (`LLAMA_BUILD_TOOLS`/`LLAMA_BUILD_SERVER` are forced OFF and the server TUs are compiled straight into `jllama`), while `common/`, `src/`, `tools/mtmd/` and `ggml/src/ggml-cpu/` are — and dropping PCH also removes the one thing that could have interfered with the sccache-over-Depot compiler launcher. #28749 changes only *when* `common_context_can_seq_rm` answers: for a recurrent/hybrid context it now returns `COMMON_CONTEXT_SEQ_RM_TYPE_RS` before the `llama_memory_clear` + throwaway `llama_decode` probe instead of after it — same enum, same signature, one fewer decode; the consumer is `server-context.cpp`'s `ctx_tgt_seq_rm_type`, compiled into `libjllama`. `LLM_ARCH_MAPLE` is a **mid-enum insert**, but `llm_arch` lives in `src/llama-arch.h` and is never exposed through public `llama.h`, so it carries no ABI consequence for a consumer. The SYCL `top_k` files are picked up by ggml-sycl's own glob (`ggml-sycl/CMakeLists.txt` did not move), so the three `sycl-*` classifiers build unchanged, and the CUDA BF16 fallback only widens device support for `cuda13-*`. | | b10948–b10969 | patches + upstream verification | **All nine patches apply, and none is droppable.** Following the b10933–b10934 precedent the whole set was applied against the **target** tag up front — `git apply -p1` of all nine, in filename order, against a pristine `b10976` worktree, every one clean — and then independently against pristine `b10969` so this intermediate commit is a valid bisect point rather than an untested waypoint. Two patch targets move in this chunk and both are harmless: `src/llama-model.cpp` (`0012`) gains three additive lines at the `LLM_ARCH_MAPLE` dispatch and rope-type switch, far from the `load_tensors` split arithmetic the patch rewrites, and `tests/CMakeLists.txt` (`0012`) *loses* the `test-peg-parser` PCH line near the peg-parser block, nowhere near the patch's `llama_build_and_test(test-model-split.cpp)` registration. Everything else the nine patches touch — `common/arg.{cpp,h}`, `common/peg-parser.cpp`, all of `tools/server/*.cpp`, and the ~34 standalone `main()` call sites — is **untouched by the entire b10948→b10976 range**. **#28776 is the direct sequel to the previous bump's `0013` drop**: having merged this project's #28775, upstream reverted their temporary in-tree carry of it and added a **non-VXE s390x build to their own CI** (`.github/workflows/build-ibm.yml`), plus the one `UNUSED(nb)` in `ggml/src/ggml-cpu/arch/s390/quants.c` that the scalar path exposed — so the scalar, `-DGGML_NATIVE=OFF` configuration `build-linux-s390x` builds is now guarded upstream as well as here, which is the outcome the `0013` note in `CLAUDE.md` was written to anticipate. | +| b10969–b10976 | 7 commits, **20 KiB** — the final chunk, reaching the target release. Nothing under `common/`, `include/`, `src/` or `tools/server/*.{cpp,h}` at all. **#28771** ("cmake : use `PROJECT_SOURCE_DIR` instead of `CMAKE_SOURCE_DIR`") retargets five path references — `tools/server/CMakeLists.txt` (×2), `app/`, `tools/tuning/`, `examples/eval-callback/` and `tests/` — and extends `examples/test-cmake` to cover consumption as a **subproject**. Backend work: **#28897** CUDA enables `i16`/`i32` for `GGML_OP_DUP` (new `int16_t` arm in `cpy.cu`; `ggml_backend_cuda_device_supports_op` now returns `true` unconditionally for `DUP` instead of excluding those two types), and **#28576** HIP switches flash-attention MMA to **fp32 accumulation on MFMA devices** (`fattn-mma-f16.cuh`: `T_C_VKQ` becomes `tile<16,16,float>`, MFMA gets its own `VKQ_C` extent separate from WMMA, and one `GGML_CUDA_FATTN_MMA_CONFIG_CASE` drops 4→3 warps), with **#28909** relaxing upstream's own HIP spill check for it. The rest is **#28646** (WebUI: stop re-probing a disabled `/tools` endpoint on every message) and upstream CI (**#28911**, **#28936**), plus `docs/ops.md` + `docs/ops/CUDA.csv` regeneration. 22 files, 88 insertions, 42 deletions. | **No project source change and nothing on the priority review list** — the chunk touches no header on it and no implementation behind one. The `CMAKE_SOURCE_DIR` → `PROJECT_SOURCE_DIR` fix is precisely the bug class that bites a **FetchContent subproject** like this one (`CMAKE_SOURCE_DIR` resolves to the *top-level* project — here `llama/` — not llama.cpp's own root), but none of the five files it repairs is processed by this build: `LLAMA_BUILD_TOOLS`/`LLAMA_BUILD_SERVER`/`LLAMA_BUILD_APP` are forced OFF and the server TUs are compiled straight into `jllama`, `tests/` and `examples/` are never added, and `tools/mtmd` — the one subdirectory this build *does* add explicitly — was already correct. So the fix is inert here today and removes a latent trap tomorrow; upstream's new `examples/test-cmake` subproject coverage makes a regression of it their CI's problem rather than this project's. The two backend changes are classifier-scoped and additive in effect: the CUDA `DUP` widening only adds accepted types for `cuda13-*`, and the HIP fp32 accumulation is a numerical-accuracy change confined to AMD **MFMA** hardware, which no GitHub-hosted runner has — the `rocm-*` jobs are build-only, so CI proves it compiles and nothing more, as designed. The WebUI commit is auto-followed: `build-webui` rebuilds the Svelte UI from the pinned `GIT_TAG`, so it needs no action here. | +| b10969–b10976 | patches + upstream verification | End of the two-chunk walk, and **the patch set is unchanged at nine** — nothing dropped, nothing refreshed. Verified against the pristine target rather than inferred: `git apply -p1` of all nine, in filename order, into a clean `b10976` worktree, every one clean. **All standing drop-checks still say "still required"**, run against the pristine tag because the fail-loud applier detects "does not apply" but never "upstream already fixed this": `0001` (`common_params_parse_main` 0 occurrences in `b10976:common/arg.h`; the WIN32 `argv = utf8.ptrs.data()` override still at `common/arg.cpp:1282`), `0010` (`{"vocab_type", meta.model_vocab_type}` still uncast at `b10976:tools/server/server-context.cpp:4554`), `0012` (bare `splits[i] /= split_sum;` still at `b10976:src/llama-model.cpp:1493`, no zero-sum guard), `0002` (`params_base.load_progress_callback = load_progress_callback;` still unguarded at `server-context.cpp:1095`), and `0003`/`0006`/`0008` (`get_slot_prompt_similarity`, `llama_server_set_embedded`/`llama_server_attach` and `LLAMA_SERVER_WORKER_CMD` all absent upstream). The whole b10948→b10976 range leaves **every** patch target except `src/llama-model.cpp` and `tests/CMakeLists.txt` byte-identical, and both of those move only at a distance from the patched regions. Verified end-to-end for real: `rm -rf build` then `cmake -B build -DBUILD_TESTING=ON` through the real `FetchContent` path, configure clean, stamp at head `987498f4592a76897863cf53711dce38380c082b` (= `b10976`) with **nine** SHA-256 lines; extraction unchanged at **138 CLI / 57 request / 15 trainer** names; full `cmake --build --config Release` clean with **zero** errors; `ctest` **537/537**; `nm -D` **40** `Java_*` exports, **0** mangled; `mvn -pl llama clean test -Dtest=NativeLibraryLoadSmokeTest` **4/4, 0 skipped** — the check that cross-validates the bumped `LLAMA_CPP_VERSION` constant against the linked `build-info`, needing the `clean` because the constant is inlined into the already-compiled test class. | diff --git a/llama/CMakeLists.txt b/llama/CMakeLists.txt index 06d611ef..0077bd6b 100644 --- a/llama/CMakeLists.txt +++ b/llama/CMakeLists.txt @@ -173,7 +173,7 @@ set(LLAMA_BUILD_APP OFF CACHE BOOL "" FORCE) FetchContent_Declare( llama.cpp GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git - GIT_TAG b10969 + GIT_TAG b10976 PATCH_COMMAND ${CMAKE_COMMAND} -DPATCH_DIR=${CMAKE_CURRENT_SOURCE_DIR}/patches -DLLAMA_SRC= diff --git a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java index 38f45486..2fe79fe9 100644 --- a/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java +++ b/llama/src/main/java/net/ladenthin/llama/value/LlamaCppVersion.java @@ -10,13 +10,13 @@ * library was compiled against, exposed as a compile-time constant so callers can render a badge or * emit a startup log line without loading the native library. * - *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10969"}) that mirrors the + *

{@link #LLAMA_CPP_VERSION} is a pure-Java string ({@code "b10976"}) that mirrors the * {@code GIT_TAG} in {@code llama/CMakeLists.txt}. It is available even when {@code libjllama} is * absent (pure-Java checkout, before {@code System.load}), which is what makes it suitable for a * lightweight version badge in Android or other UIs.

* *

For the authoritative value that is baked into the native binary — the build number - * plus the resolved upstream commit, e.g. {@code "b10969-"} — call + * plus the resolved upstream commit, e.g. {@code "b10976-"} — call * {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} instead; that reads llama.cpp's own * {@code build-info} through JNI and therefore cannot drift from the compiled library (but requires * the native library to be loaded).

@@ -24,14 +24,14 @@ public final class LlamaCppVersion { /** - * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10969"}. + * The pinned llama.cpp release tag this library was built against, e.g. {@code "b10976"}. * *

Kept in lockstep with {@code GIT_TAG} in {@code llama/CMakeLists.txt} — see the * "Upgrading/Downgrading llama.cpp Version" checklist in {@code CLAUDE.md}. This is the * compile-time pin; use {@link net.ladenthin.llama.LlamaModel#getLlamaCppBuildInfo()} for the * value actually linked into the native binary.

*/ - public static final String LLAMA_CPP_VERSION = "b10969"; + public static final String LLAMA_CPP_VERSION = "b10976"; // Constants holder — not instantiable. private LlamaCppVersion() {} From 46e1714e4485764d7f08bc04199c6d97bec2fc09 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 15 Sep 2026 16:36:22 +0000 Subject: [PATCH 3/6] fix: stop C++ exceptions crossing the JNI boundary at every entry point An exception that escapes a native method and unwinds across the JNI boundary is undefined behaviour and aborts the JVM on most implementations. Before this change, of the 40 Java_* entry points only ONE had a catch-all: jllama.cpp had 34 with a single catch (...) between them, native_server.cpp's 5 had no handler of any kind, and train_engine.cpp's finetuneNative guarded only its finetune() call, not the JSON config parse (whose own handler had no catch (...)) nor the GetStringUTFChars copy before it. load_model_impl -- the entire model load path -- and handleDetokenize were fully unguarded. Adds jni_guard_impl to jni_helpers.hpp (Layer A) and routes all 39 wrappable entry points through it. It is additive: an entry point that already converts std::exception keeps doing so and never reaches the new handlers. What it adds everywhere is the catch (...) arm, which is the only backstop for an exception type not derived from std::exception. On a catch it returns the zero/nullptr sentinel for the return type. Two refusals the handler keeps, both pinned by tests: never ThrowNew over an already-pending Java exception (the JNI spec forbids most calls in that state, and the pending one is the more precise error), and never ThrowNew with a null class. Three entry points use a function-try-block instead, which also avoids reindenting a goto-carrying body: JNI_OnLoad runs before c_llama_error is cached and JNI_OnUnload after it is released, so neither has a class to throw with -- OnLoad returns JNI_ERR (surfaced as UnsatisfiedLinkError), OnUnload swallows. finetuneNative reports failure as its return string rather than a Java exception, and train_engine.cpp deliberately never includes jni_helpers.hpp, so its backstop preserves that contract. The jllama.cpp diff is large because every body gained one indent level. It is provably mechanical: a token-level comparison against the previous file removes ZERO tokens -- every added token run is the wrapper or a function-try-block. native_server.cpp's 14 removed tokens are exactly the inlined FindClass that moved into the new llama_exception_class helper; its two unnamed JNIEnv* parameters had to be named to reach env. Verified: clang-format 22.1.8 clean, full Release build with 0 errors and 0 warnings, ctest 544/544 (537 + 7 new guard tests), nm -D still shows 40 Java_* exports and 0 mangled, NativeLibraryLoadSmokeTest 4/4 with 0 skipped (JNI_OnLoad still loads after the shape change). Also removes the PR #251 "JNI safety and server hardening" TODO section: its other three items were already fixed (parse_string_array has null guards, an ExceptionCheck and paired DeleteLocalRef; close() is synchronized; OpenAiServerConfig has maxRequestBodyBytes), and this change closes the fourth. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Cft7guQngfyKycdJfBEfJP --- CLAUDE.md | 36 +- TODO.md | 32 - llama/src/main/cpp/jllama.cpp | 1495 ++++++++++++----------- llama/src/main/cpp/jni_helpers.hpp | 47 + llama/src/main/cpp/native_server.cpp | 190 +-- llama/src/main/cpp/train_engine.cpp | 10 +- llama/src/test/cpp/test_jni_helpers.cpp | 125 ++ 7 files changed, 1102 insertions(+), 833 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 929e8480..c2338787 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1384,6 +1384,38 @@ Functions with `_impl` suffix are called directly from `jllama.cpp`. - If it needs upstream server types, put it in Layer B (after the `json_helpers.hpp` include). - Add tests to `src/test/cpp/test_jni_helpers.cpp`. +### The JNI exception boundary — `jni_guard_impl` + +An exception that escapes a native method and unwinds across the JNI boundary is **undefined +behaviour and aborts the JVM** on most implementations. **Every `Java_*` entry point must therefore +convert anything that escapes into a Java exception**, and there are 40 of them across three TUs — +`jllama.cpp` (34), `native_server.cpp` (5), `train_engine.cpp` (1). + +The mechanism is `jni_guard_impl(env, exception_class, [&]() -> Ret { … })` (`jni_helpers.hpp`, +Layer A). It is **additive**: an entry point that already converts `std::exception` itself keeps +doing so and never reaches the guard's handlers. What the guard adds everywhere is the +**`catch (...)` arm** — the case for an exception type not derived from `std::exception`, which +otherwise has no backstop at all. On a catch it returns the zero/`nullptr` sentinel for the entry +point's return type. + +Two rules the handler keeps, both pinned by tests in `test_jni_helpers.cpp`: + +- **Never `ThrowNew` over an already-pending Java exception.** The JNI spec forbids most calls in + that state, and the pending exception is the more precise error — so it is left in place. +- **Never `ThrowNew` with a null class.** + +**Three entry points are deliberately NOT routed through it, and each uses a function-try-block +instead** (which also avoids reindenting a `goto`-carrying body): `JNI_OnLoad` runs before +`c_llama_error` is cached and `JNI_OnUnload` after it is released, so neither has a class to throw +with — `JNI_OnLoad` returns `JNI_ERR` (the JVM surfaces that as `UnsatisfiedLinkError`) and +`JNI_OnUnload` swallows. `LlamaTrainer_finetuneNative` reports failure as its **return string** +rather than a Java exception, and `train_engine.cpp` deliberately keeps its own `nlohmann` alias +and never includes `jni_helpers.hpp`, so its backstop returns an error string to preserve that +contract. + +**When you add a native method, wrap it.** The guard is not enforced by a test — a new unguarded +entry point is invisible until something throws through it in production. + ### Parameter Flow Java parameters are serialized to JSON strings and passed to native code, which deserializes them using nlohmann/json. This avoids complex JNI field mapping for the many llama.cpp parameters. @@ -1551,14 +1583,14 @@ ctest --test-dir build --output-on-failure -R "ResultsToJson" | `src/test/cpp/test_server.cpp` | 206 | Upstream result types: `server_slot_stats` (the `timings` JSON payload; replaced `result_timings` in b10408), `task_params::to_json()` (incl. `dry_sequence_breakers`, `preserved_tokens`, `timings_per_token`), `completion_token_output`, `server_task_result_cmpl_partial` (non-oaicompat + `to_json_oaicompat` + logprobs + `to_json_oaicompat_chat` + `to_json_anthropic` + dispatcher), `server_task_result_cmpl_final` (non-oaicompat + `to_json_oaicompat` + `to_json_oaicompat_chat` + `to_json_oaicompat_chat_stream` + `to_json_anthropic` + `to_json_anthropic_stream` + tool_calls + dispatcher), `server_task_result_embd`, `server_task_result_rerank`, `server_task_result_metrics` (`to_metrics()` = the `/metrics` Prometheus exposition text; its `to_json()` has been unused since b10519 and returns `json{}` = JSON null), `server_task_result_slots` (`to_json()` = the `/slots` array, fed by the b10519 `SERVER_TASK_TYPE_SLOT_GET` task), `server_task_result_slot_save_load`, `server_task_result_slot_erase`, `server_task_result_apply_lora`, `server_task_result_get_lora`, `server_task_result_error`, `format_error_response`, `server_task::need_sampling()`, `server_task::n_tokens()`, `server_schema::eval_llama_cmpl_schema()` (parsing pipeline + grammar routing + error paths + per-request `dry_*` and `sse_ping_interval` field round-trips incl. hard-limit + server-default inheritance), `response_fields` projection | | `src/test/cpp/test_json_helpers.cpp` | 63 | All functions in `json_helpers.hpp`: `get_result_error_message`, `results_to_json`, `rerank_results_to_json` (incl. missing/out-of-range `index` rejection), `parse_encoding_format`, `extract_embedding_prompt`, `is_infill_request`, `parse_slot_prompt_similarity`, `parse_positive_int_config`, `wrap_stream_chunk`, `server_metrics_to_json` | | `src/test/cpp/test_log_helpers.cpp` | 13 | All functions in `log_helpers.hpp`: `log_level_name`, `format_log_as_json` | -| `src/test/cpp/test_jni_helpers.cpp` | 56 | All functions in `jni_helpers.hpp` using a zero-filled `JNINativeInterface_` mock (incl. the `utf8_to_jstring_impl` byte-array string path: emoji byte-preservation, truncated-UTF-8 replace-not-throw) | +| `src/test/cpp/test_jni_helpers.cpp` | 63 | All functions in `jni_helpers.hpp` using a zero-filled `JNINativeInterface_` mock (incl. the `utf8_to_jstring_impl` byte-array string path: emoji byte-preservation, truncated-UTF-8 replace-not-throw). The last 7 pin `jni_guard_impl` — the JNI exception boundary every `Java_*` entry point runs inside — including the `catch (...)` arm that is the only backstop for a non-`std::exception` type, and its two refusals (never `ThrowNew` over a pending Java exception, never with a null class). | | `src/test/cpp/test_tts_wav.cpp` | 2 | The in-memory WAV writer `pcm_to_wav16_bytes` in `tts_wav.hpp` (WAV header/payload + little-endian clamping) — our own code, not upstream. The Qwen3-TTS pipeline it pairs with (`mtmd_helper::gen_audio`) is entirely upstream-owned (no project-side DSP to unit-test here). The load path is additionally covered by `test_tts_params.cpp` (3 tests over `tts_params.hpp`'s `build_tts_params`, plus 2 pinning the upstream `-1` default it depends on), which pins the CPU-thread resolution whose absence used to crash the JVM on every platform — see the `TODO.md` entry for the mechanism. End-to-end coverage is `TtsIntegrationTest`, which is model-gated. | | `src/test/cpp/test_tts_params.cpp` | 13 | The **three** builders every hand-assembled `common_params` goes through: `build_tts_params` (`tts_params.hpp`), `build_train_params` (`train_params.hpp`) and the shared `jllama::resolve_cpu_params` (`cpu_params.hpp`). Each builder is guarded separately on purpose — testing the resolver alone does **not** cover its call sites, because `train_engine.cpp` is compiled into `jllama` only, never into `jllama_test`, and `LlamaTrainerIntegrationTest` is gated on `net.ladenthin.llama.train.model`, which no CI job sets. Without these the JVM-abort bug could regress in the trainer on every platform, unseen. | | `src/test/cpp/test_model_split.cpp` | 7 | The two `load_tensors()` split helpers that `patches/0012` extracts out of llama.cpp's `src/llama-model.cpp` — `llama_model_splits_normalize` (proportional split, single device, and the zero-sum case that used to produce NaN, **and the cancelling `--tensor-split` case** — `-ts 1,-1` reaches the identical line on any backend with no GPU memory pressure at all) and `llama_model_splits_select_device` (every layer maps to a real device index; malformed split points throw a message that names the function, the layer, the index and the split values instead of libc++'s bare `"vector"`). **This is the runnable guard for `0012`**: the patch also ships an upstream `tests/test-model-split.cpp`, but a FetchContent subproject builds with `LLAMA_BUILD_TESTS=OFF`, so that one is applied-but-never-compiled here. This file is the only place the two functions are linked in CI, on every platform — so a bump that drops the patch fails the `C++ Tests` build outright rather than resurfacing as one red macOS Java job. It is the one test file that includes an **internal** upstream header (`llama-model.h`, via the `${llama.cpp_SOURCE_DIR}/src` include dir added for it), which is deliberate: a signature drift should fail loudly at compile time. | | `src/test/cpp/test_model_flags.cpp` | 4 | **The contract between the Java CLI-flag registries and llama.cpp's server argument parser.** CMake reads `ModelFlag.java` + `ModelOption.java` (`cmake/extract-java-wire-names.cmake` → a generated header of `{name, contract}` pairs), and this file asserts every `SERVER_PARSER` name is in `common_params_parser_init(params, LLAMA_EXAMPLE_SERVER).options`. It exists because **no Java test can catch this class**: `ModelFlagTest`/`ModelParametersExtendedTest` pin the *string mapping* (`hasKey("--mlock")`), never that llama.cpp still accepts the string, so they stay green forever while the flag is dead — and `common_params_parse` treats an unregistered option as a hard error, so the affected builder method makes the model **unloadable**, not merely ineffective. **A grep over `arg.cpp` is not a substitute**: `--grp-attn-n`/`-w` are present there at every pinned tag but `set_examples()`-scoped to `LLAMA_EXAMPLE_COMPLETION`/`PASSKEY`, so the server parser rejects them exactly like a deleted flag — only the real option table sees that. `--vocab-only` is the one exemption, and it declares itself `CliContract.PROJECT_PSEUDO` on its own constant rather than appearing in a list inside this file; the test asserts such a name is **still unknown** to the parser (an exemption upstream later registers would be hiding a real check) and that the exempt set is non-empty. | | `src/test/cpp/test_wire_contracts.cpp` | 6 | **The same contract for the two quieter surfaces.** `RequestField` against `server_schema::make_llama_cmpl_schema(...)` (5 tests) and `TrainingField` against `jllama_train::config_keys()` (1 test). Both receivers *silently ignore* an unknown key — the schema skips it, `train_engine.cpp` reads with `j.value(key, default)` and falls back — so a dead field produces no error anywhere and every string-mapping test keeps passing. `OAI_LAYER`-declared keys (consumed by `oaicompat_*_params_parse` before the schema) are exempt from the schema check, and are checked **both** ways: still unknown to the schema (the inverted check), and read by at least one upstream reader-shaped site (the configure-time sweep — this is what caught `chat_template`, a key a public builder wrote and nothing read). See [`docs/history/parameter-wire-surface.md`](docs/history/parameter-wire-surface.md). | -**Current total: 537 tests (all passing).** +**Current total: 544 tests (all passing).** #### Upstream source location (in CMake build tree) diff --git a/TODO.md b/TODO.md index 1f1305fa..5ecd8318 100644 --- a/TODO.md +++ b/TODO.md @@ -219,38 +219,6 @@ These are JNI plumbing items for upstream API additions. Policy: add only after - **Expose `llama_vocab::get_suppress_tokens()` via `LlamaModel.getSuppressTokens()`.** Added in b9490–b9495 alongside the new `tokenizer.ggml.suppress_tokens` GGUF key and the `LLM_KV_TOKENIZER_SUPPRESS_TOKENS` constant. When a GGUF declares this array, upstream stores it on `llama_vocab::impl::suppress_tokens` and exposes it via the new `llama_vocab::get_suppress_tokens()` accessor. The bias is **applied automatically** inside the model forward graph — the Gemma4 Unified graph (`src/models/gemma4.cpp`) reads the list and adds a `-INFINITY` logit bias to those token IDs via a new `llm_graph_input_logits_bias` input so the model cannot emit them (used to block `` / `` placeholders). A Java mirror would be `public int[] getSuppressTokens()` on `LlamaModel`: a read-only inspector returning the suppression list for debugging or for callers running their own sampling who want to replicate the same bias. Value is low (the bias is auto-applied, Java callers cannot change it; java-llama.cpp does not expose custom logit-bias hooks at this level); cost is trivial (one JNI passthrough + a `getSuppressTokens()` Java method). -### JNI safety and server hardening (from PR #251 contributor) - -Raised by [@vaiju1981](https://github.com/vaiju1981) in -[PR #251 comment](https://github.com/bernardladenthin/java-llama.cpp/pull/251#issuecomment-4761363838). -Feel free to contribute fixes — PRs welcome. - -- **Unhandled C++ exceptions cross the JNI boundary → JVM abort (UB).** Any `std::exception` - (or worse, an exception of unknown type) that escapes a native method and crosses the JNI - boundary causes undefined behaviour on most JVMs and typically aborts the process. Each native - method in `jllama.cpp` should wrap its body in `try { … } catch (const std::exception& e) { - env->ThrowNew(llamaExceptionClass, e.what()); return ; } catch (...) { env->ThrowNew(…, - "unknown C++ exception"); return ; }` so that errors surface as `LlamaException` on the - Java side instead of crashing the JVM. - -- **`parse_string_array` — null deref + JNI local-reference leak.** The helper that reads a - JSON string array from JNI can dereference a null pointer when an array element is absent, - and leaks JNI local references when an early exit skips the matching `DeleteLocalRef`. Fix: - guard every `GetObjectArrayElement` result and pair each reference acquisition with a - `DeleteLocalRef` before the next iteration or return. - -- **`close()` / native `delete()` double-free under concurrent close.** If two threads race to - call `LlamaModel.close()`, both can reach the native `delete` path and free the same - `jllama_context` pointer twice → heap corruption. Fix: use `AtomicBoolean closed` + a - `synchronized` guard (or `compareAndSet`) on the Java side so `close()` is idempotent and - the native pointer is nulled before the second caller can reach it. - -- **Unbounded request-body read → OOM DoS.** The HTTP handler reads the entire request body - into a `String`/`byte[]` before parsing it, with no size cap. A client that streams a - multi-gigabyte body can exhaust heap memory and crash the JVM. Fix: add a configurable - `maxRequestBodyBytes` limit (e.g. default 4 MB) and reject oversized requests with - `HTTP 413 Content Too Large` before buffering them. - ### Feature backlog from similar projects (remainder: jbang example) The consolidated investigation lives in diff --git a/llama/src/main/cpp/jllama.cpp b/llama/src/main/cpp/jllama.cpp index 04049dfd..05d873be 100644 --- a/llama/src/main/cpp/jllama.cpp +++ b/llama/src/main/cpp/jllama.cpp @@ -711,7 +711,7 @@ std::string parse_jstring(JNIEnv *env, jstring java_string) { * only requires JNI version `JNI_VERSION_1_1`. If the VM does not recognize the version number returned by `JNI_OnLoad`, the VM will unload the library and act as if the library was never loaded. */ -JNIEXPORT jint JNICALL JNI_OnLoad(JavaVM *vm, void *reserved) { +JNIEXPORT jint JNICALL JNI_OnLoad(JavaVM *vm, void *reserved) try { g_vm = vm; JNIEnv *env = nullptr; @@ -819,6 +819,12 @@ JNIEXPORT jint JNICALL JNI_OnLoad(JavaVM *vm, void *reserved) { success: return JNI_VERSION_1_6; +} catch (...) { + // A function-try-block, not jni_guard_impl: this runs BEFORE c_llama_error is cached, so there + // is no class to throw into Java with. Refusing to load is the only correct outcome — the JVM + // surfaces it to the caller as an UnsatisfiedLinkError. (It is also the one shape that leaves + // the goto/label flow above untouched.) + return JNI_ERR; } /** @@ -829,7 +835,7 @@ JNIEXPORT jint JNICALL JNI_OnLoad(JavaVM *vm, void *reserved) { * Note that `JNI_OnLoad` and `JNI_OnUnload` are two functions optionally supplied by JNI libraries, not exported from * the VM. */ -JNIEXPORT void JNICALL JNI_OnUnload(JavaVM *vm, void *reserved) { +JNIEXPORT void JNICALL JNI_OnUnload(JavaVM *vm, void *reserved) try { JNIEnv *env = nullptr; if (JNI_OK != vm->GetEnv((void **)&env, JNI_VERSION_1_6)) { @@ -848,6 +854,9 @@ JNIEXPORT void JNICALL JNI_OnUnload(JavaVM *vm, void *reserved) { } llama_backend_free(); +} catch (...) { + // Mirrors JNI_OnLoad: the cached class references are being released here, so there is nothing + // to throw into Java. Unload is best-effort by definition — swallow rather than abort the JVM. } // Trampoline state for llama.cpp's load_progress_callback. The native loader runs @@ -995,13 +1004,13 @@ static void load_model_impl(JNIEnv *env, jobject obj, jobjectArray jparams, jobj } JNIEXPORT void JNICALL Java_net_ladenthin_llama_LlamaModel_loadModel(JNIEnv *env, jobject obj, jobjectArray jparams) { - load_model_impl(env, obj, jparams, nullptr); + return jni_guard_impl(env, c_llama_error, [&]() -> void { load_model_impl(env, obj, jparams, nullptr); }); } JNIEXPORT void JNICALL Java_net_ladenthin_llama_LlamaModel_loadModelWithProgress(JNIEnv *env, jobject obj, jobjectArray jparams, jobject callback) { - load_model_impl(env, obj, jparams, callback); + return jni_guard_impl(env, c_llama_error, [&]() -> void { load_model_impl(env, obj, jparams, callback); }); } // Build the special-token id map (a token is -1 / LLAMA_TOKEN_NULL when the model defines none). @@ -1013,141 +1022,149 @@ static json special_tokens_json(const llama_vocab *vocab) { } JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_getModelMetaJson(JNIEnv *env, jobject obj) { - REQUIRE_SERVER_CONTEXT(nullptr); - if (jctx->vocab_only) { - json meta = { - // static_cast: an unscoped enum binds to common_json_value(bool) and would serialise - // as true/false — see the "vocab_type" note in docs/history/llama-cpp-breaking-changes.md - {"vocab_type", static_cast(llama_vocab_type(jctx->vocab))}, - {"n_vocab", llama_vocab_n_tokens(jctx->vocab)}, - {"special_tokens", special_tokens_json(jctx->vocab)}, - }; - return json_to_jstring(env, meta); - } - auto m = wake_and_get_meta(jctx); - // Read general.architecture from GGUF metadata via the llama C API. Size the buffer - // dynamically: llama_model_meta_val_str returns the required length when given a null/0 - // buffer, so a long architecture name is never silently truncated. A char vector (not a - // std::string written in place) keeps the API's terminating '\0' write off the string's - // internal terminator slot, which the standard does not allow to be written. - std::string arch; - const llama_model *mdl = llama_get_model(ctx_server->get_llama_context()); - if (mdl) { - const int need = llama_model_meta_val_str(mdl, "general.architecture", nullptr, 0); - if (need > 0) { - std::vector buf(static_cast(need) + 1, '\0'); - llama_model_meta_val_str(mdl, "general.architecture", buf.data(), buf.size()); - arch.assign(buf.data()); + return jni_guard_impl(env, c_llama_error, [&]() -> jstring { + REQUIRE_SERVER_CONTEXT(nullptr); + if (jctx->vocab_only) { + json meta = { + // static_cast: an unscoped enum binds to common_json_value(bool) and would serialise + // as true/false — see the "vocab_type" note in docs/history/llama-cpp-breaking-changes.md + {"vocab_type", static_cast(llama_vocab_type(jctx->vocab))}, + {"n_vocab", llama_vocab_n_tokens(jctx->vocab)}, + {"special_tokens", special_tokens_json(jctx->vocab)}, + }; + return json_to_jstring(env, meta); } - } - json j = { - // static_cast: see the vocab_only branch above — an unscoped enum would become a boolean - {"vocab_type", static_cast(m.model_vocab_type)}, - {"n_vocab", m.model_vocab_n_tokens}, - {"n_ctx_train", m.model_n_ctx_train}, - {"n_embd", m.model_n_embd_inp}, - {"n_params", m.model_n_params}, - {"size", m.model_size}, - // All three modalities upstream tracks. `video` was omitted here while server_context_meta - // has carried it for releases, and upstream's own /props emits all three -- a consumer - // feature-detecting from getModelMeta() would have concluded no model ever accepts video. - {"modalities", {{"vision", m.has_inp_image}, {"audio", m.has_inp_audio}, {"video", m.has_inp_video}}}, - {"name", m.model_name}, - {"architecture", arch}, - {"ftype", m.model_ftype}, - }; - // Resolved default chat template (Jinja); empty when the model ships none. - const char *chat_tmpl = mdl != nullptr ? llama_model_chat_template(mdl, /*name*/ nullptr) : nullptr; - j["chat_template"] = chat_tmpl != nullptr ? std::string(chat_tmpl) : std::string(); - j["special_tokens"] = special_tokens_json(jctx->vocab); - // Full GGUF metadata key/value map. - if (mdl != nullptr) { - json meta_map = json::object(); - const int meta_count = llama_model_meta_count(mdl); - for (int i = 0; i < meta_count; i++) { - char key_buf[256] = {}; - // ponytail: 2 KB/value cap — scalar metadata fits; huge array values - // (tokenizer tokens/merges) truncate rather than bloating the JSON. - char val_buf[2048] = {}; - if (llama_model_meta_key_by_index(mdl, i, key_buf, sizeof(key_buf)) >= 0 && - llama_model_meta_val_str_by_index(mdl, i, val_buf, sizeof(val_buf)) >= 0) { - meta_map[std::string(key_buf)] = std::string(val_buf); + auto m = wake_and_get_meta(jctx); + // Read general.architecture from GGUF metadata via the llama C API. Size the buffer + // dynamically: llama_model_meta_val_str returns the required length when given a null/0 + // buffer, so a long architecture name is never silently truncated. A char vector (not a + // std::string written in place) keeps the API's terminating '\0' write off the string's + // internal terminator slot, which the standard does not allow to be written. + std::string arch; + const llama_model *mdl = llama_get_model(ctx_server->get_llama_context()); + if (mdl) { + const int need = llama_model_meta_val_str(mdl, "general.architecture", nullptr, 0); + if (need > 0) { + std::vector buf(static_cast(need) + 1, '\0'); + llama_model_meta_val_str(mdl, "general.architecture", buf.data(), buf.size()); + arch.assign(buf.data()); } } - j["metadata"] = std::move(meta_map); - } - return json_to_jstring(env, j); + json j = { + // static_cast: see the vocab_only branch above — an unscoped enum would become a boolean + {"vocab_type", static_cast(m.model_vocab_type)}, + {"n_vocab", m.model_vocab_n_tokens}, + {"n_ctx_train", m.model_n_ctx_train}, + {"n_embd", m.model_n_embd_inp}, + {"n_params", m.model_n_params}, + {"size", m.model_size}, + // All three modalities upstream tracks. `video` was omitted here while server_context_meta + // has carried it for releases, and upstream's own /props emits all three -- a consumer + // feature-detecting from getModelMeta() would have concluded no model ever accepts video. + {"modalities", {{"vision", m.has_inp_image}, {"audio", m.has_inp_audio}, {"video", m.has_inp_video}}}, + {"name", m.model_name}, + {"architecture", arch}, + {"ftype", m.model_ftype}, + }; + // Resolved default chat template (Jinja); empty when the model ships none. + const char *chat_tmpl = mdl != nullptr ? llama_model_chat_template(mdl, /*name*/ nullptr) : nullptr; + j["chat_template"] = chat_tmpl != nullptr ? std::string(chat_tmpl) : std::string(); + j["special_tokens"] = special_tokens_json(jctx->vocab); + // Full GGUF metadata key/value map. + if (mdl != nullptr) { + json meta_map = json::object(); + const int meta_count = llama_model_meta_count(mdl); + for (int i = 0; i < meta_count; i++) { + char key_buf[256] = {}; + // ponytail: 2 KB/value cap — scalar metadata fits; huge array values + // (tokenizer tokens/merges) truncate rather than bloating the JSON. + char val_buf[2048] = {}; + if (llama_model_meta_key_by_index(mdl, i, key_buf, sizeof(key_buf)) >= 0 && + llama_model_meta_val_str_by_index(mdl, i, val_buf, sizeof(val_buf)) >= 0) { + meta_map[std::string(key_buf)] = std::string(val_buf); + } + } + j["metadata"] = std::move(meta_map); + } + return json_to_jstring(env, j); + }); } JNIEXPORT jint JNICALL Java_net_ladenthin_llama_LlamaModel_requestCompletion(JNIEnv *env, jobject obj, jstring jparams) { - REQUIRE_SERVER_CONTEXT(0); + return jni_guard_impl(env, c_llama_error, [&]() -> jint { + REQUIRE_SERVER_CONTEXT(0); - json data; - if (!parse_json_params(env, jparams, data)) { - return 0; - } + json data; + if (!parse_json_params(env, jparams, data)) { + return 0; + } - const server_task_type type = is_infill_request(data) ? SERVER_TASK_TYPE_INFILL : SERVER_TASK_TYPE_COMPLETION; + const server_task_type type = is_infill_request(data) ? SERVER_TASK_TYPE_INFILL : SERVER_TASK_TYPE_COMPLETION; - return dispatch_streaming_completion(env, jctx, data, type, TASK_RESPONSE_TYPE_NONE); + return dispatch_streaming_completion(env, jctx, data, type, TASK_RESPONSE_TYPE_NONE); + }); } JNIEXPORT void JNICALL Java_net_ladenthin_llama_LlamaModel_releaseTask(JNIEnv *env, jobject obj, jint id_task) { - REQUIRE_SERVER_CONTEXT(); - erase_reader(jctx, id_task); + return jni_guard_impl(env, c_llama_error, [&]() -> void { + REQUIRE_SERVER_CONTEXT(); + erase_reader(jctx, id_task); + }); } JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_receiveCompletionJson(JNIEnv *env, jobject obj, jint id_task) { - REQUIRE_SERVER_CONTEXT(nullptr); - - // Copy the shared_ptr out under the lock so the reader stays alive across next() below, - // which runs without the lock and may race a concurrent erase_reader()/close(). - std::shared_ptr rd; - { - std::lock_guard lk(jctx->readers_mutex); - auto it = jctx->readers.find(id_task); - if (it == jctx->readers.end()) { - env->ThrowNew(c_llama_error, "Task not found"); - return nullptr; - } - rd = it->second; - } + return jni_guard_impl(env, c_llama_error, [&]() -> jstring { + REQUIRE_SERVER_CONTEXT(nullptr); - // Upstream b9437 added is_begin partial results whose to_json() returns - // a nullptr sentinel meaning "HTTP-headers-only, no body". Loop past - // those so the Java iterator only ever sees real events. - json response; - try { - while (true) { - server_task_result_ptr result = rd->next([jctx] { return jctx->closing.load(); }); - - if (!result_ok_or_throw(env, result)) { - erase_reader(jctx, id_task); + // Copy the shared_ptr out under the lock so the reader stays alive across next() below, + // which runs without the lock and may race a concurrent erase_reader()/close(). + std::shared_ptr rd; + { + std::lock_guard lk(jctx->readers_mutex); + auto it = jctx->readers.find(id_task); + if (it == jctx->readers.end()) { + env->ThrowNew(c_llama_error, "Task not found"); return nullptr; } + rd = it->second; + } - response = result->to_json(); - if (response.is_null()) { - continue; - } - response["stop"] = result->is_stop(); + // Upstream b9437 added is_begin partial results whose to_json() returns + // a nullptr sentinel meaning "HTTP-headers-only, no body". Loop past + // those so the Java iterator only ever sees real events. + json response; + try { + while (true) { + server_task_result_ptr result = rd->next([jctx] { return jctx->closing.load(); }); + + if (!result_ok_or_throw(env, result)) { + erase_reader(jctx, id_task); + return nullptr; + } - if (result->is_stop()) { - erase_reader(jctx, id_task); + response = result->to_json(); + if (response.is_null()) { + continue; + } + response["stop"] = result->is_stop(); + + if (result->is_stop()) { + erase_reader(jctx, id_task); + } + break; } - break; + } catch (const std::exception &e) { + // A throwing to_json() must surface as a LlamaException, not abort the JVM. The task is + // over for the Java caller, so release its reader entry like the other error paths. + erase_reader(jctx, id_task); + env->ThrowNew(c_llama_error, e.what()); + return nullptr; } - } catch (const std::exception &e) { - // A throwing to_json() must surface as a LlamaException, not abort the JVM. The task is - // over for the Java caller, so release its reader entry like the other error paths. - erase_reader(jctx, id_task); - env->ThrowNew(c_llama_error, e.what()); - return nullptr; - } - return json_to_jstring(env, response); + return json_to_jstring(env, response); + }); } // Streaming OpenAI chat: poll one step of a chat.completion.chunk stream. @@ -1158,202 +1175,214 @@ JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_receiveCompletionJ // header-only nullptr sentinels (upstream b9437+) and releases the reader on stop. JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_receiveChatCompletionChunk(JNIEnv *env, jobject obj, jint id_task) { - REQUIRE_SERVER_CONTEXT(nullptr); + return jni_guard_impl(env, c_llama_error, [&]() -> jstring { + REQUIRE_SERVER_CONTEXT(nullptr); - // Copy the shared_ptr out under the lock so the reader stays alive across next() below, - // which runs without the lock and may race a concurrent erase_reader()/close(). - std::shared_ptr rd; - { - std::lock_guard lk(jctx->readers_mutex); - auto it = jctx->readers.find(id_task); - if (it == jctx->readers.end()) { - env->ThrowNew(c_llama_error, "Task not found"); - return nullptr; + // Copy the shared_ptr out under the lock so the reader stays alive across next() below, + // which runs without the lock and may race a concurrent erase_reader()/close(). + std::shared_ptr rd; + { + std::lock_guard lk(jctx->readers_mutex); + auto it = jctx->readers.find(id_task); + if (it == jctx->readers.end()) { + env->ThrowNew(c_llama_error, "Task not found"); + return nullptr; + } + rd = it->second; } - rd = it->second; - } - json payload; - bool stop = false; - try { - while (true) { - server_task_result_ptr result = rd->next([jctx] { return jctx->closing.load(); }); + json payload; + bool stop = false; + try { + while (true) { + server_task_result_ptr result = rd->next([jctx] { return jctx->closing.load(); }); - if (!result_ok_or_throw(env, result)) { - erase_reader(jctx, id_task); - return nullptr; - } + if (!result_ok_or_throw(env, result)) { + erase_reader(jctx, id_task); + return nullptr; + } - json chunk = result->to_json(); - if (chunk.is_null()) { - continue; - } - payload = std::move(chunk); - stop = result->is_stop(); - if (stop) { - erase_reader(jctx, id_task); + json chunk = result->to_json(); + if (chunk.is_null()) { + continue; + } + payload = std::move(chunk); + stop = result->is_stop(); + if (stop) { + erase_reader(jctx, id_task); + } + break; } - break; + } catch (const std::exception &e) { + // A throwing to_json() must surface as a LlamaException, not abort the JVM. The task is + // over for the Java caller, so release its reader entry like the other error paths. + erase_reader(jctx, id_task); + env->ThrowNew(c_llama_error, e.what()); + return nullptr; } - } catch (const std::exception &e) { - // A throwing to_json() must surface as a LlamaException, not abort the JVM. The task is - // over for the Java caller, so release its reader entry like the other error paths. - erase_reader(jctx, id_task); - env->ThrowNew(c_llama_error, e.what()); - return nullptr; - } - return json_to_jstring(env, wrap_stream_chunk(std::move(payload), stop)); + return json_to_jstring(env, wrap_stream_chunk(std::move(payload), stop)); + }); } JNIEXPORT jfloatArray JNICALL Java_net_ladenthin_llama_LlamaModel_embed(JNIEnv *env, jobject obj, jstring jprompt) { - REQUIRE_SERVER_CONTEXT(nullptr); + return jni_guard_impl(env, c_llama_error, [&]() -> jfloatArray { + REQUIRE_SERVER_CONTEXT(nullptr); - // Idle sleep frees the model and invalidates the cached vocab; wake and re-sync first. - wake_server(jctx); - - if (!require_embedding_support(env, jctx->params.embedding, c_llama_error)) { - return nullptr; - } + // Idle sleep frees the model and invalidates the cached vocab; wake and re-sync first. + wake_server(jctx); - const std::string prompt = parse_jstring(env, jprompt); - SRV_INF("Calling embedding '%s'\n", prompt.c_str()); + if (!require_embedding_support(env, jctx->params.embedding, c_llama_error)) { + return nullptr; + } - llama_tokens tokens; - try { - tokens = tokenize_mixed(jctx->vocab, prompt, true, true); - } catch (const std::exception &e) { - env->ThrowNew(c_llama_error, e.what()); - return nullptr; - } - auto rd = ctx_server->get_response_reader(); - server_task task(SERVER_TASK_TYPE_EMBEDDING); - task.id = rd.get_new_id(); - task.tokens = server_tokens(tokens, false); - task.index = 0; - wake_and_post(rd, std::move(task)); + const std::string prompt = parse_jstring(env, jprompt); + SRV_INF("Calling embedding '%s'\n", prompt.c_str()); - auto br = rd.wait_for_all([jctx] { return jctx->closing.load(); }); - if (!batch_ok_or_throw(env, br)) - return nullptr; + llama_tokens tokens; + try { + tokens = tokenize_mixed(jctx->vocab, prompt, true, true); + } catch (const std::exception &e) { + env->ThrowNew(c_llama_error, e.what()); + return nullptr; + } + auto rd = ctx_server->get_response_reader(); + server_task task(SERVER_TASK_TYPE_EMBEDDING); + task.id = rd.get_new_id(); + task.tokens = server_tokens(tokens, false); + task.index = 0; + wake_and_post(rd, std::move(task)); + + auto br = rd.wait_for_all([jctx] { return jctx->closing.load(); }); + if (!batch_ok_or_throw(env, br)) + return nullptr; - if (br.results.empty()) { - env->ThrowNew(c_llama_error, "embedding result is empty"); - return nullptr; - } - auto *embd_result = dynamic_cast(br.results[0].get()); - if (!embd_result || embd_result->embedding.empty() || embd_result->embedding[0].empty()) { - env->ThrowNew(c_llama_error, "embedding result is empty"); - return nullptr; - } - const std::vector &first_row = embd_result->embedding[0]; + if (br.results.empty()) { + env->ThrowNew(c_llama_error, "embedding result is empty"); + return nullptr; + } + auto *embd_result = dynamic_cast(br.results[0].get()); + if (!embd_result || embd_result->embedding.empty() || embd_result->embedding[0].empty()) { + env->ThrowNew(c_llama_error, "embedding result is empty"); + return nullptr; + } + const std::vector &first_row = embd_result->embedding[0]; - SRV_INF("Embedding has %d columns\n", static_cast(first_row.size())); - return embedding_to_jfloat_array_impl(env, first_row, c_error_oom); + SRV_INF("Embedding has %d columns\n", static_cast(first_row.size())); + return embedding_to_jfloat_array_impl(env, first_row, c_error_oom); + }); } JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_handleRerank(JNIEnv *env, jobject obj, jstring jprompt, jobjectArray documents) { - REQUIRE_SERVER_CONTEXT(nullptr); + return jni_guard_impl(env, c_llama_error, [&]() -> jstring { + REQUIRE_SERVER_CONTEXT(nullptr); - { - auto meta = wake_and_get_meta(jctx); - if (!jctx->params.embedding || meta.pooling_type != LLAMA_POOLING_TYPE_RANK) { - env->ThrowNew( - c_llama_error, - "This server does not support reranking. Start it with `--reranking` and without `--embedding`"); - return nullptr; + { + auto meta = wake_and_get_meta(jctx); + if (!jctx->params.embedding || meta.pooling_type != LLAMA_POOLING_TYPE_RANK) { + env->ThrowNew( + c_llama_error, + "This server does not support reranking. Start it with `--reranking` and without `--embedding`"); + return nullptr; + } } - } - const std::string prompt = parse_jstring(env, jprompt); - - const jsize amount_documents = env->GetArrayLength(documents); - auto *document_array = parse_string_array(env, documents, amount_documents); - auto document_vector = std::vector(document_array, document_array + amount_documents); - free_string_array(document_array, amount_documents); - - const llama_model *model = llama_get_model(ctx_server->get_llama_context()); - auto rd = ctx_server->get_response_reader(); - std::vector tasks; - tasks.reserve(document_vector.size()); - for (size_t i = 0; i < document_vector.size(); i++) { - tasks.push_back( - build_indexed_token_task(rd, SERVER_TASK_TYPE_RERANK, - format_prompt_rerank(model, jctx->vocab, nullptr, prompt, document_vector[i], - mtmd_helper_init_opt_default()), - static_cast(i), TASK_RESPONSE_TYPE_NONE)); - } - wake_and_post(rd, std::move(tasks)); + const std::string prompt = parse_jstring(env, jprompt); + + const jsize amount_documents = env->GetArrayLength(documents); + auto *document_array = parse_string_array(env, documents, amount_documents); + auto document_vector = std::vector(document_array, document_array + amount_documents); + free_string_array(document_array, amount_documents); + + const llama_model *model = llama_get_model(ctx_server->get_llama_context()); + auto rd = ctx_server->get_response_reader(); + std::vector tasks; + tasks.reserve(document_vector.size()); + for (size_t i = 0; i < document_vector.size(); i++) { + tasks.push_back( + build_indexed_token_task(rd, SERVER_TASK_TYPE_RERANK, + format_prompt_rerank(model, jctx->vocab, nullptr, prompt, document_vector[i], + mtmd_helper_init_opt_default()), + static_cast(i), TASK_RESPONSE_TYPE_NONE)); + } + wake_and_post(rd, std::move(tasks)); - auto br = rd.wait_for_all([jctx] { return jctx->closing.load(); }); - if (!batch_ok_or_throw(env, br)) - return nullptr; - // rerank_results_to_json throws std::invalid_argument on a malformed/out-of-range - // result index; unwrap it into a LlamaException instead of letting it cross - // the JNI boundary (undefined behaviour / JVM abort). - try { - return json_to_jstring(env, rerank_results_to_json(br.results, document_vector)); - } catch (const std::exception &e) { - env->ThrowNew(c_llama_error, e.what()); - return nullptr; - } + auto br = rd.wait_for_all([jctx] { return jctx->closing.load(); }); + if (!batch_ok_or_throw(env, br)) + return nullptr; + // rerank_results_to_json throws std::invalid_argument on a malformed/out-of-range + // result index; unwrap it into a LlamaException instead of letting it cross + // the JNI boundary (undefined behaviour / JVM abort). + try { + return json_to_jstring(env, rerank_results_to_json(br.results, document_vector)); + } catch (const std::exception &e) { + env->ThrowNew(c_llama_error, e.what()); + return nullptr; + } + }); } JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_applyTemplate(JNIEnv *env, jobject obj, jstring jparams) { - REQUIRE_SERVER_CONTEXT(nullptr); + return jni_guard_impl(env, c_llama_error, [&]() -> jstring { + REQUIRE_SERVER_CONTEXT(nullptr); - json data; - if (!parse_json_params(env, jparams, data)) { - return nullptr; - } + json data; + if (!parse_json_params(env, jparams, data)) { + return nullptr; + } - json templateData; - std::vector files; - if (!parse_oai_chat_params(env, jctx, data, templateData, files)) - return nullptr; + json templateData; + std::vector files; + if (!parse_oai_chat_params(env, jctx, data, templateData, files)) + return nullptr; - if (!templateData.contains("prompt") || !templateData.at("prompt").is_string()) { - env->ThrowNew(c_llama_error, "applyTemplate did not produce a string prompt"); - return nullptr; - } - std::string tok_str = templateData.at("prompt"); - return utf8_to_jstring(env, tok_str); + if (!templateData.contains("prompt") || !templateData.at("prompt").is_string()) { + env->ThrowNew(c_llama_error, "applyTemplate did not produce a string prompt"); + return nullptr; + } + std::string tok_str = templateData.at("prompt"); + return utf8_to_jstring(env, tok_str); + }); } JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_handleChatCompletions(JNIEnv *env, jobject obj, jstring jparams) { - REQUIRE_SERVER_CONTEXT(nullptr); + return jni_guard_impl(env, c_llama_error, [&]() -> jstring { + REQUIRE_SERVER_CONTEXT(nullptr); - json body; - if (!parse_json_params(env, jparams, body)) { - return nullptr; - } - json data; - std::vector files; - if (!parse_oai_chat_params(env, jctx, body, data, files)) - return nullptr; + json body; + if (!parse_json_params(env, jparams, body)) { + return nullptr; + } + json data; + std::vector files; + if (!parse_oai_chat_params(env, jctx, body, data, files)) + return nullptr; - return dispatch_blocking_completion(env, jctx, data, SERVER_TASK_TYPE_COMPLETION, TASK_RESPONSE_TYPE_OAI_CHAT, - std::move(files)); + return dispatch_blocking_completion(env, jctx, data, SERVER_TASK_TYPE_COMPLETION, TASK_RESPONSE_TYPE_OAI_CHAT, + std::move(files)); + }); } JNIEXPORT jint JNICALL Java_net_ladenthin_llama_LlamaModel_requestChatCompletion(JNIEnv *env, jobject obj, jstring jparams) { - REQUIRE_SERVER_CONTEXT(0); + return jni_guard_impl(env, c_llama_error, [&]() -> jint { + REQUIRE_SERVER_CONTEXT(0); - json body; - if (!parse_json_params(env, jparams, body)) { - return 0; - } - // Chat template already applied by parse_oai_chat_params; no OAI wrapping on the streaming path. - json data; - std::vector files; - if (!parse_oai_chat_params(env, jctx, body, data, files)) - return 0; + json body; + if (!parse_json_params(env, jparams, body)) { + return 0; + } + // Chat template already applied by parse_oai_chat_params; no OAI wrapping on the streaming path. + json data; + std::vector files; + if (!parse_oai_chat_params(env, jctx, body, data, files)) + return 0; - return dispatch_streaming_completion(env, jctx, data, SERVER_TASK_TYPE_COMPLETION, TASK_RESPONSE_TYPE_NONE, - std::move(files)); + return dispatch_streaming_completion(env, jctx, data, SERVER_TASK_TYPE_COMPLETION, TASK_RESPONSE_TYPE_NONE, + std::move(files)); + }); } // Streaming OpenAI chat with OAI-formatted chunks. Mirrors requestChatCompletion @@ -1363,35 +1392,39 @@ JNIEXPORT jint JNICALL Java_net_ladenthin_llama_LlamaModel_requestChatCompletion // the returned task id with receiveChatCompletionChunk. JNIEXPORT jint JNICALL Java_net_ladenthin_llama_LlamaModel_requestChatCompletionStream(JNIEnv *env, jobject obj, jstring jparams) { - REQUIRE_SERVER_CONTEXT(0); + return jni_guard_impl(env, c_llama_error, [&]() -> jint { + REQUIRE_SERVER_CONTEXT(0); - json body; - if (!parse_json_params(env, jparams, body)) { - return 0; - } - json data; - std::vector files; - if (!parse_oai_chat_params(env, jctx, body, data, files)) - return 0; + json body; + if (!parse_json_params(env, jparams, body)) { + return 0; + } + json data; + std::vector files; + if (!parse_oai_chat_params(env, jctx, body, data, files)) + return 0; - return dispatch_streaming_completion(env, jctx, data, SERVER_TASK_TYPE_COMPLETION, TASK_RESPONSE_TYPE_OAI_CHAT, - std::move(files)); + return dispatch_streaming_completion(env, jctx, data, SERVER_TASK_TYPE_COMPLETION, TASK_RESPONSE_TYPE_OAI_CHAT, + std::move(files)); + }); } JNIEXPORT jintArray JNICALL Java_net_ladenthin_llama_LlamaModel_encode(JNIEnv *env, jobject obj, jstring jprompt) { - REQUIRE_SERVER_CONTEXT(nullptr); - - // Idle sleep frees the model and invalidates the cached vocab; wake and re-sync first. - wake_server(jctx); - - const std::string c_prompt = parse_jstring(env, jprompt); - try { - llama_tokens tokens = tokenize_mixed(jctx->vocab, c_prompt, false, true); - return tokens_to_jint_array_impl(env, tokens, c_error_oom); - } catch (const std::exception &e) { - env->ThrowNew(c_llama_error, e.what()); - return nullptr; - } + return jni_guard_impl(env, c_llama_error, [&]() -> jintArray { + REQUIRE_SERVER_CONTEXT(nullptr); + + // Idle sleep frees the model and invalidates the cached vocab; wake and re-sync first. + wake_server(jctx); + + const std::string c_prompt = parse_jstring(env, jprompt); + try { + llama_tokens tokens = tokenize_mixed(jctx->vocab, c_prompt, false, true); + return tokens_to_jint_array_impl(env, tokens, c_error_oom); + } catch (const std::exception &e) { + env->ThrowNew(c_llama_error, e.what()); + return nullptr; + } + }); } // Detokenise a token sequence to UTF-8, dispatching on vocab-only vs full context. @@ -1404,419 +1437,448 @@ static std::string detokenize(jllama_context *jctx, const std::vector jbyteArray { + REQUIRE_SERVER_CONTEXT(nullptr); - // Idle sleep frees the model and invalidates the cached vocab; wake and re-sync first. - wake_server(jctx); + // Idle sleep frees the model and invalidates the cached vocab; wake and re-sync first. + wake_server(jctx); - const auto tokens = jint_array_to_tokens_impl(env, java_tokens); - return parse_jbytes(env, detokenize(jctx, tokens)); + const auto tokens = jint_array_to_tokens_impl(env, java_tokens); + return parse_jbytes(env, detokenize(jctx, tokens)); + }); } JNIEXPORT void JNICALL Java_net_ladenthin_llama_LlamaModel_delete(JNIEnv *env, jobject obj) { - auto *jctx = get_jllama_context(env, obj); - if (!jctx) - return; - - // Null the Java handle under g_ctx_mutex so no NEW entry point can acquire this context - // (acquire_jllama_context_impl will see 0 and return null). In-flight calls that already - // hold a user reference keep running until they release on their own scope exit. - { - std::lock_guard lk(g_ctx_mutex); - env->SetLongField(obj, f_model_pointer, 0); - } + return jni_guard_impl(env, c_llama_error, [&]() -> void { + auto *jctx = get_jllama_context(env, obj); + if (!jctx) + return; - // Signal teardown to any in-flight reader. The streaming / blocking should_stop lambdas - // observe this and make next()/wait_for_all() return within one poll (~1s), so the - // in-flight JNI call unwinds and releases its user reference — otherwise the reader would - // poll forever (the worker only stops the task queue, not the results queue) and close() - // would hang. - jctx->closing.store(true, std::memory_order_release); - if (!jctx->vocab_only) { - // Cancel any pending streaming readers before stopping the server. - std::lock_guard lk(jctx->readers_mutex); - jctx->readers.clear(); - } + // Null the Java handle under g_ctx_mutex so no NEW entry point can acquire this context + // (acquire_jllama_context_impl will see 0 and return null). In-flight calls that already + // hold a user reference keep running until they release on their own scope exit. + { + std::lock_guard lk(g_ctx_mutex); + env->SetLongField(obj, f_model_pointer, 0); + } - // Wait for every in-flight JNI call (an entry point still inside its jllama_context_guard) - // to release its user reference BEFORE tearing anything down — a concurrent call could - // otherwise still be using the worker, the server_context, or the vocab-only model while - // they are being stopped/freed (use-after-free). New calls cannot start (the handle is - // already nulled above), and the closing flag bounds this wait to ~1s. - { - std::unique_lock lk(jctx->m); - jctx->cv.wait(lk, [&] { return jctx->users.load() == 0; }); - } + // Signal teardown to any in-flight reader. The streaming / blocking should_stop lambdas + // observe this and make next()/wait_for_all() return within one poll (~1s), so the + // in-flight JNI call unwinds and releases its user reference — otherwise the reader would + // poll forever (the worker only stops the task queue, not the results queue) and close() + // would hang. + jctx->closing.store(true, std::memory_order_release); + if (!jctx->vocab_only) { + // Cancel any pending streaming readers before stopping the server. + std::lock_guard lk(jctx->readers_mutex); + jctx->readers.clear(); + } - if (!jctx->vocab_only) { - while (!jctx->worker_ready.load()) { - std::this_thread::yield(); + // Wait for every in-flight JNI call (an entry point still inside its jllama_context_guard) + // to release its user reference BEFORE tearing anything down — a concurrent call could + // otherwise still be using the worker, the server_context, or the vocab-only model while + // they are being stopped/freed (use-after-free). New calls cannot start (the handle is + // already nulled above), and the closing flag bounds this wait to ~1s. + { + std::unique_lock lk(jctx->m); + jctx->cv.wait(lk, [&] { return jctx->users.load() == 0; }); } - // Signal the background thread to stop. Call twice with a brief sleep - // to close the race where the thread signalled ready but start_loop() - // hasn't yet set its internal running flag. - jctx->server.terminate(); - std::this_thread::sleep_for(std::chrono::milliseconds(1)); - jctx->server.terminate(); - if (jctx->worker.joinable()) { - jctx->worker.join(); + + if (!jctx->vocab_only) { + while (!jctx->worker_ready.load()) { + std::this_thread::yield(); + } + // Signal the background thread to stop. Call twice with a brief sleep + // to close the race where the thread signalled ready but start_loop() + // hasn't yet set its internal running flag. + jctx->server.terminate(); + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + jctx->server.terminate(); + if (jctx->worker.joinable()) { + jctx->worker.join(); + } } - } - if (jctx->vocab_only_model) { - llama_model_free(jctx->vocab_only_model); - } + if (jctx->vocab_only_model) { + llama_model_free(jctx->vocab_only_model); + } - delete jctx; + delete jctx; + }); } JNIEXPORT void JNICALL Java_net_ladenthin_llama_LlamaModel_cancelCompletion(JNIEnv *env, jobject obj, jint id_task) { - REQUIRE_SERVER_CONTEXT(); - erase_reader(jctx, id_task); + return jni_guard_impl(env, c_llama_error, [&]() -> void { + REQUIRE_SERVER_CONTEXT(); + erase_reader(jctx, id_task); + }); } JNIEXPORT void JNICALL Java_net_ladenthin_llama_LlamaModel_setLogger(JNIEnv *env, jclass clazz, jobject log_format, jobject jcallback) { - // Serialize the whole swap under the logger mutex: first clear the live callback so no NEW - // trampoline invocation can copy a lambda that still references the global ref we are about - // to delete, then DRAIN trampolines already executing a copied callback (g_log_active), then - // delete the old ref, then install the new one. Without the drain, an in-flight trampoline - // could still call into the just-deleted global ref. Note: this makes setLogger block until - // running log callbacks return — do not call setLogger from within a log callback. - std::unique_lock lk(g_log_mutex); - log_callback = nullptr; - g_log_cv.wait(lk, [] { return g_log_active == 0; }); - if (o_log_callback != nullptr) { - env->DeleteGlobalRef(o_log_callback); - o_log_callback = nullptr; - } + return jni_guard_impl(env, c_llama_error, [&]() -> void { + // Serialize the whole swap under the logger mutex: first clear the live callback so no NEW + // trampoline invocation can copy a lambda that still references the global ref we are about + // to delete, then DRAIN trampolines already executing a copied callback (g_log_active), then + // delete the old ref, then install the new one. Without the drain, an in-flight trampoline + // could still call into the just-deleted global ref. Note: this makes setLogger block until + // running log callbacks return — do not call setLogger from within a log callback. + std::unique_lock lk(g_log_mutex); + log_callback = nullptr; + g_log_cv.wait(lk, [] { return g_log_active == 0; }); + if (o_log_callback != nullptr) { + env->DeleteGlobalRef(o_log_callback); + o_log_callback = nullptr; + } - log_json = env->IsSameObject(log_format, o_log_format_json); + log_json = env->IsSameObject(log_format, o_log_format_json); - if (jcallback == nullptr) { - log_callback = nullptr; - llama_log_set(nullptr, nullptr); - } else { - o_log_callback = env->NewGlobalRef(jcallback); - // Capture copies of the global ref and method id so the callback never dereferences the - // logger globals at call time (those may be swapped by a concurrent setLogger). - jobject cb_ref = o_log_callback; - log_callback = [cb_ref](enum ggml_log_level level, const char *text, void *user_data) noexcept { - // Logging can fire from internal native threads with no JNIEnv; skip rather than - // throw (an exception here would unwind through llama.cpp's C frames). - JNIEnv *env = get_jni_env_or_null(); - if (env == nullptr || text == nullptr) { - return; - } - // Log lines can embed payload text (prompts, model metadata), so the - // message must cross as standard UTF-8, not Modified UTF-8. - jstring message = utf8_to_jstring(env, text); - if (message == nullptr) { - env->ExceptionClear(); // allocation failed; drop this log line - return; - } - jobject log_level = log_level_to_jobject(level); - env->CallVoidMethod(cb_ref, m_biconsumer_accept, log_level, message); - env->DeleteLocalRef(message); - }; - // Always set the trampoline — it handles JSON formatting internally - llama_log_set(log_callback_trampoline, nullptr); - } + if (jcallback == nullptr) { + log_callback = nullptr; + llama_log_set(nullptr, nullptr); + } else { + o_log_callback = env->NewGlobalRef(jcallback); + // Capture copies of the global ref and method id so the callback never dereferences the + // logger globals at call time (those may be swapped by a concurrent setLogger). + jobject cb_ref = o_log_callback; + log_callback = [cb_ref](enum ggml_log_level level, const char *text, void *user_data) noexcept { + // Logging can fire from internal native threads with no JNIEnv; skip rather than + // throw (an exception here would unwind through llama.cpp's C frames). + JNIEnv *env = get_jni_env_or_null(); + if (env == nullptr || text == nullptr) { + return; + } + // Log lines can embed payload text (prompts, model metadata), so the + // message must cross as standard UTF-8, not Modified UTF-8. + jstring message = utf8_to_jstring(env, text); + if (message == nullptr) { + env->ExceptionClear(); // allocation failed; drop this log line + return; + } + jobject log_level = log_level_to_jobject(level); + env->CallVoidMethod(cb_ref, m_biconsumer_accept, log_level, message); + env->DeleteLocalRef(message); + }; + // Always set the trampoline — it handles JSON formatting internally + llama_log_set(log_callback_trampoline, nullptr); + } + }); } JNIEXPORT jbyteArray JNICALL Java_net_ladenthin_llama_LlamaModel_jsonSchemaToGrammarBytes(JNIEnv *env, jclass clazz, jstring j_schema) { - try { - const std::string c_schema = parse_jstring(env, j_schema); - const json c_schema_json = json::parse(c_schema); - const std::string c_grammar = json_schema_to_grammar(c_schema_json); - return parse_jbytes(env, c_grammar); - } catch (const std::exception &e) { - env->ThrowNew(c_llama_error, e.what()); - return nullptr; - } + return jni_guard_impl(env, c_llama_error, [&]() -> jbyteArray { + try { + const std::string c_schema = parse_jstring(env, j_schema); + const json c_schema_json = json::parse(c_schema); + const std::string c_grammar = json_schema_to_grammar(c_schema_json); + return parse_jbytes(env, c_grammar); + } catch (const std::exception &e) { + env->ThrowNew(c_llama_error, e.what()); + return nullptr; + } + }); } // Returns llama.cpp's own build identifier ("b-") from the linked-in build-info // symbols. Pure-ASCII, so NewStringUTF is safe here (the modified-UTF-8 caveat only applies to // model-generated payload text). JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_nativeLlamaCppBuildInfo(JNIEnv *env, jclass clazz) { - return env->NewStringUTF(llama_build_info()); + return jni_guard_impl(env, c_llama_error, [&]() -> jstring { return env->NewStringUTF(llama_build_info()); }); } JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_handleCompletions(JNIEnv *env, jobject obj, jstring jparams) { - REQUIRE_SERVER_CONTEXT(nullptr); + return jni_guard_impl(env, c_llama_error, [&]() -> jstring { + REQUIRE_SERVER_CONTEXT(nullptr); - json data; - if (!parse_json_params(env, jparams, data)) { - return nullptr; - } - return dispatch_blocking_completion(env, jctx, data, SERVER_TASK_TYPE_COMPLETION, TASK_RESPONSE_TYPE_NONE); + json data; + if (!parse_json_params(env, jparams, data)) { + return nullptr; + } + return dispatch_blocking_completion(env, jctx, data, SERVER_TASK_TYPE_COMPLETION, TASK_RESPONSE_TYPE_NONE); + }); } JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_handleCompletionsOai(JNIEnv *env, jobject obj, jstring jparams) { - REQUIRE_SERVER_CONTEXT(nullptr); + return jni_guard_impl(env, c_llama_error, [&]() -> jstring { + REQUIRE_SERVER_CONTEXT(nullptr); - json body; - if (!parse_json_params(env, jparams, body)) { - return nullptr; - } - json data; - try { - data = oaicompat_completion_params_parse(body); - } catch (const std::exception &e) { - throw_invalid_request(env, e); - return nullptr; - } + json body; + if (!parse_json_params(env, jparams, body)) { + return nullptr; + } + json data; + try { + data = oaicompat_completion_params_parse(body); + } catch (const std::exception &e) { + throw_invalid_request(env, e); + return nullptr; + } - return dispatch_blocking_completion(env, jctx, data, SERVER_TASK_TYPE_COMPLETION, TASK_RESPONSE_TYPE_OAI_CMPL); + return dispatch_blocking_completion(env, jctx, data, SERVER_TASK_TYPE_COMPLETION, TASK_RESPONSE_TYPE_OAI_CMPL); + }); } JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_handleInfill(JNIEnv *env, jobject obj, jstring jparams) { - REQUIRE_SERVER_CONTEXT(nullptr); + return jni_guard_impl(env, c_llama_error, [&]() -> jstring { + REQUIRE_SERVER_CONTEXT(nullptr); - // Check FIM token support via server_context_meta (populated from the - // same llama_vocab_fim_* calls inside server-context). - auto meta = wake_and_get_meta(jctx); - if (meta.fim_pre_token == LLAMA_TOKEN_NULL || meta.fim_sub_token == LLAMA_TOKEN_NULL || - meta.fim_mid_token == LLAMA_TOKEN_NULL) { - env->ThrowNew(c_llama_error, "Model does not support fill-in-the-middle infill"); - return nullptr; - } + // Check FIM token support via server_context_meta (populated from the + // same llama_vocab_fim_* calls inside server-context). + auto meta = wake_and_get_meta(jctx); + if (meta.fim_pre_token == LLAMA_TOKEN_NULL || meta.fim_sub_token == LLAMA_TOKEN_NULL || + meta.fim_mid_token == LLAMA_TOKEN_NULL) { + env->ThrowNew(c_llama_error, "Model does not support fill-in-the-middle infill"); + return nullptr; + } - json data; - if (!parse_json_params(env, jparams, data)) { - return nullptr; - } + json data; + if (!parse_json_params(env, jparams, data)) { + return nullptr; + } - if (!require_json_field(env, data, "input_prefix")) - return nullptr; - if (!require_json_field(env, data, "input_suffix")) - return nullptr; + if (!require_json_field(env, data, "input_prefix")) + return nullptr; + if (!require_json_field(env, data, "input_suffix")) + return nullptr; - json input_extra = json_value(data, "input_extra", json::array()); - data["input_extra"] = input_extra; + json input_extra = json_value(data, "input_extra", json::array()); + data["input_extra"] = input_extra; - std::string prompt = json_value(data, "prompt", std::string()); - try { - std::vector tokenized_prompts = - tokenize_input_prompts(jctx->vocab, nullptr, prompt, false, true, mtmd_helper_init_opt_default()); + std::string prompt = json_value(data, "prompt", std::string()); + try { + std::vector tokenized_prompts = + tokenize_input_prompts(jctx->vocab, nullptr, prompt, false, true, mtmd_helper_init_opt_default()); - data["prompt"] = - format_prompt_infill(jctx->vocab, data.at("input_prefix"), data.at("input_suffix"), data.at("input_extra"), - jctx->params.n_batch, jctx->params.n_predict, meta.slot_n_ctx, jctx->params.spm_infill, - tokenized_prompts.empty() ? llama_tokens() : tokenized_prompts[0].get_tokens()); - } catch (const std::exception &e) { - env->ThrowNew(c_llama_error, e.what()); - return nullptr; - } + data["prompt"] = format_prompt_infill( + jctx->vocab, data.at("input_prefix"), data.at("input_suffix"), data.at("input_extra"), + jctx->params.n_batch, jctx->params.n_predict, meta.slot_n_ctx, jctx->params.spm_infill, + tokenized_prompts.empty() ? llama_tokens() : tokenized_prompts[0].get_tokens()); + } catch (const std::exception &e) { + env->ThrowNew(c_llama_error, e.what()); + return nullptr; + } - return dispatch_blocking_completion(env, jctx, data, SERVER_TASK_TYPE_INFILL, TASK_RESPONSE_TYPE_NONE); + return dispatch_blocking_completion(env, jctx, data, SERVER_TASK_TYPE_INFILL, TASK_RESPONSE_TYPE_NONE); + }); } JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_handleEmbeddings(JNIEnv *env, jobject obj, jstring jparams, jboolean joaiCompat) { - REQUIRE_SERVER_CONTEXT(nullptr); - - if (!require_embedding_support(env, jctx->params.embedding, c_llama_error)) { - return nullptr; - } - - task_response_type res_type = joaiCompat ? TASK_RESPONSE_TYPE_OAI_EMBD : TASK_RESPONSE_TYPE_NONE; + return jni_guard_impl(env, c_llama_error, [&]() -> jstring { + REQUIRE_SERVER_CONTEXT(nullptr); - { - auto meta = wake_and_get_meta(jctx); - if (res_type != TASK_RESPONSE_TYPE_NONE && meta.pooling_type == LLAMA_POOLING_TYPE_NONE) { - env->ThrowNew(c_llama_error, - "Pooling type 'none' is not OAI compatible. Please use a different pooling type"); + if (!require_embedding_support(env, jctx->params.embedding, c_llama_error)) { return nullptr; } - } - json body; - if (!parse_json_params(env, jparams, body)) { - return nullptr; - } + task_response_type res_type = joaiCompat ? TASK_RESPONSE_TYPE_OAI_EMBD : TASK_RESPONSE_TYPE_NONE; - bool force_no_oaicompat = false; - json prompt; - bool use_base64 = false; - try { - prompt = extract_embedding_prompt(body, force_no_oaicompat); - use_base64 = parse_encoding_format(body); - } catch (const std::exception &e) { - env->ThrowNew(c_llama_error, e.what()); - return nullptr; - } - if (force_no_oaicompat) - res_type = TASK_RESPONSE_TYPE_NONE; + { + auto meta = wake_and_get_meta(jctx); + if (res_type != TASK_RESPONSE_TYPE_NONE && meta.pooling_type == LLAMA_POOLING_TYPE_NONE) { + env->ThrowNew(c_llama_error, + "Pooling type 'none' is not OAI compatible. Please use a different pooling type"); + return nullptr; + } + } - std::vector tokenized_prompts; - try { - tokenized_prompts = - tokenize_input_prompts(jctx->vocab, nullptr, prompt, true, true, mtmd_helper_init_opt_default()); - } catch (const std::exception &e) { - env->ThrowNew(c_llama_error, e.what()); - return nullptr; - } + json body; + if (!parse_json_params(env, jparams, body)) { + return nullptr; + } - for (const auto &toks : tokenized_prompts) { - if (toks.get_tokens().empty()) { - env->ThrowNew(c_llama_error, "Input content cannot be empty"); + bool force_no_oaicompat = false; + json prompt; + bool use_base64 = false; + try { + prompt = extract_embedding_prompt(body, force_no_oaicompat); + use_base64 = parse_encoding_format(body); + } catch (const std::exception &e) { + env->ThrowNew(c_llama_error, e.what()); + return nullptr; + } + if (force_no_oaicompat) + res_type = TASK_RESPONSE_TYPE_NONE; + + std::vector tokenized_prompts; + try { + tokenized_prompts = + tokenize_input_prompts(jctx->vocab, nullptr, prompt, true, true, mtmd_helper_init_opt_default()); + } catch (const std::exception &e) { + env->ThrowNew(c_llama_error, e.what()); return nullptr; } - } - auto rd = ctx_server->get_response_reader(); - std::vector tasks; - tasks.reserve(tokenized_prompts.size()); - for (size_t i = 0; i < tokenized_prompts.size(); i++) { - tasks.push_back(build_indexed_token_task(rd, SERVER_TASK_TYPE_EMBEDDING, - server_tokens(tokenized_prompts[i].get_tokens(), false), - static_cast(i), res_type)); - } - wake_and_post(rd, std::move(tasks)); + for (const auto &toks : tokenized_prompts) { + if (toks.get_tokens().empty()) { + env->ThrowNew(c_llama_error, "Input content cannot be empty"); + return nullptr; + } + } - auto br = rd.wait_for_all([jctx] { return jctx->closing.load(); }); - if (!batch_ok_or_throw(env, br)) - return nullptr; + auto rd = ctx_server->get_response_reader(); + std::vector tasks; + tasks.reserve(tokenized_prompts.size()); + for (size_t i = 0; i < tokenized_prompts.size(); i++) { + tasks.push_back(build_indexed_token_task(rd, SERVER_TASK_TYPE_EMBEDDING, + server_tokens(tokenized_prompts[i].get_tokens(), false), + static_cast(i), res_type)); + } + wake_and_post(rd, std::move(tasks)); - json responses = json::array(); - for (const auto &result : br.results) { - responses.push_back(result->to_json()); - } - json out = (res_type == TASK_RESPONSE_TYPE_OAI_EMBD) - ? format_embeddings_response_oaicompat( - body, json_value(body, "model", std::string(DEFAULT_OAICOMPAT_MODEL)), responses, use_base64) - : responses; - try { - return json_to_jstring(env, out); - } catch (const std::exception &e) { - env->ThrowNew(c_llama_error, e.what()); - return nullptr; - } + auto br = rd.wait_for_all([jctx] { return jctx->closing.load(); }); + if (!batch_ok_or_throw(env, br)) + return nullptr; + + json responses = json::array(); + for (const auto &result : br.results) { + responses.push_back(result->to_json()); + } + json out = + (res_type == TASK_RESPONSE_TYPE_OAI_EMBD) + ? format_embeddings_response_oaicompat( + body, json_value(body, "model", std::string(DEFAULT_OAICOMPAT_MODEL)), responses, use_base64) + : responses; + try { + return json_to_jstring(env, out); + } catch (const std::exception &e) { + env->ThrowNew(c_llama_error, e.what()); + return nullptr; + } + }); } JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_handleTokenize(JNIEnv *env, jobject obj, jstring jcontent, jboolean jaddSpecial, jboolean jwithPieces) { - REQUIRE_SERVER_CONTEXT(nullptr); + return jni_guard_impl(env, c_llama_error, [&]() -> jstring { + REQUIRE_SERVER_CONTEXT(nullptr); - // Idle sleep frees the model and invalidates the cached vocab; wake and re-sync first. - wake_server(jctx); + // Idle sleep frees the model and invalidates the cached vocab; wake and re-sync first. + wake_server(jctx); - const std::string content = parse_jstring(env, jcontent); - const bool add_special = jaddSpecial; - const bool with_pieces = jwithPieces; + const std::string content = parse_jstring(env, jcontent); + const bool add_special = jaddSpecial; + const bool with_pieces = jwithPieces; - llama_tokens tokens; - try { - tokens = tokenize_mixed(jctx->vocab, content, add_special, true); - } catch (const std::exception &e) { - env->ThrowNew(c_llama_error, e.what()); - return nullptr; - } + llama_tokens tokens; + try { + tokens = tokenize_mixed(jctx->vocab, content, add_special, true); + } catch (const std::exception &e) { + env->ThrowNew(c_llama_error, e.what()); + return nullptr; + } - json tokens_response = json::array(); - - if (with_pieces) { - llama_context *lctx = jctx->vocab_only ? nullptr : jctx->server.get_llama_context(); - for (const auto &token : tokens) { - std::string piece; - if (lctx) { - piece = common_token_to_piece(lctx, token); - } else { - char buf[256]; - int n = llama_token_to_piece(jctx->vocab, token, buf, static_cast(sizeof(buf)), 0, false); - piece = n > 0 ? std::string(buf, n) : std::string(); + json tokens_response = json::array(); + + if (with_pieces) { + llama_context *lctx = jctx->vocab_only ? nullptr : jctx->server.get_llama_context(); + for (const auto &token : tokens) { + std::string piece; + if (lctx) { + piece = common_token_to_piece(lctx, token); + } else { + char buf[256]; + int n = llama_token_to_piece(jctx->vocab, token, buf, static_cast(sizeof(buf)), 0, false); + piece = n > 0 ? std::string(buf, n) : std::string(); + } + tokens_response.push_back({{"id", token}, {"piece", token_piece_value(piece)}}); } - tokens_response.push_back({{"id", token}, {"piece", token_piece_value(piece)}}); + } else { + tokens_response = tokens; } - } else { - tokens_response = tokens; - } - return json_to_jstring(env, format_tokenizer_response(tokens_response)); + return json_to_jstring(env, format_tokenizer_response(tokens_response)); + }); } JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_handleDetokenize(JNIEnv *env, jobject obj, jintArray jtokens) { - REQUIRE_SERVER_CONTEXT(nullptr); + return jni_guard_impl(env, c_llama_error, [&]() -> jstring { + REQUIRE_SERVER_CONTEXT(nullptr); - // Idle sleep frees the model and invalidates the cached vocab; wake and re-sync first. - wake_server(jctx); + // Idle sleep frees the model and invalidates the cached vocab; wake and re-sync first. + wake_server(jctx); - const auto tokens = jint_array_to_tokens_impl(env, jtokens); - return json_to_jstring(env, format_detokenized_response(detokenize(jctx, tokens))); + const auto tokens = jint_array_to_tokens_impl(env, jtokens); + return json_to_jstring(env, format_detokenized_response(detokenize(jctx, tokens))); + }); } JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_handleSlotAction(JNIEnv *env, jobject obj, jint action, jint slotId, jstring jfilename) { - REQUIRE_SERVER_CONTEXT(nullptr); - - switch (action) { - case 0: { // LIST — the full server-introspection payload - // b10408 (upstream #26920) reduced server_task_result_metrics::to_json() to the slot array - // and b10519 (#27376) split the task in two: METRICS keeps only the counters (its to_json() - // is unused and returns JSON null; to_metrics() renders them as Prometheus text) and - // SERVER_TASK_TYPE_SLOT_GET carries the slot array plus the idle-slot count. Post both and - // merge them so getMetrics() keeps returning the single documented object. - auto metrics_result = post_and_wait(env, jctx, server_task(SERVER_TASK_TYPE_METRICS)); - if (!metrics_result) - return nullptr; - auto slots_result = post_and_wait(env, jctx, server_task(SERVER_TASK_TYPE_SLOT_GET)); - if (!slots_result) - return nullptr; + return jni_guard_impl(env, c_llama_error, [&]() -> jstring { + REQUIRE_SERVER_CONTEXT(nullptr); + + switch (action) { + case 0: { // LIST — the full server-introspection payload + // b10408 (upstream #26920) reduced server_task_result_metrics::to_json() to the slot array + // and b10519 (#27376) split the task in two: METRICS keeps only the counters (its to_json() + // is unused and returns JSON null; to_metrics() renders them as Prometheus text) and + // SERVER_TASK_TYPE_SLOT_GET carries the slot array plus the idle-slot count. Post both and + // merge them so getMetrics() keeps returning the single documented object. + auto metrics_result = post_and_wait(env, jctx, server_task(SERVER_TASK_TYPE_METRICS)); + if (!metrics_result) + return nullptr; + auto slots_result = post_and_wait(env, jctx, server_task(SERVER_TASK_TYPE_SLOT_GET)); + if (!slots_result) + return nullptr; - const auto *metrics = dynamic_cast(metrics_result.get()); - const auto *slots = dynamic_cast(slots_result.get()); - if (metrics == nullptr || slots == nullptr) { - env->ThrowNew(c_llama_error, "Unexpected result type for server metrics"); + const auto *metrics = dynamic_cast(metrics_result.get()); + const auto *slots = dynamic_cast(slots_result.get()); + if (metrics == nullptr || slots == nullptr) { + env->ThrowNew(c_llama_error, "Unexpected result type for server metrics"); + return nullptr; + } + return json_to_jstring(env, server_metrics_to_json(*metrics, *slots)); + } + case 1: // SAVE + return exec_slot_file_task(env, jctx, slotId, jfilename, SERVER_TASK_TYPE_SLOT_SAVE, + "Filename is required for slot save"); + case 2: // RESTORE + return exec_slot_file_task(env, jctx, slotId, jfilename, SERVER_TASK_TYPE_SLOT_RESTORE, + "Filename is required for slot restore"); + case 3: { // ERASE + server_task task(SERVER_TASK_TYPE_SLOT_ERASE); + task.slot_action.id_slot = slotId; + return dispatch_one_shot_task(env, jctx, std::move(task)); + } + default: + env->ThrowNew(c_llama_error, "Invalid slot action"); return nullptr; } - return json_to_jstring(env, server_metrics_to_json(*metrics, *slots)); - } - case 1: // SAVE - return exec_slot_file_task(env, jctx, slotId, jfilename, SERVER_TASK_TYPE_SLOT_SAVE, - "Filename is required for slot save"); - case 2: // RESTORE - return exec_slot_file_task(env, jctx, slotId, jfilename, SERVER_TASK_TYPE_SLOT_RESTORE, - "Filename is required for slot restore"); - case 3: { // ERASE - server_task task(SERVER_TASK_TYPE_SLOT_ERASE); - task.slot_action.id_slot = slotId; - return dispatch_one_shot_task(env, jctx, std::move(task)); - } - default: - env->ThrowNew(c_llama_error, "Invalid slot action"); - return nullptr; - } + }); } JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_getLoraAdaptersJson(JNIEnv *env, jobject obj) { - REQUIRE_SERVER_CONTEXT(nullptr); + return jni_guard_impl(env, c_llama_error, [&]() -> jstring { + REQUIRE_SERVER_CONTEXT(nullptr); - return dispatch_one_shot_task(env, jctx, server_task(SERVER_TASK_TYPE_GET_LORA)); + return dispatch_one_shot_task(env, jctx, server_task(SERVER_TASK_TYPE_GET_LORA)); + }); } JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_setLoraAdaptersJson(JNIEnv *env, jobject obj, jstring jadapters) { - REQUIRE_SERVER_CONTEXT(nullptr); + return jni_guard_impl(env, c_llama_error, [&]() -> jstring { + REQUIRE_SERVER_CONTEXT(nullptr); - json data; - if (!parse_json_params(env, jadapters, data)) { - return nullptr; - } - if (!data.is_array()) { - // Same contract as the upstream POST /lora-adapters route body. - env->ThrowNew(c_llama_error, "LoRA adapter list must be a JSON array of {id, scale} objects"); - return nullptr; - } - server_task task(SERVER_TASK_TYPE_SET_LORA); - task.set_lora = parse_lora_request(data); - return dispatch_one_shot_task(env, jctx, std::move(task)); + json data; + if (!parse_json_params(env, jadapters, data)) { + return nullptr; + } + if (!data.is_array()) { + // Same contract as the upstream POST /lora-adapters route body. + env->ThrowNew(c_llama_error, "LoRA adapter list must be a JSON array of {id, scale} objects"); + return nullptr; + } + server_task task(SERVER_TASK_TYPE_SET_LORA); + task.set_lora = parse_lora_request(data); + return dispatch_one_shot_task(env, jctx, std::move(task)); + }); } // LlamaQuantizer is not part of the javac-generated jllama.h (that header only covers LlamaModel), @@ -1826,82 +1888,86 @@ JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaModel_setLoraAdaptersJso // wraps the NativeServer entry points in an extern "C" block. extern "C" JNIEXPORT void JNICALL Java_net_ladenthin_llama_LlamaQuantizer_quantizeNative( JNIEnv *env, jclass, jstring jinput, jstring joutput, jint ftype, jint nthread, jboolean allowRequantize) { - try { - const std::string input_path = parse_jstring(env, jinput); - const std::string output_path = parse_jstring(env, joutput); - // Idempotent; intentionally never paired with llama_backend_free here — a LlamaModel - // loaded in the same JVM shares the backend and must not have it freed underneath it. - llama_backend_init(); - llama_model_quantize_params qparams = llama_model_quantize_default_params(); - qparams.ftype = static_cast(ftype); - qparams.nthread = nthread; - qparams.allow_requantize = (allowRequantize == JNI_TRUE); - const uint32_t rc = llama_model_quantize(input_path.c_str(), output_path.c_str(), &qparams); - if (rc != 0) { - const std::string msg = "Quantization of '" + input_path + "' failed with code " + std::to_string(rc); - env->ThrowNew(c_llama_error, msg.c_str()); + return jni_guard_impl(env, c_llama_error, [&]() -> void { + try { + const std::string input_path = parse_jstring(env, jinput); + const std::string output_path = parse_jstring(env, joutput); + // Idempotent; intentionally never paired with llama_backend_free here — a LlamaModel + // loaded in the same JVM shares the backend and must not have it freed underneath it. + llama_backend_init(); + llama_model_quantize_params qparams = llama_model_quantize_default_params(); + qparams.ftype = static_cast(ftype); + qparams.nthread = nthread; + qparams.allow_requantize = (allowRequantize == JNI_TRUE); + const uint32_t rc = llama_model_quantize(input_path.c_str(), output_path.c_str(), &qparams); + if (rc != 0) { + const std::string msg = "Quantization of '" + input_path + "' failed with code " + std::to_string(rc); + env->ThrowNew(c_llama_error, msg.c_str()); + } + } catch (const std::exception &e) { + env->ThrowNew(c_llama_error, e.what()); + } catch (...) { + env->ThrowNew(c_llama_error, "Unknown C++ exception during quantization"); } - } catch (const std::exception &e) { - env->ThrowNew(c_llama_error, e.what()); - } catch (...) { - env->ThrowNew(c_llama_error, "Unknown C++ exception during quantization"); - } + }); } JNIEXPORT jboolean JNICALL Java_net_ladenthin_llama_LlamaModel_configureParallelInference(JNIEnv *env, jobject obj, jstring jconfig) { - REQUIRE_SERVER_CONTEXT(JNI_FALSE); - (void)obj; - - json config; - if (!parse_json_params(env, jconfig, config)) { - return JNI_FALSE; - } + return jni_guard_impl(env, c_llama_error, [&]() -> jboolean { + REQUIRE_SERVER_CONTEXT(JNI_FALSE); + (void)obj; - std::optional slot_sim_opt; - std::optional n_threads_opt; - std::optional n_threads_batch_opt; - try { - slot_sim_opt = parse_slot_prompt_similarity(config); - n_threads_opt = parse_positive_int_config(config, "n_threads"); - n_threads_batch_opt = parse_positive_int_config(config, "n_threads_batch"); - } catch (const std::exception &e) { - env->ThrowNew(c_llama_error, e.what()); - return JNI_FALSE; - } + json config; + if (!parse_json_params(env, jconfig, config)) { + return JNI_FALSE; + } - // Apply n_threads / n_threads_batch via the public llama.h API. The setter - // requires both values; fill any missing one from the cached common_params - // captured at load_model time so a single-field update behaves as a no-op - // for the unspecified field. - if (n_threads_opt.has_value() || n_threads_batch_opt.has_value()) { - llama_context *lctx = ctx_server->get_llama_context(); - if (lctx == nullptr) { - env->ThrowNew(c_llama_error, "configureParallelInference: llama_context not available " - "(model sleeping or not loaded)"); + std::optional slot_sim_opt; + std::optional n_threads_opt; + std::optional n_threads_batch_opt; + try { + slot_sim_opt = parse_slot_prompt_similarity(config); + n_threads_opt = parse_positive_int_config(config, "n_threads"); + n_threads_batch_opt = parse_positive_int_config(config, "n_threads_batch"); + } catch (const std::exception &e) { + env->ThrowNew(c_llama_error, e.what()); return JNI_FALSE; } - const int n = n_threads_opt.value_or(jctx->params.cpuparams.n_threads); - const int nb = n_threads_batch_opt.value_or(jctx->params.cpuparams_batch.n_threads); - llama_set_n_threads(lctx, n, nb); - // Keep the cached params in sync so a follow-up call that supplies only - // the other field reads back the value just applied, not the original. - jctx->params.cpuparams.n_threads = n; - jctx->params.cpuparams_batch.n_threads = nb; - } - // slot_prompt_similarity: validated above (the [0.0, 1.0] range check still - // throws for out-of-range values, preserving the existing exception - // contract). Live mutation uses server_context::set_slot_prompt_similarity(), - // added upstream by https://github.com/ggml-org/llama.cpp/pull/22393 and carried - // in this repo as patches/0003-pr22393-... until it merges upstream (the patch - // applies against the pinned llama.cpp GIT_TAG in CMakeLists.txt). not thread-safe - // per the upstream contract — main-thread only, which this JNI call is. - if (slot_sim_opt.has_value()) { - ctx_server->set_slot_prompt_similarity(*slot_sim_opt); - } + // Apply n_threads / n_threads_batch via the public llama.h API. The setter + // requires both values; fill any missing one from the cached common_params + // captured at load_model time so a single-field update behaves as a no-op + // for the unspecified field. + if (n_threads_opt.has_value() || n_threads_batch_opt.has_value()) { + llama_context *lctx = ctx_server->get_llama_context(); + if (lctx == nullptr) { + env->ThrowNew(c_llama_error, "configureParallelInference: llama_context not available " + "(model sleeping or not loaded)"); + return JNI_FALSE; + } + const int n = n_threads_opt.value_or(jctx->params.cpuparams.n_threads); + const int nb = n_threads_batch_opt.value_or(jctx->params.cpuparams_batch.n_threads); + llama_set_n_threads(lctx, n, nb); + // Keep the cached params in sync so a follow-up call that supplies only + // the other field reads back the value just applied, not the original. + jctx->params.cpuparams.n_threads = n; + jctx->params.cpuparams_batch.n_threads = nb; + } + + // slot_prompt_similarity: validated above (the [0.0, 1.0] range check still + // throws for out-of-range values, preserving the existing exception + // contract). Live mutation uses server_context::set_slot_prompt_similarity(), + // added upstream by https://github.com/ggml-org/llama.cpp/pull/22393 and carried + // in this repo as patches/0003-pr22393-... until it merges upstream (the patch + // applies against the pinned llama.cpp GIT_TAG in CMakeLists.txt). not thread-safe + // per the upstream contract — main-thread only, which this JNI call is. + if (slot_sim_opt.has_value()) { + ctx_server->set_slot_prompt_similarity(*slot_sim_opt); + } - return JNI_TRUE; + return JNI_TRUE; + }); } // --------------------------------------------------------------------------- @@ -1914,22 +1980,24 @@ extern "C" { JNIEXPORT jlong JNICALL Java_net_ladenthin_llama_TextToSpeech_loadNative(JNIEnv *env, jclass clazz, jstring jmodel, jstring jmmproj, jint gpu_layers, jint threads) { - (void)clazz; - try { - const std::string model = parse_jstring(env, jmodel); - const std::string mmproj = parse_jstring(env, jmmproj); - std::string err; - jllama_tts::tts_engine *engine = - jllama_tts::engine_init(model, mmproj, static_cast(gpu_layers), static_cast(threads), err); - if (engine == nullptr) { - env->ThrowNew(c_llama_error, err.c_str()); + return jni_guard_impl(env, c_llama_error, [&]() -> jlong { + (void)clazz; + try { + const std::string model = parse_jstring(env, jmodel); + const std::string mmproj = parse_jstring(env, jmmproj); + std::string err; + jllama_tts::tts_engine *engine = + jllama_tts::engine_init(model, mmproj, static_cast(gpu_layers), static_cast(threads), err); + if (engine == nullptr) { + env->ThrowNew(c_llama_error, err.c_str()); + return 0; + } + return reinterpret_cast(engine); + } catch (const std::exception &e) { + env->ThrowNew(c_llama_error, e.what()); return 0; } - return reinterpret_cast(engine); - } catch (const std::exception &e) { - env->ThrowNew(c_llama_error, e.what()); - return 0; - } + }); } JNIEXPORT jbyteArray JNICALL Java_net_ladenthin_llama_TextToSpeech_synthesizeNative(JNIEnv *env, jclass clazz, @@ -1937,41 +2005,46 @@ JNIEXPORT jbyteArray JNICALL Java_net_ladenthin_llama_TextToSpeech_synthesizeNat jstring jspeaker_reference_path, jstring jlang, jint max_frames, jint top_k, jint seed) { - (void)clazz; - try { - auto *engine = reinterpret_cast(handle); - if (engine == nullptr) { - env->ThrowNew(c_llama_error, "TextToSpeech handle is null"); - return nullptr; - } - const std::string text = parse_jstring(env, jtext); - const std::string speaker_reference_path = - jspeaker_reference_path == nullptr ? std::string() : parse_jstring(env, jspeaker_reference_path); - const std::string lang = jlang == nullptr ? std::string() : parse_jstring(env, jlang); - std::vector wav; - std::string err; - if (!jllama_tts::engine_synthesize(engine, text, speaker_reference_path, lang, static_cast(max_frames), - static_cast(top_k), static_cast(seed), wav, err)) { - env->ThrowNew(c_llama_error, err.c_str()); - return nullptr; - } - jbyteArray out = env->NewByteArray(static_cast(wav.size())); - if (out == nullptr) { - env->ThrowNew(c_error_oom, "could not allocate WAV byte array"); + return jni_guard_impl(env, c_llama_error, [&]() -> jbyteArray { + (void)clazz; + try { + auto *engine = reinterpret_cast(handle); + if (engine == nullptr) { + env->ThrowNew(c_llama_error, "TextToSpeech handle is null"); + return nullptr; + } + const std::string text = parse_jstring(env, jtext); + const std::string speaker_reference_path = + jspeaker_reference_path == nullptr ? std::string() : parse_jstring(env, jspeaker_reference_path); + const std::string lang = jlang == nullptr ? std::string() : parse_jstring(env, jlang); + std::vector wav; + std::string err; + if (!jllama_tts::engine_synthesize(engine, text, speaker_reference_path, lang, static_cast(max_frames), + static_cast(top_k), static_cast(seed), wav, err)) { + env->ThrowNew(c_llama_error, err.c_str()); + return nullptr; + } + jbyteArray out = env->NewByteArray(static_cast(wav.size())); + if (out == nullptr) { + env->ThrowNew(c_error_oom, "could not allocate WAV byte array"); + return nullptr; + } + env->SetByteArrayRegion(out, 0, static_cast(wav.size()), + reinterpret_cast(wav.data())); + return out; + } catch (const std::exception &e) { + env->ThrowNew(c_llama_error, e.what()); return nullptr; } - env->SetByteArrayRegion(out, 0, static_cast(wav.size()), reinterpret_cast(wav.data())); - return out; - } catch (const std::exception &e) { - env->ThrowNew(c_llama_error, e.what()); - return nullptr; - } + }); } JNIEXPORT void JNICALL Java_net_ladenthin_llama_TextToSpeech_deleteNative(JNIEnv *env, jclass clazz, jlong handle) { - (void)env; - (void)clazz; - jllama_tts::engine_free(reinterpret_cast(handle)); + return jni_guard_impl(env, c_llama_error, [&]() -> void { + (void)env; + (void)clazz; + jllama_tts::engine_free(reinterpret_cast(handle)); + }); } } // extern "C" diff --git a/llama/src/main/cpp/jni_helpers.hpp b/llama/src/main/cpp/jni_helpers.hpp index 0e70f7cd..3272fada 100644 --- a/llama/src/main/cpp/jni_helpers.hpp +++ b/llama/src/main/cpp/jni_helpers.hpp @@ -34,6 +34,7 @@ #include #include #include +#include #include // Forward declarations. @@ -217,6 +218,52 @@ template return tokens; } +// --------------------------------------------------------------------------- +// report_cxx_exception_impl / jni_guard_impl — the JNI exception boundary +// +// An exception that escapes a native method and unwinds across the JNI boundary +// is undefined behaviour and aborts the JVM on most implementations. Every +// `Java_*` entry point in this project therefore runs its body inside +// jni_guard_impl, which converts anything that escapes into a Java exception and +// returns a zero/nullptr sentinel. +// +// The guard is ADDITIVE: an entry point that already converts std::exception +// itself keeps doing so and never reaches the handlers here. What the guard adds +// everywhere is the `catch (...)` arm — the case for an exception type not +// derived from std::exception, which otherwise has no backstop at all. +// +// Two rules the handlers must keep: +// * Never call ThrowNew while a Java exception is already pending. The JNI +// spec forbids most calls in that state, and the pending exception is the +// more precise error anyway — so it is left in place. +// * Never call ThrowNew with a null class. JNI_OnLoad has not cached the +// exception class yet and JNI_OnUnload has already released it, so those two +// carry their own local handlers rather than routing through here. +// --------------------------------------------------------------------------- +inline void report_cxx_exception_impl(JNIEnv *env, jclass exception_class, const char *what) { + if (env->ExceptionCheck()) { + return; + } + if (exception_class == nullptr) { + return; + } + env->ThrowNew(exception_class, what); +} + +template auto jni_guard_impl(JNIEnv *env, jclass exception_class, Fn &&fn) -> decltype(fn()) { + using result_type = decltype(fn()); + try { + return fn(); + } catch (const std::exception &e) { + report_cxx_exception_impl(env, exception_class, e.what()); + } catch (...) { + report_cxx_exception_impl(env, exception_class, "unknown C++ exception crossed the JNI boundary"); + } + if constexpr (!std::is_void::value) { + return result_type{}; + } +} + // =========================================================================== // Layer B — JNI + server orchestration // (upstream server headers must be included by the TU before this header) diff --git a/llama/src/main/cpp/native_server.cpp b/llama/src/main/cpp/native_server.cpp index 28566057..5ab687cd 100644 --- a/llama/src/main/cpp/native_server.cpp +++ b/llama/src/main/cpp/native_server.cpp @@ -75,10 +75,15 @@ void fill_native_server_args(JNIEnv *env, jobjectArray jargs, native_server *srv } } +// Resolves net.ladenthin.llama.exception.LlamaException. Unlike jllama.cpp this TU caches no +// global class reference, so the lookup happens per call; every entry point here is a cold +// lifecycle operation (start/stop/isRunning), never an inference hot path. +jclass llama_exception_class(JNIEnv *env) { return env->FindClass("net/ladenthin/llama/exception/LlamaException"); } + // Throws net.ladenthin.llama.exception.LlamaException with the given message (best-effort: if the // class cannot be resolved the pending NoClassDefFoundError is surfaced instead). void throw_llama_exception(JNIEnv *env, const char *message) { - jclass exception_class = env->FindClass("net/ladenthin/llama/exception/LlamaException"); + jclass exception_class = llama_exception_class(env); if (exception_class != nullptr) { env->ThrowNew(exception_class, message); } @@ -90,115 +95,126 @@ extern "C" { JNIEXPORT jlong JNICALL Java_net_ladenthin_llama_server_NativeServer_startNativeServer(JNIEnv *env, jclass, jobjectArray jargs) { - auto *srv = new native_server(); - fill_native_server_args(env, jargs, srv); + return jni_guard_impl(env, llama_exception_class(env), [&]() -> jlong { + auto *srv = new native_server(); + fill_native_server_args(env, jargs, srv); - // Embedded mode: no process signal handlers, honor the forwarded argv (see patches/0006). - llama_server_set_embedded(true); + // Embedded mode: no process signal handlers, honor the forwarded argv (see patches/0006). + llama_server_set_embedded(true); - srv->worker = std::thread([srv]() { - srv->exit_code = llama_server(static_cast(srv->argv.size()), srv->argv.data()); - srv->finished.store(true); - }); + srv->worker = std::thread([srv]() { + srv->exit_code = llama_server(static_cast(srv->argv.size()), srv->argv.data()); + srv->finished.store(true); + }); - return reinterpret_cast(srv); + return reinterpret_cast(srv); + }); } -JNIEXPORT void JNICALL Java_net_ladenthin_llama_server_NativeServer_stopNativeServer(JNIEnv *, jclass, jlong handle) { - auto *srv = reinterpret_cast(handle); - if (srv == nullptr) { - return; - } - // Signal shutdown, retrying until the worker actually returns: a stop issued before the server - // finished starting (shutdown_handler not yet installed by llama_server) would otherwise be - // lost. Once the handler is installed the first signal takes effect; if the model failed to - // load, llama_server has already returned and `finished` is set. - while (!srv->finished.load()) { - llama_server_request_shutdown(); - if (srv->finished.load()) { - break; +JNIEXPORT void JNICALL Java_net_ladenthin_llama_server_NativeServer_stopNativeServer(JNIEnv *env, jclass, + jlong handle) { + return jni_guard_impl(env, llama_exception_class(env), [&]() -> void { + auto *srv = reinterpret_cast(handle); + if (srv == nullptr) { + return; } - std::this_thread::sleep_for(std::chrono::milliseconds(50)); - } - if (srv->worker.joinable()) { - srv->worker.join(); - } - delete srv; + // Signal shutdown, retrying until the worker actually returns: a stop issued before the server + // finished starting (shutdown_handler not yet installed by llama_server) would otherwise be + // lost. Once the handler is installed the first signal takes effect; if the model failed to + // load, llama_server has already returned and `finished` is set. + while (!srv->finished.load()) { + llama_server_request_shutdown(); + if (srv->finished.load()) { + break; + } + std::this_thread::sleep_for(std::chrono::milliseconds(50)); + } + if (srv->worker.joinable()) { + srv->worker.join(); + } + delete srv; + }); } -JNIEXPORT jboolean JNICALL Java_net_ladenthin_llama_server_NativeServer_isRunningNative(JNIEnv *, jclass, +JNIEXPORT jboolean JNICALL Java_net_ladenthin_llama_server_NativeServer_isRunningNative(JNIEnv *env, jclass, jlong handle) { - auto *srv = reinterpret_cast(handle); - return (srv != nullptr && !srv->finished.load()) ? JNI_TRUE : JNI_FALSE; + return jni_guard_impl(env, llama_exception_class(env), [&]() -> jboolean { + auto *srv = reinterpret_cast(handle); + return (srv != nullptr && !srv->finished.load()) ? JNI_TRUE : JNI_FALSE; + }); } JNIEXPORT jlong JNICALL Java_net_ladenthin_llama_server_NativeServer_startAttachedNativeServer(JNIEnv *env, jclass, jobject jmodel, jobjectArray jargs) { - if (jmodel == nullptr) { - throw_llama_exception(env, "model must not be null"); - return 0; - } - // Resolve the LlamaModel's native context handle (the same "ctx" long field jllama.cpp - // caches in JNI_OnLoad; this TU resolves it itself to stay decoupled from those globals). - jclass model_class = env->GetObjectClass(jmodel); - jfieldID ctx_field = env->GetFieldID(model_class, "ctx", "J"); - if (ctx_field == nullptr) { - return 0; // NoSuchFieldError already pending - } - // Acquire a jllama_context user reference through the same handle protocol as every other - // JNI entry point (field read + user increment under g_ctx_mutex), so a concurrent - // LlamaModel.close() cannot free jctx between the handle read and the increment. The - // reference is held for the lifetime of the attach worker: delete() in jllama.cpp waits on - // the user-count condition variable, so close() blocks until this thread exits (close the - // server before the model — see the NativeServer attach-constructor Javadoc). - jllama_context *jctx = acquire_jllama_context_impl(env, jmodel, ctx_field); - if (jctx == nullptr) { - throw_llama_exception(env, "model is not loaded (or already closed)"); - return 0; - } + return jni_guard_impl(env, llama_exception_class(env), [&]() -> jlong { + if (jmodel == nullptr) { + throw_llama_exception(env, "model must not be null"); + return 0; + } + // Resolve the LlamaModel's native context handle (the same "ctx" long field jllama.cpp + // caches in JNI_OnLoad; this TU resolves it itself to stay decoupled from those globals). + jclass model_class = env->GetObjectClass(jmodel); + jfieldID ctx_field = env->GetFieldID(model_class, "ctx", "J"); + if (ctx_field == nullptr) { + return 0; // NoSuchFieldError already pending + } + // Acquire a jllama_context user reference through the same handle protocol as every other + // JNI entry point (field read + user increment under g_ctx_mutex), so a concurrent + // LlamaModel.close() cannot free jctx between the handle read and the increment. The + // reference is held for the lifetime of the attach worker: delete() in jllama.cpp waits on + // the user-count condition variable, so close() blocks until this thread exits (close the + // server before the model — see the NativeServer attach-constructor Javadoc). + jllama_context *jctx = acquire_jllama_context_impl(env, jmodel, ctx_field); + if (jctx == nullptr) { + throw_llama_exception(env, "model is not loaded (or already closed)"); + return 0; + } - // Contract of the attach path: llama_server_attach drives the HTTP frontend on a NEW worker - // thread but must NOT start a second task-processing loop on the shared server_context — the - // model's own worker (spawned in load_model_impl) already runs start_loop(). Likewise, - // stopNativeServer's llama_server_request_shutdown() must terminate only the HTTP frontend, - // not the attached model's worker, so the LlamaModel remains usable after the server closes - // (documented order: close server first, then model). If a future upstream change breaks - // either assumption, this attach path needs revisiting. - auto *srv = new native_server(); - fill_native_server_args(env, jargs, srv); - - // The attach entry always parses the forwarded argv; set the embedded flag anyway so any - // shared embedded-mode behavior in server.cpp stays consistent with startNativeServer. - llama_server_set_embedded(true); - - server_context *ctx_server = &jctx->server; - srv->worker = std::thread([srv, ctx_server, jctx]() { - srv->exit_code = llama_server_attach(static_cast(srv->argv.size()), srv->argv.data(), *ctx_server); - srv->finished.store(true); - release_jllama_context_impl(jctx); + // Contract of the attach path: llama_server_attach drives the HTTP frontend on a NEW worker + // thread but must NOT start a second task-processing loop on the shared server_context — the + // model's own worker (spawned in load_model_impl) already runs start_loop(). Likewise, + // stopNativeServer's llama_server_request_shutdown() must terminate only the HTTP frontend, + // not the attached model's worker, so the LlamaModel remains usable after the server closes + // (documented order: close server first, then model). If a future upstream change breaks + // either assumption, this attach path needs revisiting. + auto *srv = new native_server(); + fill_native_server_args(env, jargs, srv); + + // The attach entry always parses the forwarded argv; set the embedded flag anyway so any + // shared embedded-mode behavior in server.cpp stays consistent with startNativeServer. + llama_server_set_embedded(true); + + server_context *ctx_server = &jctx->server; + srv->worker = std::thread([srv, ctx_server, jctx]() { + srv->exit_code = llama_server_attach(static_cast(srv->argv.size()), srv->argv.data(), *ctx_server); + srv->finished.store(true); + release_jllama_context_impl(jctx); + }); + + return reinterpret_cast(srv); }); - - return reinterpret_cast(srv); } JNIEXPORT void JNICALL Java_net_ladenthin_llama_server_NativeServer_setWorkerCommandNative(JNIEnv *env, jclass, jstring jcommand) { - // Sets/clears LLAMA_SERVER_WORKER_CMD in the process environment, which the router-mode - // model manager (server-models.cpp, patches/0008) reads when spawning worker instances. - std::string value; - if (jcommand != nullptr) { - value = parse_jstring(env, jcommand); - } + return jni_guard_impl(env, llama_exception_class(env), [&]() -> void { + // Sets/clears LLAMA_SERVER_WORKER_CMD in the process environment, which the router-mode + // model manager (server-models.cpp, patches/0008) reads when spawning worker instances. + std::string value; + if (jcommand != nullptr) { + value = parse_jstring(env, jcommand); + } #if defined(_WIN32) - _putenv_s("LLAMA_SERVER_WORKER_CMD", value.c_str()); // empty value removes the variable + _putenv_s("LLAMA_SERVER_WORKER_CMD", value.c_str()); // empty value removes the variable #else - if (value.empty()) { - unsetenv("LLAMA_SERVER_WORKER_CMD"); - } else { - setenv("LLAMA_SERVER_WORKER_CMD", value.c_str(), 1); - } + if (value.empty()) { + unsetenv("LLAMA_SERVER_WORKER_CMD"); + } else { + setenv("LLAMA_SERVER_WORKER_CMD", value.c_str(), 1); + } #endif + }); } } // extern "C" diff --git a/llama/src/main/cpp/train_engine.cpp b/llama/src/main/cpp/train_engine.cpp index 030c1ade..3caeaed2 100644 --- a/llama/src/main/cpp/train_engine.cpp +++ b/llama/src/main/cpp/train_engine.cpp @@ -93,7 +93,7 @@ bool finetune(const finetune_config &cfg, std::string &err) { } // namespace jllama_train extern "C" JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaTrainer_finetuneNative(JNIEnv *env, jclass, - jstring jconfig) { + jstring jconfig) try { std::string config_json; if (jconfig != nullptr) { const char *c = env->GetStringUTFChars(jconfig, nullptr); @@ -136,4 +136,12 @@ extern "C" JNIEXPORT jstring JNICALL Java_net_ladenthin_llama_LlamaTrainer_finet err = "unknown C++ exception during fine-tuning"; } return env->NewStringUTF(err.c_str()); +} catch (...) { + // Function-level backstop, not jni_guard_impl: this TU deliberately keeps its own nlohmann + // alias and never includes jni_helpers.hpp (see CLAUDE.md). The two inner handlers above give + // better messages and are reached first; this one covers what they cannot — a non-std + // exception from the config parse, and anything thrown before either try block (the + // GetStringUTFChars copy can std::bad_alloc). This method reports failure as its return + // string rather than a Java exception, so the backstop keeps that contract. + return env->NewStringUTF("unknown C++ exception crossed the JNI boundary"); } diff --git a/llama/src/test/cpp/test_jni_helpers.cpp b/llama/src/test/cpp/test_jni_helpers.cpp index b81feb52..c1b74178 100644 --- a/llama/src/test/cpp/test_jni_helpers.cpp +++ b/llama/src/test/cpp/test_jni_helpers.cpp @@ -770,3 +770,128 @@ TEST(ConfigureTaskSlot, ExplicitIdPinsTask) { configure_task_slot_impl(task, {{"id_slot", 3}}); EXPECT_EQ(task.id_slot, 3); } + +// --------------------------------------------------------------------------- +// jni_guard_impl — the JNI exception boundary. +// +// Every Java_* entry point in this project runs its body inside this guard, so +// what it does with an escaping exception is a shipped contract, not a detail. +// The mock records whether ThrowNew was reached and with what message. +// --------------------------------------------------------------------------- +namespace { + +struct guard_probe { + bool threw = false; + std::string message; + bool exception_pending = false; +}; + +guard_probe g_guard_probe; + +JNIEnv *make_guard_env(JNIEnv_ &fake_env, JNINativeInterface_ &iface) { + g_guard_probe = guard_probe{}; + iface = {}; + iface.ExceptionCheck = [](JNIEnv *) -> jboolean { return g_guard_probe.exception_pending ? JNI_TRUE : JNI_FALSE; }; + iface.ThrowNew = [](JNIEnv *, jclass, const char *msg) -> jint { + g_guard_probe.threw = true; + g_guard_probe.message = msg != nullptr ? msg : ""; + return 0; + }; + fake_env = {}; + fake_env.functions = &iface; + return &fake_env; +} + +// Any non-null value works: the mock ThrowNew never dereferences the class. +jclass fake_exception_class() { return reinterpret_cast(&g_guard_probe); } + +} // namespace + +TEST(JniGuard, PassesThroughTheReturnValueWhenNothingThrows) { + JNIEnv_ fake_env; + JNINativeInterface_ iface; + JNIEnv *env = make_guard_env(fake_env, iface); + + const jint result = jni_guard_impl(env, fake_exception_class(), [&]() -> jint { return 42; }); + + EXPECT_EQ(result, 42); + EXPECT_FALSE(g_guard_probe.threw); +} + +TEST(JniGuard, ConvertsStdExceptionIntoAJavaThrowAndReturnsZero) { + JNIEnv_ fake_env; + JNINativeInterface_ iface; + JNIEnv *env = make_guard_env(fake_env, iface); + + const jint result = + jni_guard_impl(env, fake_exception_class(), [&]() -> jint { throw std::runtime_error("boom"); }); + + EXPECT_EQ(result, 0); + EXPECT_TRUE(g_guard_probe.threw); + EXPECT_EQ(g_guard_probe.message, "boom"); +} + +// The arm that matters: an exception NOT derived from std::exception has no other backstop +// anywhere, and unwinding it across the JNI boundary is undefined behaviour. +TEST(JniGuard, ConvertsANonStdExceptionInsteadOfLettingItEscape) { + JNIEnv_ fake_env; + JNINativeInterface_ iface; + JNIEnv *env = make_guard_env(fake_env, iface); + + const jint result = jni_guard_impl(env, fake_exception_class(), [&]() -> jint { throw 17; }); + + EXPECT_EQ(result, 0); + EXPECT_TRUE(g_guard_probe.threw); + EXPECT_EQ(g_guard_probe.message, "unknown C++ exception crossed the JNI boundary"); +} + +TEST(JniGuard, PointerReturningEntryPointsYieldNullptrOnThrow) { + JNIEnv_ fake_env; + JNINativeInterface_ iface; + JNIEnv *env = make_guard_env(fake_env, iface); + + const jstring result = + jni_guard_impl(env, fake_exception_class(), [&]() -> jstring { throw std::runtime_error("boom"); }); + + EXPECT_EQ(result, nullptr); + EXPECT_TRUE(g_guard_probe.threw); +} + +TEST(JniGuard, VoidEntryPointsStillReportTheException) { + JNIEnv_ fake_env; + JNINativeInterface_ iface; + JNIEnv *env = make_guard_env(fake_env, iface); + + jni_guard_impl(env, fake_exception_class(), [&]() -> void { throw std::runtime_error("boom"); }); + + EXPECT_TRUE(g_guard_probe.threw); + EXPECT_EQ(g_guard_probe.message, "boom"); +} + +// The JNI spec forbids most calls while a Java exception is pending, and the pending one is the +// more precise error — so the guard must leave it alone rather than overwrite it. +TEST(JniGuard, DoesNotThrowOverAnAlreadyPendingJavaException) { + JNIEnv_ fake_env; + JNINativeInterface_ iface; + JNIEnv *env = make_guard_env(fake_env, iface); + g_guard_probe.exception_pending = true; + + const jint result = + jni_guard_impl(env, fake_exception_class(), [&]() -> jint { throw std::runtime_error("boom"); }); + + EXPECT_EQ(result, 0); + EXPECT_FALSE(g_guard_probe.threw); +} + +// JNI_OnLoad has not cached the exception class yet and JNI_OnUnload has already released it; +// a null class must never reach ThrowNew. +TEST(JniGuard, DoesNotThrowWithoutAnExceptionClass) { + JNIEnv_ fake_env; + JNINativeInterface_ iface; + JNIEnv *env = make_guard_env(fake_env, iface); + + const jint result = jni_guard_impl(env, nullptr, [&]() -> jint { throw std::runtime_error("boom"); }); + + EXPECT_EQ(result, 0); + EXPECT_FALSE(g_guard_probe.threw); +} From 29e1e73310bbe232832003b746ed57f27bb93c94 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 15 Sep 2026 16:36:49 +0000 Subject: [PATCH 4/6] ci: smoke the two aarch64 fat jars before they are attached to a release package-fatjars.sh emits four OS/arch server fat jars, and all four are uploaded, GPG-signed and attached to every release by github-release-signed / github-snapshot. Only the two x86-64 ones were ever launched: grepping publish.yml for all-linux-aarch64 or all-windows-aarch64 returned zero hits, so neither was downloaded or run anywhere in the pipeline. That is precisely what workspace/policies/fat-jar-release-assets.md forbids -- "No release asset is attached that CI has not run" -- and that rule exists because a corrupt macOS dylib shipped in three releases under a fully green pipeline. These two jars were the remaining assets in the same blind spot. Adds smoke-fatjar-linux-aarch64 (ubuntu-24.04-arm) and smoke-fatjar-windows-arm64 (windows-11-arm), mirroring the existing x86-64 jobs -- the only differences are the runner and the jar glob. Both GitHub ARM runners are already used by this workflow (the aarch64 CPU/Vulkan builds and build-windows-arm64), and that arm64 job already resolves temurin at this java-version on windows-11-arm, so no new infrastructure is involved. package-fatjars gains the two matching single-jar upload artifacts so the smokes do not pull the multi-GB set, and both jobs join the publish-snapshot and publish-release needs graphs as fail-loud gates. The Linux job also re-runs verify-bytecode-version.sh over its jar: package-fatjars rewrites the zip per OS/arch, so the aarch64 asset is a different artifact from the one smoke-fatjar-linux checks even though the classes are identical. Verified: the workflow parses (64 jobs, no dangling needs), both new jobs gate both publish paths, the classifier set in llama/pom.xml really yields the linux-aarch64 and windows-aarch64 targets (vulkan-linux-aarch64 and opencl-windows-aarch64 respectively), so the globs match real jar names, and smoke-test-fatjar.sh/.ps1 are arch-agnostic (java -jar, /health, /v1/chat/completions). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Cft7guQngfyKycdJfBEfJP --- .github/workflows/publish.yml | 112 +++++++++++++++++++++++++++++++++- CLAUDE.md | 18 ++++-- TODO.md | 22 +------ 3 files changed, 126 insertions(+), 26 deletions(-) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 9219e6d4..d9e905b1 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -3232,6 +3232,26 @@ jobs: compression-level: 0 retention-days: 7 if-no-files-found: error + # The two aarch64 fat jars are release assets exactly like the x86-64 pair, so they get the + # same treatment: package-fatjars emits four, and for a long time only the two x86-64 ones + # were ever launched. GitHub's free ARM runners (ubuntu-24.04-arm / windows-11-arm) are + # already used by the aarch64 build jobs, so there is no reason to leave these unrun. + - name: Upload Linux aarch64 smoke jar + uses: actions/upload-artifact@v7 + with: + name: llama-fatjar-smoke-linux-aarch64 + path: fatjars/llama-*-all-linux-aarch64-jar-with-dependencies.jar + compression-level: 0 + retention-days: 7 + if-no-files-found: error + - name: Upload Windows arm64 smoke jar + uses: actions/upload-artifact@v7 + with: + name: llama-fatjar-smoke-windows-arm64 + path: fatjars/llama-*-all-windows-aarch64-jar-with-dependencies.jar + compression-level: 0 + retention-days: 7 + if-no-files-found: error # GPU-less runners: every manifest backend must fail its load cleanly (missing vendor # runtimes) and the server must come up on the default CPU natives — this exercises @@ -3319,6 +3339,94 @@ jobs: server-err.log if-no-files-found: warn + # The aarch64 halves of the same convention. Identical in shape to the two jobs above — the only + # differences are the runner and the jar glob — and they exist because `all-linux-aarch64` and + # `all-windows-aarch64` were built, GPG-signed and attached to every release without CI ever + # launching them, which is exactly what workspace/policies/fat-jar-release-assets.md forbids + # ("No release asset is attached that CI has not run"). That rule exists because a corrupt macOS + # dylib shipped in three releases under a fully green pipeline; these two jars were the remaining + # assets in the same blind spot. + smoke-fatjar-linux-aarch64: + name: Smoke test all-backends fat jar (Linux aarch64) + needs: [package-fatjars, verify-model-cache] + runs-on: ubuntu-24.04-arm + steps: + - uses: actions/checkout@v7 + - uses: actions/download-artifact@v8 + with: + name: llama-fatjar-smoke-linux-aarch64 + path: fatjars/ + - name: Restore shared GGUF model cache (populated by download-models; no re-download) + # Restore-only: consumer jobs can NEVER write the cache, so a job running on a + # cache miss cannot re-save an empty/partial entry under the immutable key. + # download-models is the single writer; enableCrossOsArchive matches its + # cross-OS entry version so every consumer restores the same ubuntu-built entry. + uses: actions/cache/restore@v6 + with: + path: models/ + key: gguf-models-${{ hashFiles('.github/models.csv') }} + enableCrossOsArchive: true + - name: Validate model files + run: .github/validate-models.sh + - uses: actions/setup-java@v6 + with: + distribution: 'temurin' + java-version: ${{ env.JAVA_VERSION }} + # Same floor, re-checked on this assembled release asset: package-fatjars rewrites the zip + # per OS/arch, so the aarch64 jar is a different artifact from the one smoke-fatjar-linux + # verifies even though the classes are identical. + - name: Verify Java 8 bytecode (no class newer than major 52) + run: .github/verify-bytecode-version.sh --max-major 52 fatjars + - name: Run fat-jar server smoke test + run: .github/smoke-test-fatjar.sh fatjars 'llama-*-all-linux-aarch64-jar-with-dependencies.jar' "models/${DRAFT_MODEL_NAME}" + - name: Upload server logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: fatjar-smoke-linux-aarch64-logs + path: | + server-out.log + server-err.log + if-no-files-found: warn + + smoke-fatjar-windows-arm64: + name: Smoke test all-backends fat jar (Windows arm64) + needs: [package-fatjars, verify-model-cache] + runs-on: windows-11-arm + steps: + - uses: actions/checkout@v7 + - uses: actions/download-artifact@v8 + with: + name: llama-fatjar-smoke-windows-arm64 + path: fatjars/ + - name: Restore shared GGUF model cache (populated by download-models; no re-download) + # Restore-only: see the note on the Linux job above. + uses: actions/cache/restore@v6 + with: + path: models/ + key: gguf-models-${{ hashFiles('.github/models.csv') }} + enableCrossOsArchive: true + - name: Validate model files + run: .github\validate-models.bat + # temurin publishes a Windows/AArch64 JDK for this java-version; the build-windows-arm64 + # job already resolves it on this same runner. + - uses: actions/setup-java@v6 + with: + distribution: 'temurin' + java-version: ${{ env.JAVA_VERSION }} + - name: Run fat-jar server smoke test + shell: pwsh + run: .github/smoke-test-fatjar.ps1 -JarDir fatjars -JarGlob 'llama-*-all-windows-aarch64-jar-with-dependencies.jar' -Model "models/$env:DRAFT_MODEL_NAME" + - name: Upload server logs + if: failure() + uses: actions/upload-artifact@v7 + with: + name: fatjar-smoke-windows-arm64-logs + path: | + server-out.log + server-err.log + if-no-files-found: warn + # --------------------------------------------------------------------------- # macOS member of the cross-repo "no release asset is attached that CI has not run" convention # (workspace/policies/fat-jar-release-assets.md; BitcoinAddressFinder and srcmorph run the shared @@ -3407,7 +3515,7 @@ jobs: publish-snapshot: name: Publish Snapshot to Central - needs: [check-snapshot, crosscompile-linux-x86_64-cuda, crosscompile-android-aarch64-opencl, package-android-aar, test-android-emulator, code-style, test-java-llama-langchain4j, test-java-llama-kotlin, package-fatjars, smoke-fatjar-linux, smoke-fatjar-windows, smoke-fatjar-macos] + needs: [check-snapshot, crosscompile-linux-x86_64-cuda, crosscompile-android-aarch64-opencl, package-android-aar, test-android-emulator, code-style, test-java-llama-langchain4j, test-java-llama-kotlin, package-fatjars, smoke-fatjar-linux, smoke-fatjar-windows, smoke-fatjar-linux-aarch64, smoke-fatjar-windows-arm64, smoke-fatjar-macos] if: needs.check-snapshot.result == 'success' && inputs.publish_to_central runs-on: ubuntu-latest environment: maven-central @@ -3674,7 +3782,7 @@ jobs: publish-release: name: Publish Release to Central if: needs.check-tag.result == 'success' && inputs.publish_to_central - needs: [check-tag, crosscompile-linux-x86_64-cuda, crosscompile-android-aarch64-opencl, package-android-aar, test-android-emulator, code-style, test-java-llama-langchain4j, test-java-llama-kotlin, package-fatjars, smoke-fatjar-linux, smoke-fatjar-windows, smoke-fatjar-macos] + needs: [check-tag, crosscompile-linux-x86_64-cuda, crosscompile-android-aarch64-opencl, package-android-aar, test-android-emulator, code-style, test-java-llama-langchain4j, test-java-llama-kotlin, package-fatjars, smoke-fatjar-linux, smoke-fatjar-windows, smoke-fatjar-linux-aarch64, smoke-fatjar-windows-arm64, smoke-fatjar-macos] runs-on: ubuntu-latest environment: maven-central permissions: diff --git a/CLAUDE.md b/CLAUDE.md index c2338787..c272b180 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -438,11 +438,19 @@ Mechanism (three pieces): property `net.ladenthin.llama.backend` forces one backend (fail-loud) or `default`/`cpu`. Jars without a manifest take the unchanged legacy path. A backend whose extra module is already resident from a previously failed attempt is skipped (by-name import cross-wiring). -3. **`publish.yml` wiring** — `smoke-fatjar-linux` / `smoke-fatjar-windows` run the - `all--x86-64` jar via real `java -jar` on GPU-less runners (cached draft model, - `--chat-template chatml`): poll `/health` to 200, assert a `/v1/chat/completions` choice, - and require the loader's backend-selection log line. `publish-snapshot`/`publish-release` - `need` `package-fatjars` + both smokes (fail-loud gating); `github-release-signed` and +3. **`publish.yml` wiring** — **all four** OS/arch fat jars are launched, one smoke job each: + `smoke-fatjar-linux` / `smoke-fatjar-windows` (x86-64) plus `smoke-fatjar-linux-aarch64` + (`ubuntu-24.04-arm`) / `smoke-fatjar-windows-arm64` (`windows-11-arm`). Each runs its jar via + real `java -jar` on a GPU-less runner (cached draft model, `--chat-template chatml`): poll + `/health` to 200, assert a `/v1/chat/completions` choice, and require the loader's + backend-selection log line — so every manifest backend failing its load and falling back to the + CPU natives is exercised on the actual release asset. The four jobs consume four small + single-jar artifacts (`llama-fatjar-smoke-{linux,windows,linux-aarch64,windows-arm64}`) rather + than the multi-GB `llama-fatjars` set. **The two aarch64 jobs close a real gap**: those jars were + built, GPG-signed and attached to every release while `publish.yml` referenced them zero times, + which is exactly what the cross-repo rule forbids — and that rule exists because a corrupt macOS + dylib shipped in three releases under a fully green pipeline. `publish-snapshot`/`publish-release` + `need` `package-fatjars` + **all four** smokes (fail-loud gating); `github-release-signed` and `github-snapshot` additionally download `llama-fatjars` into their asset directory, then **GPG-sign each fat jar** via `.github/sign-fatjars.sh` (a detached `.asc` alongside the `.sha256`), so the fat jars land signed on the tag release and the rolling `snapshot` diff --git a/TODO.md b/TODO.md index 5ecd8318..c8334d9e 100644 --- a/TODO.md +++ b/TODO.md @@ -318,25 +318,9 @@ these are what remains. ### Release/build robustness gaps found by the b10679 audit (PR #403) -Both are **pre-existing** and orthogonal to a version bump, so they were recorded rather than folded -into that PR. - -- **Two `all-*-aarch64` fat jars are attached to releases with no smoke job.** - `.github/package-fatjars.sh` emits four OS/arch fat jars (`linux-x86-64`, `linux-aarch64`, - `windows-x86-64`, `windows-aarch64`), all uploaded as `llama-fatjars` and attached by - `github-release-signed` / `github-snapshot`. Only the two **x86-64** ones are smoked - (`smoke-fatjar-linux`, `smoke-fatjar-windows`); grepping `publish.yml` for `all-linux-aarch64` or - `all-windows-aarch64` returns nothing, so neither is ever downloaded or launched. - - That directly violates the cross-repo rule in - [`../workspace/policies/fat-jar-release-assets.md`](../workspace/policies/fat-jar-release-assets.md) - — *"No release asset is attached that CI has not run"* — which exists because a corrupt macOS dylib - shipped in three releases under a fully green pipeline. The fix is cheap: the workflow **already** - uses the free ARM runners elsewhere (`ubuntu-24.04-arm` for the aarch64 CPU and Vulkan builds, - `windows-11-arm` for the Windows arm64 build), so `smoke-fatjar-linux-aarch64` and - `smoke-fatjar-windows-arm64` can mirror the existing smoke jobs and join both publish jobs' - `needs:`. Not done in the bump PR because it widens a version bump into CI work and would gate that - PR on a pre-existing defect if either jar turns out to be broken. +Pre-existing and orthogonal to a version bump, so it was recorded rather than folded +into that PR. (The companion item — the two un-smoked `all-*-aarch64` fat jars — is now fixed: +`smoke-fatjar-linux-aarch64` and `smoke-fatjar-windows-arm64` gate both publish jobs.) - **The patch applier silently accepts a partially-reverted source tree.** The stamp file records the checked-out llama.cpp commit plus each patch's SHA-256 — **nothing about the resulting file From 5d1dc634d52fc5c368380c2de2fd3541ec667812 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 15 Sep 2026 18:34:10 +0000 Subject: [PATCH 5/6] =?UTF-8?q?chore:=20close=20four=20audit=20gaps=20?= =?UTF-8?q?=E2=80=94=20patch=20guard,=20context-guard=20tests,=20arch=20ma?= =?UTF-8?q?p,=20dead=20code?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four independent items from the repo audit, plus the TODO drift they left behind. 1. patches/0010 now has an always-on, model-free guard (.github/verify-patches-applied.sh, run in the C++ Tests job). Most patches already red CI everywhere if they go missing -- 0003/0006/0007/0008 are link errors, 0011 and 0012 have their own C++ test files -- but 0010 casts one enum inside a `static` function unreachable from jllama_test, so reverting it leaves ctest fully green and only a model-gated Java test notices. The script also asserts, self-maintainingly, that every llama/patches/*.patch is named in the applier's stamp and that the fetched tree is actually dirty, so a stale build dir or a reverted patched file fails loud. Falsified both ways before wiring: reverting only the 0010 cast, and reverting the tree with the stamp intact, each exit 1; the intact tree exits 0. 2. The close()-vs-inference use-after-free defence is now tested. acquire_/ release_jllama_context_impl and jllama_context_guard had zero references across all seven test files while their sibling get_jllama_context_impl had three; they were absent only because they are `inline` and never odr-used in jllama_test (g_ctx_mutex is extern here and defined in jllama.cpp, which this binary does not compile). A test-local definition unblocks them. 7 tests cover the reference count up and down, the guard's destructor on a normal and an early return, and the two null paths. Falsified by deleting the fetch_add from acquire: 2 of the new tests go red, and green again once restored. 3. OSInfo.archMapping's alias branch is asserted. getArchName()'s map lookup had no assertion anywhere, so a lost `amd64 -> x86_64` entry would silently send LlamaLoader to a resource directory that was never shipped. Only the 18 NON-IDENTITY aliases are pinned -- an identity entry such as s390x -> s390x is behaviourally redundant with the \W-stripping fallback, so pinning it could not detect its loss -- plus the case-insensitivity of the lookup and the fallback itself. os.arch is saved and restored alongside the existing override property. 4. Java8CompatibilityHelper is deleted. Six of its seven public methods had zero production call sites; the only live one, toString(ByteArrayOutputStream, Charset), is inlined into its single caller in ProcessRunner. Its FORMAT_STRING_MANIPULATION suppression in spotbugs-exclude.xml is removed in the same commit -- a suppression naming a method that no longer exists is silently inert, which is the exact failure class CLAUDE.md warns about. 5. TODO.md's "Upstream PR submissions" section was stale in both directions: it said "six of the seven patches" when there are nine, still listed 0009 (merged upstream and dropped at b10280), omitted 0010/0011/0012, and described 0003 as dropping automatically when its upstream PR merges -- upstream closed that PR without merging, so 0003 is permanent. Verified: clang-format 22.1.8 clean, Release build 0 errors 0 warnings, ctest 551/551 (544 + 7), mvn test 1758 Java tests 0 failures (OSInfoTest 19/19), spotbugs:check 0 bugs, spotless clean. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Cft7guQngfyKycdJfBEfJP --- .github/verify-patches-applied.sh | 78 +++++++++ .github/workflows/publish.yml | 6 + TODO.md | 28 +++- llama/spotbugs-exclude.xml | 14 -- .../loader/Java8CompatibilityHelper.java | 149 ------------------ .../ladenthin/llama/loader/ProcessRunner.java | 5 +- llama/src/test/cpp/test_jni_helpers.cpp | 95 +++++++++++ .../ladenthin/llama/loader/OSInfoTest.java | 66 ++++++++ 8 files changed, 268 insertions(+), 173 deletions(-) create mode 100755 .github/verify-patches-applied.sh delete mode 100644 llama/src/main/java/net/ladenthin/llama/loader/Java8CompatibilityHelper.java diff --git a/.github/verify-patches-applied.sh b/.github/verify-patches-applied.sh new file mode 100755 index 00000000..ab02c9ff --- /dev/null +++ b/.github/verify-patches-applied.sh @@ -0,0 +1,78 @@ +#!/usr/bin/env bash + +# SPDX-FileCopyrightText: 2026 Bernard Ladenthin +# +# SPDX-License-Identifier: MIT OR Apache-2.0 + +# Asserts that every llama/patches/*.patch really reached the fetched llama.cpp tree. +# +# WHY THIS EXISTS. The patch applier (llama/cmake/apply-llama-patches.cmake) is fail-loud on +# "does not apply", so a *stale* patch cannot ship silently. What it cannot detect is a patch +# that stops having an effect while still applying, and most patches do not need this check +# because they have a runnable guard that reds CI on every platform if they go missing: +# +# 0003, 0006, 0007, 0008 -> jllama.cpp / native_server.cpp call the symbols they add, +# so dropping one is a compile or link error. +# 0011 -> the ContentOnlyParseUtf8 tests in src/test/cpp/test_utils.cpp. +# 0012 -> src/test/cpp/test_model_split.cpp. +# 0001, 0002 -> model-gated Java jobs (Windows argv, LoadProgressCallbackTest). +# +# `0010` is the exception and the reason for this script. It casts one enum to int inside +# upstream's `get_res_model_info()`, which is `static` in server-context.cpp and therefore +# unreachable from jllama_test; reverting it leaves `ctest` completely green. Its only guard is +# NativeServerAttachIntegrationTest.models_reportNumericVocabType, which is model-gated — so the +# day a platform stops downloading models, the regression ships. This check runs in the +# always-on `C++ Tests` job, needs no model, and costs milliseconds. +# +# Usage: .github/verify-patches-applied.sh [] +# Exit codes: 0 all good, 1 a check failed. + +set -euo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +SRC="${1:-$ROOT/llama/build/_deps/llama.cpp-src}" +PATCH_DIR="$ROOT/llama/patches" +STAMP="$SRC/.jllama-patches-applied" + +fail() { + echo "ERROR: $*" >&2 + exit 1 +} + +[ -d "$SRC" ] || fail "llama.cpp source dir not found: $SRC (configure the build first)" +[ -f "$STAMP" ] || fail "patch stamp not found: $STAMP — the applier never ran, so the tree is unpatched" + +# --- 1. every patch on disk is named in the stamp ----------------------------------------------- +# Self-maintaining on purpose: adding a patch file needs no edit here. The stamp's first line is +# the checked-out llama.cpp commit; every other line is " ". +on_disk=0 +for p in "$PATCH_DIR"/*.patch; do + [ -e "$p" ] || fail "no *.patch files in $PATCH_DIR" + on_disk=$((on_disk + 1)) + name="$(basename "$p")" + grep -qF "$name" "$STAMP" || fail "patch '$name' is on disk but absent from the stamp $STAMP" +done + +in_stamp="$(($(wc -l < "$STAMP") - 1))" +[ "$in_stamp" -eq "$on_disk" ] \ + || fail "stamp lists $in_stamp patch(es) but $on_disk are on disk — the build dir is stale; configure into a fresh one" + +# --- 2. the tree is actually modified ------------------------------------------------------------ +# A valid stamp over a clean tree means the patches were reverted after the fact. +if git -C "$SRC" rev-parse --git-dir >/dev/null 2>&1; then + if git -C "$SRC" diff --quiet; then + fail "stamp says $on_disk patch(es) applied but '$SRC' is clean — the patched files were reverted" + fi +fi + +# --- 3. the one patch with no runnable guard ------------------------------------------------------ +VOCAB_CAST='(int) meta.model_vocab_type' +SERVER_CONTEXT="$SRC/tools/server/server-context.cpp" +[ -f "$SERVER_CONTEXT" ] || fail "not found: $SERVER_CONTEXT" +grep -qF "$VOCAB_CAST" "$SERVER_CONTEXT" \ + || fail "patches/0010 is not present in $SERVER_CONTEXT: expected '$VOCAB_CAST'. + Without the cast, common_json binds the unscoped enum to its bool constructor and + GET /models + GET /v1/models report vocab_type as true/false instead of a number. + If upstream added the cast themselves, DROP patch 0010 and update this check." + +echo "patches verified: $on_disk applied, tree dirty, patches/0010 cast present" diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index d9e905b1..60546c44 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -2246,6 +2246,12 @@ jobs: run: | mvn -q --no-transfer-progress -f llama/pom.xml compile .github/build.sh -DBUILD_TESTING=ON + # Most patches have a runnable guard that reds this job if they go missing (a link error, or + # test_utils.cpp / test_model_split.cpp). patches/0010 has none — it casts one enum inside a + # `static` function unreachable from jllama_test, so reverting it leaves ctest fully green and + # only a model-gated Java test notices. This is the always-on, model-free check for it. + - name: Verify llama.cpp patches are applied + run: .github/verify-patches-applied.sh - name: Run C++ unit tests run: ctest --test-dir llama/build --output-on-failure diff --git a/TODO.md b/TODO.md index c8334d9e..dcc8f887 100644 --- a/TODO.md +++ b/TODO.md @@ -96,9 +96,11 @@ workflow in `.github/workflows/`). It contributes to the `mergeable_state: block ### Upstream PR submissions — drop the carried patches (open) -Six of the seven `patches/` are upstream-submittable verbatim; each accepted PR (once the pin is -bumped past it) deletes a patch from the bump checklist. (`0003` is a carry of an already-open -upstream PR #22393 — it drops automatically when that merges.) +There are **nine** patches today (`0001`–`0003`, `0006`–`0008`, `0010`–`0012`). **Eight are +upstream-submittable verbatim**; each accepted PR (once the pin is bumped past it) deletes a patch +from the bump checklist. The exception is **`0003`**, a carry of upstream PR #22393, which upstream +**closed without merging** — it is permanent and will never be droppable via a bump. (`0003` used to +be described here as "drops automatically when that merges"; it will not.) - **`0001` Windows arg-parse embed guard** (against #24779): `common_params_parse` trusts the caller's argv; `common_params_parse_main()` keeps the standalone tools' UTF-8 recovery. Ship with the @@ -113,10 +115,22 @@ upstream PR #22393 — it drops automatically when that merges.) - **`0007` `llama_server_attach`** (HTTP frontend on an existing `server_context`). - **`0008` `LLAMA_SERVER_WORKER_CMD` router worker override** (also useful for containerized/wrapped deployments). -- **`0009` guard `posix_spawn_file_actions_addchdir_np` on old glibc** (b10154 cross-compile break on - manylinux2014 / glibc 2.17 and manylinux_2_28 / glibc 2.28; adds an overridable - `SUBPROCESS_HAVE_CWD` probe via `__GLIBC_PREREQ(2, 29)` — submitted as sheredom/subprocess.h#104, - drops automatically once llama.cpp bumps the vendored pin). +- **`0010` cast `vocab_type` for `common_json`** (one line; upstream regressed `GET /models` + + `GET /v1/models` to emit `true`/`false` instead of the numeric vocab type when they flipped the + `json` alias to `common_json` at b10585/#27511). **Not yet filed upstream.** +- **`0011` lenient invalid-UTF-8 in the PEG parser** (one malformed byte from the model turns a + finished generation into an HTTP 500; the `INVALID` branch ignores leniency while the `INCOMPLETE` + branch beside it honours it). Ships an upstream `tests/peg-parser/test-unicode.cpp` case. + **Not yet filed upstream.** +- **`0012` guard the zero split-sum and name the device index** (a GPU reporting zero free memory — + or a cancelling `--tensor-split` such as `-ts 1,-1` on any backend — makes every model load fail + with the unactionable `error loading model: vector`). Ships an upstream `tests/test-model-split.cpp`. + **Not yet filed upstream.** + +(`0009` is **not** in this list and the number is burned: upstream merged the subprocess.h fix via +ggml-org/llama.cpp#26606, so the patch was dropped at the b10280 bump. `0013` is likewise gone — +upstream merged this project's own PR ggml-org/llama.cpp#28775 and it was dropped at b10948. Both +drops are recorded in `CLAUDE.md` under the patch table.) ### llama.cpp upstream feature exposure (queued, deferred by policy) diff --git a/llama/spotbugs-exclude.xml b/llama/spotbugs-exclude.xml index 38544097..0858fe28 100644 --- a/llama/spotbugs-exclude.xml +++ b/llama/spotbugs-exclude.xml @@ -365,20 +365,6 @@ SPDX-License-Identifier: MIT - - - - - - -