[4evy]

Refactored koboldcpp

Mainly, I split it into separate packages for each backend and added some missing dependencies

The default package now builds the CPU version, and there are separate variants for CUDA, ROCm, Vulkan, and Metal.

Tested CUDA and CPU with SmolLM2-135M-Instruct-Q2_K.gguf

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.
• Follows the automation/AI policy.

[SuperSandro2000]

I would appreciate a slightly cleaner history. Some fixups including formatting where done a few commits later which is kinda noisi.

[4evy]

✅ nixpkgs-review result

Generated using nixpkgs-review-gha (b0ab82d)

Command: nixpkgs-review pr 537338
Commit: 6a618c315df32ad5564ea3278e5f41d58603119f (subsequent changes)
Merge: 875b200a9fdebeecbae6389424269e7af2d6ac9b

Logs: https://github.com/4evy/nixpkgs-review-gha/actions/runs/32663639978/attempts/1

---

x86_64-linux

✅ 4 packages built:

• koboldcpp
• koboldcpp-cuda
• koboldcpp-rocm
• koboldcpp-vulkan

---

aarch64-linux

✅ 3 packages built:

• koboldcpp
• koboldcpp-cuda
• koboldcpp-vulkan

---

x86_64-darwin

✅ No rebuilds

---

aarch64-darwin (sandbox = relaxed)

✅ 3 packages built:

• koboldcpp
• koboldcpp-metal
• koboldcpp-vulkan

[laczek14]

i tried the release26.05 koboldcpp package aka. building it into my NixOS system and it seems to not work?
does your build checks fix it?(gemma-4-32B some RP finetune is what i tried to launch)

File "/nix/store/62hlpn50p00r30jzz9n6wbwfk52i5k5b-koboldcpp-1.119/bin/.koboldcpp.unwrapped-wrapped", line 12955, in <module>
    main(launch_args=parser.parse_args(),default_args=parser.parse_args([]))
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/62hlpn50p00r30jzz9n6wbwfk52i5k5b-koboldcpp-1.119/bin/.koboldcpp.unwrapped-wrapped", line 11486, in main
    kcpp_main_process(args,global_memory,using_gui_launcher)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/62hlpn50p00r30jzz9n6wbwfk52i5k5b-koboldcpp-1.119/bin/.koboldcpp.unwrapped-wrapped", line 12134, in kcpp_main_process
    init_library() # Note: if blas does not exist and is enabled, program will crash.
    ~~~~~~~~~~~~^^
  File "/nix/store/62hlpn50p00r30jzz9n6wbwfk52i5k5b-koboldcpp-1.119/bin/.koboldcpp.unwrapped-wrapped", line 979, in init_library
    handle = ctypes.CDLL(os.path.join(dir_path, libname))
  File "/nix/store/xqnbm0vgqcq9b1b54c80qj9s0qhbwa08-python3-3.13.15/lib/python3.13/ctypes/__init__.py", line 361, in __init__
    self._handle = self._load_library(name, mode, handle, winmode)
                   ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/xqnbm0vgqcq9b1b54c80qj9s0qhbwa08-python3-3.13.15/lib/python3.13/ctypes/__init__.py", line 403, in _load_library
    return _dlopen(name, mode)
OSError: /nix/store/62hlpn50p00r30jzz9n6wbwfk52i5k5b-koboldcpp-1.119/bin/koboldcpp_cublas.so: undefined symbol: cuMemRelease

[4evy]

@laczek14 This PR is against master, so it won’t fix 26.05 on its own. But yeah, this looks like the same issue the PR fixes. I tested the CUDA build on my NVIDIA machine and the library loads correctly there

[4evy]

✅ nixpkgs-review result

Generated using nixpkgs-review-gha (05be241)

Command: nixpkgs-review pr 537338
Commit: 01985a4d10e2cd962a450b841d0833f8d565319d (subsequent changes)
Merge: b56aa4f4dd679e8d280f73e90da37f8f46b788bc

Logs: https://github.com/4evy/nixpkgs-review-gha/actions/runs/36410323418/attempts/1

---

x86_64-linux

✅ 4 packages built:

• koboldcpp
• koboldcpp-cuda
• koboldcpp-vulkan
• koboldcpp-rocm

---

aarch64-linux

✅ 3 packages built:

• koboldcpp-cuda
• koboldcpp
• koboldcpp-vulkan

---

x86_64-darwin

✅ No rebuilds

---

aarch64-darwin (sandbox = relaxed)

✅ 3 packages built:

• koboldcpp
• koboldcpp-metal
• koboldcpp-vulkan

[4evy]

✅ nixpkgs-review result

Generated using nixpkgs-review-gha (a0a3a8d)

Command: nixpkgs-review pr 537338
Commit: e1b0127126aa75dd9dde65b23c5a2a85f4c7ec56 (subsequent changes)
Merge: 3326c7c11c9d68f1817520114c9f8d666e876729

Logs: https://github.com/4evy/nixpkgs-review-gha/actions/runs/37211914018/attempts/1

---

x86_64-linux

✅ 4 packages built:

• koboldcpp
• koboldcpp-vulkan
• koboldcpp-cuda
• koboldcpp-rocm

---

aarch64-linux

✅ 3 packages built:

• koboldcpp
• koboldcpp-cuda
• koboldcpp-vulkan

---

aarch64-darwin (sandbox = relaxed)

✅ 3 packages built:

• koboldcpp
• koboldcpp-metal
• koboldcpp-vulkan

[4evy]

✅ nixpkgs-review result

Generated using nixpkgs-review-gha (c28ca54)

Command: nixpkgs-review pr 537338
Commit: c35a372aa83b19580f491188efdf6b63cf2a780c (subsequent changes)
Merge: 102fa85964300f5398b64390b585f24f5305a2a5

Logs: https://github.com/4evy/nixpkgs-review-gha/actions/runs/41240180343/attempts/1

---

x86_64-linux

✅ 4 packages built:

• koboldcpp
• koboldcpp-cuda
• koboldcpp-rocm
• koboldcpp-vulkan

---

aarch64-linux

✅ 3 packages built:

• koboldcpp
• koboldcpp-vulkan
• koboldcpp-cuda

---

aarch64-darwin (sandbox = relaxed)

✅ 3 packages built:

• koboldcpp
• koboldcpp-metal
• koboldcpp-vulkan