Skip to content

Launch the llama.cpp CUDA recipe from C++ - #14

Open
Quixyl-eng wants to merge 1 commit into
mainfrom
feature/cuda-launch
Open

Quixyl-eng wants to merge 1 commit into
mainfrom
feature/cuda-launch

Conversation

@Quixyl-eng

Copy link
Copy Markdown
Contributor

Summary

  • Add exvram-cuda, a Windows launcher that starts the CUDA llama-server build with the same full-GPU flags the benchmarks already use.
  • The decoder stays on CUDA0. --embedding-gpu on pins the token embedding there, matching the VRAM planner. The server stops when the process exits.

Test plan

  • Built with Visual Studio 2022 Build Tools (/W4, no warnings) on the RTX 5060 checkout.
  • One prompt run printed gpu_layers 25/25, CUDA0 model buffer 1211.05 MiB, graph_splits 1, ggml_cuda_loaded yes, and the answer 4. llama-server was gone afterward.
  • Confirm a machine without the local GGUF and runtime fails before starting a server, as the README describes.

exvram-cuda starts llama-server with the measured full-GPU flags and refuses to answer until the decoder is on CUDA0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant