Skip to content

fix: report no requests figure where the engine exposes none - #247

Open
outofcoffee wants to merge 3 commits into
mainfrom
fix-llamacpp-requests-metric
Open

outofcoffee wants to merge 3 commits into
mainfrom
fix-llamacpp-requests-metric

Conversation

@outofcoffee

Copy link
Copy Markdown
Collaborator

The llama.cpp metrics tile's requests figure was always 0, even though the
other engine-sourced counters worked. The collector read the figure from
request_success_total for every engine family, but that metric is vLLM's:
llama.cpp's server exposes no cumulative request counter at all, so the figure
was a zero no engine ever produced.

Summary

  • The statistics now carry the request count only where the engine family's
    metrics expose a cumulative request counter (vLLM's
    request_success_total); a family that exposes none (llama.cpp, today)
    omits the field rather than reporting a zero.
  • TokenStats.Requests becomes optional (*int, omitempty) across the
    daemon's reply, the remote relay, and the metrics JSON output.
  • The shared token block — dashboard tile, the metrics bar and table
    formats, the serve view — draws the requests: line only for a figure the
    statistics carry.
  • The unit-test fixture and the two example stacks' engine configs stop
    fabricating a llamacpp:request_success_total line a real engine would
    never serve.
  • The OpenAPI description and the control plane's TypeScript mirror mark the
    field as optional.

Implementation details

The absence is modelled as a pointer, not a zero the renderers hide: a
genuine zero from an engine that exposes the counter (started, nothing served
yet) still serialises, while a family that exposes none omits the field
entirely. That keeps "no figure" and "a zero figure" distinguishable in the
data, the same rule the engine-metrics spec already applies to the last-active
pair and the system stats — and it is what lets every rendering surface drop
the line without special-casing a runner.

Verified against a live llama.cpp server run with --metrics: the collector
yields the token counters with no requests field, and no surface draws a line
for it.

Fixes #212.

Pete Cornish added 3 commits September 25, 2026 20:31
The parser read the requests figure from request_success_total for
every engine family, but that metric is vLLM's: llama.cpp serves no
cumulative request counter, so the figure was always zero. The
statistics now carry the request count only where the engine family's
metrics expose one - a missing figure, not a zero the engine never
produced - and the token block draws the line only for a figure the
statistics carry.
The parser and the token block are exercised for the absent case at
each surface that draws the block - the daemon's metrics reply, the
dashboard tile, the serve view and the metrics formats - and for the
wire shape that makes an absent figure distinguishable from a genuine
zero.
@outofcoffee outofcoffee added the bug Something isn't working label Sep 25, 2026
@outofcoffee
outofcoffee marked this pull request as ready for review September 25, 2026 23:03

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: in the metrics tile for llama.cpp, requests is always 0

1 participant