feat(spur-cli): add scontrol show assoc_mgr for limits against live usage

[nikhilsk]

What this adds

scontrol show assoc_mgr: how much of a QOS or association each user is holding right now, next to the caps that govern them, and which caps are already exceeded.

Operators had no way to see this. The figures existed only inside the admission gate, which recomputes a user's running count and TRES per candidate job and keeps nothing queryable, so counting squeue rows was the only recourse — and that cannot show a cap already being exceeded. Usage over a cap is not a contradiction: caps apply when a job is admitted and running jobs are never re-checked, so tightening a cap under running work leaves exactly that state, silently.

QOS Records
QOS=highprio MaxWall=01:00:00 MaxJobsPU=2 MaxSubmitJobsPU=N MaxTRESPU=node=4
   GrpJobs=N(9) GrpSubmitJobs=N(11) GrpTRES=cpu=N(36),node=16(9)
   User=alice MaxJobsPU=2(6) MaxSubmitJobsPU=N(7) MaxTRESPU=cpu=N(24),node=4(6) OverLimit=MaxJobsPU,MaxTRESPU
   User=bob MaxJobsPU=2(1) MaxSubmitJobsPU=N(1) MaxTRESPU=cpu=N(4),node=4(1)

Association Records
Account=tenant-a MaxWall=N
   GrpJobs=N(1) GrpSubmitJobs=N(1) GrpTRES=node=N(1)
   User=alice MaxJobs=4(1) MaxSubmitJobs=N(1) MaxTRES=cpu=N(4),node=N(1)

Approach

A new GetAssocMgrInfo RPC on SlurmController — the controller is the only side that sees the live job table, while the accounting service owns the definitions. Aggregation runs per (user, QOS) and (user, account) through the same sum_running_tres the gate uses, so node figures are distinct-node occupancy and CPU follows each job's request. A record reads as the scheduler sees it rather than as a second opinion that could quietly disagree.

The cap comparison lives in spur-core beside the limits, not in the CLI, so the view does not re-implement enforcement semantics (including that a TRES dimension capped at 0 is not a cap). It sits next to the admission checks it deliberately differs from: those project a candidate job onto current usage, this reports what already stands over a cap.

Following Slurm's shape

The output follows Slurm's assoc_mgr where that shape is a contract other tooling already depends on:

• Limit(Consumed) in one field, N for no limit — node=4(6) is a cap of four with six in use. Scripts in the wild parse exactly this (splitting on the parenthesis), so rendering caps and usage as separate fields would have broken them for no gain. Applied per count and per TRES dimension, listing the union of dimensions capped and in use.
• Every defined QOS and association is listed, not only those with queued work, which is what Slurm's cache dump does. An over-tight cap on an idle QOS is exactly the kind of mistake that hides otherwise. This needed enumerate-all accessors on both caches. A QOS deleted after its jobs queued also stays reported while those jobs hold resources.
• Per-user entries nest under the record they qualify, as Slurm's User Limits= does. This also stops a scope's group figures from being repeated once per user, which the first draft did.

Where it diverges, deliberately: users are listed on their own lines rather than inside a single User Limits= field (more readable and greppable), an account's group figures are stated once for the account instead of per association row, and Slurm's accounts=/qos=/flags= selectors and fairshare internals (SharesRaw, UsageRaw, Lft-Rgt, GrpTRESMins) are not included. All of this is in the migration guide.

Design choices

• A QOS states its per-user caps once, on the scope line. It caps every user identically, so repeating them per user is noise — and stating them on the scope is what keeps them visible when nobody is using the QOS. An association's caps are per (user, account), so they ride on each user's line instead, and scope_caps is absent there.
• Caps and consumption stay separate fields on the wire. Composing the Limit(Consumed) display is the client's job; a machine consumer should not have to take a display string apart again.
• Cap is spelled neutrally and the server picks the wording. A QOS caps with MaxJobsPU/MaxTRESPU, an association with plain MaxJobs — same breach, different name. Deciding it controller-side means every consumer, REST included, reads one verdict.
• A cold accounting cache reports LimitsReadable=NO rather than letting an unreadable cap look like an absent one. Usage stays accurate; only the caps are missing.
• Reused sacctmgr's blank_if_unset (promoted to pub(crate)) instead of copying it, so a cap renders identically in both commands.

Proto

Additive only — a new RPC plus AssocMgrRecord, AssocMgrUserRecord, AssocMgrCaps, GetAssocMgrInfoRequest, GetAssocMgrInfoResponse. No existing message, field, or tag is touched.

Testing

Twenty new tests:

• spur-core (7) — the cap comparison and TresRecord::types(): over the job-count cap, exactly at a cap (a ceiling is not a threshold), unset caps ignored, each TRES dimension compared the way the gate compares it, a zero dimension treated as uncapped, group caps reported from the scope and never duplicated onto users, and dimension ordering stable.
• spurctld (9) — through the real cluster manager with jobs driven to running: per-user records against a QOS cap with the breach named, a QOS nobody is using still reported with its caps, a QOS the cache no longer knows kept while its jobs run, distinct-node counting, one-user filtering that leaves group totals intact, usage reported when caps could not be read, QOS and account records kept apart with each hierarchy's own caps, a defined association with no jobs still reported, and nothing at all for unscoped work.
• spur-cli (7) — the rendered layout: Limit(Consumed) with N, association labels without the per-user suffix, an idle scope with caps and no users, OverLimit omitted when compliant, a group breach on the scope line, an explicitly marked empty section, a zero cap rendering as 0, and TRES dimensions drawn from either side.

Full cargo test --locked (3,318 tests), cargo clippy --workspace --exclude spur-ffi --all-targets --locked warning-free, cargo fmt --all -- --check clean. Docs: a new section in the monitoring guide and a compatibility row in the migration guide.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 91.43519% with 74 lines in your changes missing coverage. Please review.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #746      +/-   ##
==========================================
+ Coverage   80.10%   80.24%   +0.14%     
==========================================
  Files         184      184              
  Lines       87417    88579    +1162     
==========================================
+ Hits        70024    71076    +1052     
- Misses      17393    17503     +110     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Adds an operator-facing “limits vs. live usage” view to Spur’s Slurm-compatible surface by introducing a new controller RPC and wiring it through scontrol show assoc_mgr, with docs describing the new output shape and semantics.

Changes:

• Add GetAssocMgrInfo to SlurmController and new AssocMgr* proto messages for reporting scope + per-user usage and caps.
• Implement controller-side aggregation across live job table + accounting caches, and render Slurm-shaped assoc_mgr output in spur-cli.
• Document the new command in the monitoring guide and note compatibility/differences in the Slurm migration guide.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
proto/slurm.proto	Adds new RPC/messages for assoc_mgr live limits/usage snapshot.
docs/user-guide/monitoring-jobs.rst	Documents scontrol show assoc_mgr usage, output, and interpretation.
docs/migration-from-slurm.rst	Notes compatibility contract and intentional divergences vs Slurm.
crates/spurctld/src/server.rs	Implements get_assoc_mgr_info RPC and proto conversion helpers.
crates/spurctld/src/limits_cache.rs	Exposes QosCache::all() to enumerate defined QOS for reporting.
crates/spurctld/src/cluster.rs	Computes assoc_mgr usage records from live jobs + cached definitions; adds tests.
crates/spurctld/src/association_cache.rs	Exposes AssociationCache::all() to enumerate defined associations for reporting.
crates/spur-core/src/accounting.rs	Adds cap/usage structs and exceeded-cap comparison logic; adds TresRecord::types() + tests.
crates/spur-cli/src/sinfo.rs	Extends mock RPC surface to include get_assoc_mgr_info for CLI tests.
crates/spur-cli/src/scontrol.rs	Adds scontrol show assoc_mgr rendering and related unit tests.
crates/spur-cli/src/sacctmgr.rs	Makes blank_if_unset reusable for consistent rendering.
crates/spur-cli/src/mock_controller.rs	Extends mock controller implementation with get_assoc_mgr_info.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[shiv-tyagi]

Posted comments, PTAL

One additional note (no line in this diff): GetAssocMgrInfo should be added to the diagnostic RPC exclusion list in crates/spurctld/src/rpc_middleware.rs alongside GetSchedStats, or operator polling of scontrol show assoc_mgr will skew RPC latency stats.

[yansun1996]

Nice feature — the operator gap it closes is real, and computing the verdict controller-side next to the admission checks (rather than re-deriving it in the CLI) is the right call. A few things worth a look before merge.

Authorization on the new RPC. GetAssocMgrInfo doesn't consult the caller's identity — the request's user is used purely as a filter, and an empty value means "no filter". The scope lines (Grp*, plus every QOS and account name) aren't narrowed by the filter at all, so any caller can enumerate cluster-wide account/QOS inventory and per-user consumption. get_jobs handles this with Self::authoritative_user(&mut req.user, __identity.as_ref()), and list_tokens gates enumeration on require_admin. Would it be worth following the get_jobs shape here, so a non-admin is scoped to themselves and admins can still pass an arbitrary user?

A couple of display/semantics mismatches — details inline: the scope line's MaxTRESPU= renders empty where its siblings render N; a non-users= selector is silently taken as a literal username; and the docs' "a literal 0 is a real cap" holds for count caps but not for TRES dimensions, where 0 is treated as unset.

Tests. Three gaps: scope_exceeded_caps_covers_the_group_caps_only can't fail as written (the user's caps are all None, so exceeded_caps() is empty for any implementation); the CLI tests all call render_assoc_mgr directly, so the assoc_mgr dispatch arm, the users= strip and the LimitsReadable banner are untested; and limits_readable == true is never asserted.

Also worth considering as follow-ups: GrpWall, MaxTRESPerJob and MaxSubmitJobsPerAccount are enforced but absent from the record, so a job pended by QosGrpWallLimit shows nothing here.

The LimitsReadable=NO wording points already raised below look right to me, and the same overstatement applies when accounting is disabled entirely — the refresh loop only starts on a successful DB connect, so the banner prints permanently and reads as a fault rather than "there are no caps".

[yansun1996]

Following up on the last paragraph of my earlier review — #744 now defines the predicate this banner should be using: !enabled() || (qos_loaded && assoc_loaded), evaluated per-cache rather than conjoined. That fixes both the accounting-disabled case I flagged and a second one I'd missed: with one cache warm and one cold, LimitsReadable=NO prints flat while jobs are visibly running under enforced caps. #744 also adds an AccountingUnavailable pending reason that this record has no notion of — since #744's follow-ups point at scontrol show assoc_mgr as the surface for exactly this, it'd be worth surfacing here. Details on #744; no merge conflict between the two.

[nikhilsk]

reply to 3876320563-followup / ISSUE-COMMENT (LimitsReadable + #744)

Implemented the per-cache predicate now:
limits_readable(enabled, qos_loaded, assoc_loaded) = !enabled || (qos_loaded && assoc_loaded),
as a pub(crate) free function so it's a one-line swap to #744's helper once
that lands on the branch (no reference to #744 in the source per repo rules).
This fixes the accounting-disabled case — an off cluster now reports readable
rather than a permanent LimitsReadable=NO that read as a fault — and the CLI
banner wording is softened to "some limits below may be missing" for the
enabled-but-cold case. Unit test limits_readable_gates_on_accounting_and_each_cache
covers off / warm / each-cold.

I did not add the AccountingUnavailable pending-reason surfacing here —
that machinery is #744's, and wiring a half of it into this record before it
merges would just have to be reconciled. Better to add it in one place once
#744 is in. Flagging so it isn't lost.

---

[shiv-tyagi]

LGTM

[yansun1996]

Re-reviewed the authorization path end to end: the filter now comes from the verified identity rather than the request field, an empty caller isn't treated as privileged, and scopes the filtered user has no part in are dropped rather than enumerated. The limits_readable predicate and its descriptions agree now. Approving.