ghostseat

Root-cause analysis for FlexNet Publisher (lmgrd) licence logs.

A ghost seat is a licence still checked out by a client that is already gone. It is the most common reason a pool reads as full while nobody is working, and the licence server will not tell you it happened.

A licence server writes one line per checkout and one per return, with no date and no session id. When an engineer reports "no licences available", answering why means reconstructing, by hand, who held what at that moment — across a log that rolls over midnight and interleaves every user on the site.

ghostseat does that reconstruction and names the cause:

$ ghostseat /var/tmp/agilent.log

[CRITICAL] pool-exhaustion
  dave@n04 denied "ads_rf" with 3 seat(s) held by 3 client(s)
  Denied at line 16. Daemon note: (Licensed number of users already reached. (-4,342)).
  Longest-held seats at that moment:
    alice@n01 held 20m 00s (line 9)
    bob@n02 held 19m 30s (line 10)
    carol@n03 held 19m 00s (line 11)
  lines: 16 9 10 11

[WARNING] unreturned-license
  ghost@wkstn42 holds "ads_core" for 8h 55m 00s with no IN
  Checked out at line 8 and never returned within the log. The seat stays consumed
  until the vendor daemon's TCP timeout expires, so the pool can read as full while
  the client is gone. Confirm with lmstat -a, then reclaim with lmremove if the
  client is dead.
  lines: 8

2 critical, 3 warning, 1 info

Exit status is 0 clean, 1 findings, 2 unreadable — so it drops into a cron job or a CI gate without parsing its output.

Build

No dependencies beyond a C++17 compiler.

cmake -S . -B build && cmake --build build -j
ctest --test-dir build --output-on-failure

Use

ghostseat debug.log                      # human-readable report
ghostseat debug.log --json               # stable schema, for tooling
ghostseat debug.log --quiet              # one line per finding
grep -F '"ads_rf"' debug.log | ghostseat -    # stdin
ghostseat debug.log --stale-hours 2      # tighten the leak threshold

What it detects

Finding	Meaning	Usual cause
pool-exhaustion	A denial, with the seats held at that instant	Genuine contention, or a leak masquerading as one
denial	A denial with nobody holding the feature	Expired feature, host mismatch, options-file rule
unreturned-license	Checkout with no return	Client died; the daemon holds the seat until TCP timeout
double-checkout	One client, two concurrent seats	Duplicate grouping (DUP_GROUP) not collapsing
license-churn	Many sub-second checkout cycles	A licence call inside a loop
unsupported-feature	Request for a feature not served	Client/licence-file mismatch
idle-reclaim	Daemon reclaimed the seat itself	Clients not releasing; the timeout is hiding a leak
truncated-log	Check-ins with no checkout	Log rotated mid-session; durations are lower bounds

Separating those three categories — contention, waste, misconfiguration — is the point. Only the first is fixed by buying more seats.

How it works

debug.log ──▶ parser ──▶ sessions ──▶ detectors ──▶ report ──▶ explain.py
              events     OUT/IN       findings      text/json   analysis
                         paired                                 request

Three details drive most of the design:

The clock has no date. Every line is HH:MM:SS. A checkout at 23:59 returned at 00:01 reads as minus 86,280 seconds held, so the parser detects the backward jump and stitches a monotonic timeline — while tolerating the small out-of-order writes that concurrent daemon threads produce, which are not midnight.

Check-ins do not name their checkout. Pairing uses the licence handle when the daemon logged one, and falls back to oldest-outstanding-first for the same user, host, and feature when it did not. Getting this wrong silently misreports every hold time in the file.

An unreturned seat is still held on the last line. Clamping open sessions to the end of the log makes a denial logged as the final record show zero holders — which is exactly the case worth catching. Open intervals stay open.

Writing it up

explain.py turns findings into a grounded analysis request for one of three readers, printed to stdout. Paste it into whichever assistant you use:

ghostseat debug.log --json | ./explain.py                      # support engineer
ghostseat debug.log --json | ./explain.py --audience manager   # buy more seats?
ghostseat debug.log --json | ./explain.py --audience ticket    # customer reply

The split is deliberate: the engine decides what is true, the model only explains it. Detection is deterministic and unit-tested; the request carries a few kilobytes of structured findings that already hold their line numbers, and instructs the model never to introduce a user, feature, or count absent from that input. Feeding raw logs to a model instead re-reads megabytes per question and produces claims nobody can trace back to a line.

There is no API client here — no key, no network, no vendor SDK. That keeps the whole path reproducible by anyone who clones the repo, and keeps the constraints legible as text rather than buried in a call.

Tests

ctest --test-dir build

Six suites: parser, sessions, and detectors in C++; the CLI end-to-end through real files and exit codes; the prompt builder; and a generator round-trip.

The round-trip is the interesting one. Real licence logs are customer data and cannot be committed, so tools/gen_log.py synthesises them — and because it plants each fault deliberately, it knows the exact set of findings the analyzer should return. tests/test_harness.py generates 120 logs — four scenarios, 25 seeds, and two site scales — and asserts the two sets match exactly, which catches both a detector that stops firing and a detector that invents findings in healthy traffic. The clean scenario is the control: it must stay silent, and the 450-seat scale catches faults planted against hardcoded seat counts that quietly stop happening on a big pool.

The suite was checked by injecting a fault (disabling the leak threshold) and confirming it goes red, rather than trusting a green run.

Every path in the repo is covered, because every path is local: there is no network call anywhere in it.

Performance

On a synthetic 121k-line (8 MB) 24-hour log from a 450-seat site: 98 ms, about 1.2M lines/s, single-threaded, best of five on a warm cache (gen_log.py --scale 150). Parsing is hand-rolled string_view scanning rather than std::regex — a deliberate choice for a hot path this narrow, though the two were not benchmarked against each other here.

Limits

• Parses the vendor-daemon debug log. Report logs (lmreread/lmstat binaries) are a different, undocumented binary format and are out of scope.
• Concurrency at a denial is an O(denials × sessions) scan, and duplicate detection is O(k²) per client. Both are marked in the source with the sweep or interval-index that replaces them if logs outgrow it.
• Single-threaded. It is I/O bound on a cold file well before it is CPU bound.
• Thresholds are heuristics with defaults chosen for an 8-hour working day; a 24/7 batch farm wants --stale-hours raised.