Ephemeral consumer leak: thousands of never-delivered consumers accumulate on production (5,099 created within seconds)

[majkelx]

Observation (production OCA broker, 2026-07-22)

JetStream account state: 8,838 consumers, including 6,556 on the tic_status stream alone. Creation-date histogram of tic_status consumers (from /jsz?consumers=true):

2026-06-24: 5,099   ← burst, created within seconds (~21:45:15)
2026-06-17:   276
2026-06-25:   230
2026-05-20:   138
...

5,099 consumers created in one burst on 2026-06-24 — the day the halina service (serverish 2.0.5 client) was restarted. This looks like a creation retry storm: consumer-create requests being retried in a loop where each attempt actually succeeds server-side (response lost/slow?), each retry registering a fresh ephemeral consumer. Pattern-wise a cousin of #30's "retry forever on 4xx" (fixed), but for the create path + non-idempotent creates.

These consumers show Last Delivery: never and survive for a month+. Note: consumers created by current serverish 2.0.5 carry inactive_threshold: 300s (observed on a live one), so month-old leftovers either predate that default or were created through a path that doesn't set it — worth auditing both.

Impact

• 6.5k consumers on one stream degrade JS API responsiveness for every client of that broker (slow consumer-create/info responses → more client-side timeouts → more retries — self-reinforcing).
• Operationally invisible until someone lists consumers.

Suggested fixes

1. Idempotent creates: name ephemeral consumers deterministically per reader instance (client-generated name), so a retried create with the same name is a no-op server-side instead of a new consumer (JetStream treats same-name create as idempotent update).
2. Ensure every consumer-creation path sets inactive_threshold (bounded, e.g. 300 s), so even leaked ones self-destruct.
3. MsgReader.close() should explicitly delete its ephemeral consumer (best effort) instead of relying on inactivity GC.
4. One-off ops cleanup script for existing brokers: delete consumers with delivered.last == never older than N days (happy to contribute).

[majkelx]

Attribution update (from /jsz?consumers=true&config=true fingerprinting of all 8.8k consumers):

The immortal consumers form one clear cohort: deliver_policy: last_per_subject, inactive_threshold: 60,000,000 s (~1.9 years!), across subject sets that fingerprint GUI client sessions: tic.status.<tel>.{mount,dome,camera,filterwheel,covercalibrator,toi.status,toi.ob,fits.pipeline.*} (468 copies each), tic.journal.<tel>.quality (468), telemetry.weather.davis (156), tic.config.observatory (156), svc.publish.guider.<tel>.guider_beso.* (232 each), svc.status.{guiding_svc,thumbnail_svc}.> (252/151). I.e. TOI / oca_monitor / guider-UI sessions — one full reader set leaked per app session, accumulated since ~May.

Root cause of immortality: pre-2.0 serverish (msg_reader default was 'inactive_threshold': 60 before commit 92aee35 "stability of the reader", Dec 2025, which changed it to 300) combined with a unit-conversion bug: the server-side stored value is 60,000,000 seconds — exactly the old 60 inflated ×10⁶ somewhere in the seconds→ns conversion chain of the old client stack. Current 2.0.x consumers on the same broker correctly show 300 s (or 60 s) and self-clean.

Consequences for the fix list:

• Items 1–3 stand, but the biggest operational lever is upgrading deployed GUI clients (TOI, ocam, guider UI on observer machines) to serverish ≥2.0 — they are the ones minting immortal consumers every session.
• The cleanup script (item 4) can safely target the cohort by fingerprint: inactive_threshold > 1 day && delivered.last == never (or num_ack_pending == 0 && last_delivery == never), which precisely selects the leaked set.
• The 2026-06-24 burst of 5,099-in-seconds remains consistent with a create-retry storm from one such old client during a network flap.