fleetwatch

Supervision for agents nobody is watching.

Observability tools assume a human is looking at the dashboard. A fleet of autonomous agents inverts that assumption: the agents are awake, and you are asleep. What you need at 3am is not a trace viewer. It is a small number of correct decisions made without you — and a very small number of pages that are worth waking up for.

fleetwatch is the operational layer that came out of running unattended agents against real money for several months. Every design note in this repository is a lesson that was paid for by an incident.

The four questions it answers

Is the agent actually working, or merely alive? A crashed process pages you. A process that is alive, looping and producing nothing does not. deadman inverts the default: the agent must keep saying it is working, and the absence of that statement is the alarm.

Is this alarm worth a human? Most are not. notify routes by a criterion with exactly four cases (see the module docstring); everything else is filed for the agent to triage and repair itself. A supervision system that pages a human for everything becomes a paging system, and a paging system that cries wolf gets muted — after which you have no supervision at all.

Is this thing supposed to be down? When you deliberately decommission something, every watcher aimed at it starts screaming, and the reflex is to mute the alarm. That is how supervision rots. retire records that the SUBJECT is out of service, with an author, a date and a restore path, and watchers consult that record before firing. The alarm stays armed for everything else it covers.

Has the agent side stopped coping? Filing an alert for the agent is only half a loop. inbox drains the queue, persists attempts, and escalates two distinct symptoms: a record nobody has touched (the agent is not waking) and a record picked up repeatedly without resolution (the agent is awake and cannot fix it). Both are the third case above.

What it does not do

It does not track spend. Metering an agent's API burn is a real problem and this package does not solve it; a previous version of this README implied otherwise, which was the same over-claiming the rest of the document argues against.

It does not tell you your host died. It cannot: a deadman running beside the agent dies with the machine, and the one silence you most needed to hear is the one nobody is left to notice. Covering that needs a witness on other hardware. deadman.beat(witness_url=...) is the seam — point it at anything that escalates when your pings stop.

This limit is stated here rather than buried, because a supervision tool that quietly fails to cover its own failure mode is precisely the thing this repository argues against.

Install

pip install -e .

State lives in ~/.fleetwatch (override with FLEETWATCH_DIR).

Use

from fleetwatch import notify, deadman, retire

# in the agent's work loop — where the WORK happens, not in a bare timer thread
deadman.beat("collector", note="batch 41")

# from a supervisor, out of band
ok, quiet_for = deadman.check("collector", max_silence=600)
if not ok and not retire.is_retired("collector"):
    how_long = f"{quiet_for:.0f}s" if quiet_for is not None else "since forever (never reported)"
    notify.notify(f"collector silent for {how_long}", source="supervisor",
                  key="collector-silent", dedup_sec=3600)

# deliberate shutdown: silence the subject, not the alarm
retire.retire("collector",
              reason="pipeline replaced by v2",
              decided_by="owner, 2026-08-31",
              restore_hint="re-enable the unit, then delete this marker",
              expected_alarms=("collector-silent",))

Reaching a human is a pluggable transport, because who you page is not this library's business:

from fleetwatch import transports
notify.set_human_transport(transports.ntfy("my-fleet-topic"))     # or .webhook(url) / .telegram(...)

Without a transport installed, a human-priority alert degrades to the inbox and says so loudly in the log — an undelivered page is itself case 3 of the criterion.

A complete, runnable supervisor loop wiring all of this together is in examples/supervisor.py — twenty lines you copy and own, rather than a daemon that owns your escalation policy.

Design notes worth reading before you copy this

• The mute guard sits on the transport, not on caller discipline. A mute that every test is obliged to remember does not hold a boundary: the first caller who forgets reaches a human at 3am. Detection defaults to safe.
• Self-tests need their own addressee. A pipe check that files a real record into the live queue is a real alert — it will wake somebody, and an honest "ignore, this is a test" in the body does not stop that. Self-tests write straight to the handled archive: the pipe is proven all the way to the file, and nothing wakes.
• A dead watchdog is worse than no watchdog. It manufactures confidence. An inbox that is silent because it is broken must be distinguishable from an inbox that is silent because all is well — that is why "delivery machinery is broken" is one of the four cases that may wake a human.
• "No data" is not "fine". A check with nothing to check returns not-ok — and returns None, not a sentinel number, so no caller prints "silent for -1s".

"Alertmanager has had silences for a decade"

It has, and they are a different thing. A silence attaches to the ALARM, carries a duration, and expires on its own. A retirement marker attaches to the SUBJECT, never expires, names the alarms it makes false, and carries the restore path. The distinction is not cosmetic: a silence tells a future reader nothing about whether re-enabling the thing is safe, and an expiring silence re-arms alarms on a schedule nobody remembers setting. If your alarms are already subject-scoped and your silences never expire, you have this and you do not need the package.

Incidents

Real failures, written up for people who will hit them too:

• Alarms that were true and useless — a retired bot, a watchdog that kept screaming about it, and why muting the alarm was the wrong repair.

License

MIT.