Detect a node that dies when nothing else is left to report it

[bburda]

Pull Request

Summary

A node dies and nothing reports it. The node cannot report it itself, and the other detectors
cannot see it: qos_mismatch needs both sides present to compare profiles, orphan looks at a
topic whose publisher took the topic with it, and lifecycle_expectation only watches nodes the
operator listed as required-active.

This adds node_death, which raises GRAPH_NODE_DISAPPEARED for a node that was alive and armed
and stopped being alive, together with the suppression framework the umbrella issue asks to be
opt-in and explicit.

Alive is not membership in the entity snapshot. In runtime discovery a dead node leaves the
snapshot, so the two look the same; in manifest and hybrid discovery the manifest keeps the App
and only clears its online flag, so a detector counting membership would make a manifest node
immortal. Tracking is keyed on the stable fully qualified name rather than the app id, because an
id is recomputed each sweep and gains a namespace prefix once a bare-name collision exists, so a
live node held under its old id would be reported as gone.

The grace period is wall clock, not ticks. The entity cache is rebuilt on a debounced graph
event, so every tick between two refreshes sees the same snapshot and one absent cache generation
is counted again on each of them. At a fast tick a two-tick tolerance is shorter than one refresh
cycle, so the window has a floor in milliseconds and says so when it raises the configured value.

Suppression is opt-in. Nothing is suppressed unless suppress names it. A configured
allowlist that suppress does not name has no effect and warns that it has none. Two
suppressors ship: an operator allowlist, and clean lifecycle shutdown. A suppressor also declares
whether its veto is durable, because only a durable veto may reclaim tracker bookkeeping - a veto
that can lift later would otherwise lose a real fault for good once the condition ends.

Also in this change: the absence boundary

GRAPH_NODE_INACTIVE says a required node is not active. GRAPH_NODE_DISAPPEARED says a node is
gone. Both can be true of one node, they have different repairs, and both standing at once is
correct - this change does not remove that.

What changed is narrower. Sustained absence used to mature a violation that had not yet been
reported, so a node observed non-active for a few ticks and then killed acquired an inactive fault
built entirely from evidence gathered after it could no longer be observed. That rule existed
because no presence detector existed to own departures. Now one does, so absence continues a
violation that has already matured and no longer creates one that has not.

With one exception, which is the part worth reviewing closely: the reliability gate refuses to arm
a managed node that never reads active, and node_death only tracks armed nodes. For such a
node the handover has no receiver, so absence may still mature its violation - otherwise a
required node that comes up unconfigured and dies is reported by nothing at all. The split is
keyed on whether the node was ever armed, latched rather than recomputed, because a managed node
legitimately reads not-armed again after an ordinary deactivate while node_death is still
tracking it.

Known limitation

A second node's death, while the fault from the first is still outstanding, is added to the
fault's description but arrives as an update to an already-confirmed record: no state transition,
no freeze frame and no recording of its own. Acknowledging the fault between the two deaths avoids
this, because a report arriving after an acknowledgement is counted as a new occurrence and gets
its own evidence.

Two ways of forcing it from the detector were measured and rejected rather than skipped. Sending a
clear and a raise back to back does nothing: a single opposite-direction report cannot move the
fault manager's hysteresis latch, so the raise lands on the already-confirmed branch. Calling the
clear service instead is worse than not doing it, because it deletes the per-topic readings
captured for that fault and by default clears every symptom the correlation engine attributes to
it. A correct fix belongs in the fault manager, as an operation that re-confirms a record without
clearing it. This is documented in the package README and design doc.

Also in this change: presence ownership rests on knowledge

Two detectors divide one job, and until now the division was decided by a fact the plugin did not
have. GRAPH_NODE_DISAPPEARED is raised for a node the reliability gate has armed; the gate arms a
managed node whose lifecycle state reads active, and it also arms one whose state has never been
measured, because it deliberately treats an unknown state as "do not gate" so that a node nobody
can read is not silenced across every other detector. Those two answers are not the same fact, and
the second was being latched permanently into the flag lifecycle_expectation consults before it
lets absence mature a violation. A required node that never configured could therefore be reported
by the presence detector and, by the same latch, not reported by the only detector that could still
have described what was wrong with it.

The division now asks a second, stricter question, and the gate itself is unchanged. Ownership is
earned when the node is not managed at all or its label reads active; it is withheld while the
state is merely unread and the watcher still has re-seed attempts left; and once those attempts are
spent it is provisional rather than earned - the node is owned, because otherwise nothing would
ever report its departure, but that grant yields the moment a real label finally arrives. A grant
made on knowledge latches, because a later deactivate does not hand the node back. A grant made on
ignorance does not, because the reason for it stops being true the instant somebody else can report
the node.

One consequence is worth stating rather than leaving to be found: a node that departs during its own
warmup is reported by nobody. That is the bringup-quiesce trade-off behaving as designed, it
predates this change, and it is written down where the boundary is described.

Also in this change: the sanitizer test budget

The sanitizer jobs walk the workspace one package at a time under a single 45-minute budget for
the whole step. This package's end-to-end suite runs 24 minutes of that under instrumentation, up
from 10:38 on main, so the step stopped fitting: the TSan run was cut inside the package that
follows this one, and ASan finished the same run with 20 seconds to spare. No single scenario is
responsible - the slowest is 106 s and the rest is a 30-50 s tail across 73 targets, because these
tests wait on grace windows, respawn delays and restart loops rather than compute.

quality.yml now tests this package in a job of its own, once per sanitizer, and ci.yml does the
same over humble and lyrical, where the same cap applies to the step rather than to the job: lyrical
was cut inside this package and humble finished with three minutes to spare. The sweeps skip it. The job builds only the chain up to the package, which still includes the
integration-test package the scenarios launch from, and restores the matching sweep's ccache
instead of saving a second copy of the same objects. Instrumentation, the timeout multiplier and
the sanitizer options are unchanged from the sweep it left, and the job is not path-filtered,
because the plugin drives the gateway, the fault manager and discovery, so the changes most likely
to break it are outside its own tree.

The scenarios' own wall-clock budgets now scale with MEDKIT_TEST_TIME_SCALE, which the sanitizer
jobs already set to the factor they apply to every declared CTest timeout. A budget asserted inside
a test is invisible to that rewrite, so an instrumented graph that is slow to forget a departed node
blows one and the failure reads as a detector that never reported. Only give-up bounds scale: poll
intervals, enforced respawn delays and the sustained-observation windows keep their values, and the
scale is 1.0 whenever the variable is unset, so the normal jobs are unchanged.

---

Issue

• closes A node dies and nothing is left to report it #624

---

Type

• Bug fix
• New feature or tests
• Breaking change
• Documentation only

---

Testing

The tests were written and merged before the detector, so every scenario was watched failing for
the right reason first, and the suite is what the implementation had to satisfy rather than a
description of it.

73 ctest targets, all green, of which 42 are end-to-end against a real gateway, a real fault
manager and a real ROS graph. 23 of those scenarios are new here.

The end-to-end set covers: a process that exits, its return and the clearing that follows, a
managed node that deactivates but keeps running, a manifest node that never came online, CLI-
convention node names, two nodes sharing a bare name in different namespaces, a fast tick against
a stale cache generation, a node restarting repeatedly across a window, a gateway restart with a
death outstanding, each suppressor including the inert-allowlist warning, pruning that must not
heal a fault that is still true, and each side of the absence boundary.

Every scenario that asserts an ABSENCE gates on the plugin being armed first and uses a helper
that fails when the fault surface is unreachable, so it cannot pass against a stack where the
plugin never loaded. Two such helpers are added here, each with a test that points it at a surface
which dies mid-window and fails if the helper still passes.

The whole suite was also run with the plugin path pointed at a file that does not exist: every
scenario fails, naming the load failure.

Reviewers can check the boundary with ctest -R "node_death_boundary_e2e", the suppressors with
ctest -R "node_death_suppression_e2e", and the tracker's bound with
ctest -R "test_node_liveness_tracker".

No route is added or changed. The only public surface difference is additive content inside the
existing x-medkit-watchdog payload, which gains a node_death block.

---

Checklist

• Breaking changes are clearly described (and announced in docs / changelog if needed)
• Tests were added or updated if needed
• Docs were updated if behavior or public API changed

[Copilot]

Pull request overview

Adds a new node_death detector to the ros2_medkit_graph_watchdog plugin to raise GRAPH_NODE_DISAPPEARED when a previously-armed node disappears and no other detector can observe/report the departure, plus an opt-in suppression framework (allowlist + clean lifecycle shutdown) and supporting tracking/retention logic.

Changes:

• Introduce node_death detector with a dedicated liveness tracker keyed by stable FQN, wall-clock-based grace flooring, and bounded bookkeeping under identity churn.
• Add opt-in suppression framework via a Suppressor interface, including allowlist suppression and lifecycle clean-shutdown suppression, plus pruning semantics tied to “durable” suppressors.
• Expand unit/integration/e2e test coverage and strengthen e2e harness window assertions (silence/persistence/description-scoping).

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/test_suppressor.cpp	Adds unit tests for the Suppressor contract and apply_suppressors() behavior.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/test_node_liveness_tracker.cpp	Adds extensive unit tests for NodeLivenessTracker state machine, pruning, caps, and description ordering.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/test_lifecycle_shutdown_suppressor.cpp	Adds tests for lifecycle clean-shutdown suppression behavior against a real ReliabilityGate.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/test_lifecycle_expectation_tracker.cpp	Updates lifecycle expectation tracker tests to reflect the new “absence boundary” and armed-vs-never-armed split.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/test_lifecycle_expectation_integration.cpp	Updates integration scenarios and expectations around absence handling and boundary behavior with the new presence detector.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/test_graph_watchdog_plugin.cpp	Adds regression coverage for node_death retention/config fallback behavior and oversized integer validation.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/test_allowlist_suppressor.cpp	Adds unit tests for allowlist suppression matching forms and durability.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/test/e2e/harness.py	Adds stronger e2e assertion helpers for sustained presence and scoped “never names” checks; extends flaky server harness.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/src/graph_watchdog_plugin.cpp	Adds shared node_death prune_grace resolution and aligns retention math with detector-side validation/flooring.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/src/detectors/node_death_detector.cpp	Introduces the new node_death detector implementation (tracking, suppression, pruning, reporting/clearing guard).
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/src/detectors/lifecycle_expectation_detector.cpp	Wires reliability-gate arming into lifecycle expectation matches to support the absence-boundary split.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/include/ros2_medkit_graph_watchdog/suppressor.hpp	Adds the suppression interface and apply_suppressors() helper.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/include/ros2_medkit_graph_watchdog/node_liveness_tracker.hpp	Adds NodeLivenessTracker for armed-node disappearance tracking with cap/collapse logic and freshness ordering.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/include/ros2_medkit_graph_watchdog/lifecycle_shutdown_suppressor.hpp	Adds durable suppressor for clean lifecycle shutdown departures.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/include/ros2_medkit_graph_watchdog/lifecycle_expectation_tracker.hpp	Extends tracker model and absence behavior to incorporate “ever armed” logic and prevent absence-maturing where node_death should own departures.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/include/ros2_medkit_graph_watchdog/graph_watchdog_plugin.hpp	Exposes a test seam for departed-retention tick computation.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/include/ros2_medkit_graph_watchdog/detector.hpp	Changes DetectorContext::{raise_fault,clear_fault} to return bool to signal whether an async request was actually enqueued.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/include/ros2_medkit_graph_watchdog/detector_config_keys.hpp	Adds node_death grace ceiling + wall-clock miss_grace floor computation helpers.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/include/ros2_medkit_graph_watchdog/allowlist_suppressor.hpp	Adds allowlist suppressor implementation and matching semantics documentation.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/include/ros2_medkit_graph_watchdog/aggregated_fault.hpp	Updates aggregated fault emission helpers to return the underlying send/enqueue boolean.
src/ros2_medkit_plugins/ros2_medkit_graph_watchdog/CMakeLists.txt	Registers new unit/integration/e2e tests and adds node_death sources to relevant test targets.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[mfaferek93]

armed = app.is_online && presence_ownership(...) == kEarned mirrors the gate but not node_death's admission rule, and the two now disagree on one sweep. A dying managed node loses its get_state service one sweep before its App goes (the shape c556377 guards against in node_death), the watcher drops the record, and the gate answers kEarned for a node measured unconfigured a tick earlier. That latches ever_armed (tracker.hpp:556), so absence HOLDS the below-grace streak (tracker.hpp:659) instead of maturing it - while node_death never admitted the key (disowned on every measured tick, and the released-until-active rule at node_death_detector.cpp:318 refuses the kEarned sweep).

Net effect: a require_active node that sat unconfigured and then died raises nothing, forever, and the INACTIVE clear for every other node stays withheld. On main it matured within grace + absence_grace + 1 ticks.

I reproduced it with both detectors on one gate: require_active: ["/victim"], grace 30; 10 present ticks labelled unconfigured; one present tick with no lifecycle record; 60 absent ticks -> 0 GRAPH_NODE_INACTIVE, 0 GRAPH_NODE_DISAPPEARED. The same sequence without the drop sweep gives 37 INACTIVE FAILED. Cleanest fix is probably not to re-derive ownership here at all and ask node_death what it tracks (NodeLivenessTracker::known_keys() is already a seam); short of that, armed needs the same released-until-active rule, so a kEarned resting on "no record" never latches for a node this tracker already measured non-active.

[mfaferek93]

The hand-back here only checks earned_, and earned_ is pruned to the live graph every tick (390-392), so a node this detector has already REPORTED dead loses its earned ground during the outage. When it respawns, a lifecycle node's first GetState reads unconfigured (nothing re-drives it without a lifecycle manager) - that is kDisowned with earned_ empty, so the key is released, the dead set empties and PASSED heals the fault. If it crashes again, node_death is not tracking it, and lifecycle_expectation holds the streak because ever_armed survives across incarnations (tracker.hpp:556), so the !ever_armed advance at :659 never runs. One report, a heal, then silence for the rest of the crash loop - the case the README says this detector most exists for. README:724 justifies the hold with "node_death tracks any node the gate has admitted at least once", which does not hold across a departure.

Reproduced twice: plain node_death (tracked 2 -> 1 on the respawn, 1 PASSED, FAILED count 1 before and after the second death) and with lifecycle_expectation on the same gate and require_active: ["/victim"] (PASSED x2 on respawn, DISAPPEARED 1/1, INACTIVE 0 after 60 absent ticks). Suggest not handing back a key whose tracker entry is currently carrying a death (expose NodeLivenessTracker::is_departed(key) and keep it on kDisowned while set), and re-deriving ever_armed for a returning incarnation the way node_death re-derives earned_.

[mfaferek93]

This row never reaches the suppressor. CLEAN_NODE is launched with auto_activate=False (:345), polled as unconfigured (:605), driven UNCONFIGURED_SHUTDOWN to finalized and only then killed - it is never active, so presence_ownership() answers kDisowned on every measured tick and node_death never admits the key. It never enters report.dead, so LifecycleShutdownSuppressor is never consulted, and forbidden=[CLEAN_NODE] stays true with suppress: ["lifecycle"] removed or the suppressor deleted. That contradicts the docstring at :55-57 / :93-95 and the README test-tier claim; the only real-stack proof of the mechanism is vacuous (the integration tests through the seam are fine). To exercise it the node has to be measured active first, then go through deactivate/cleanup/shutdown with the watcher seeing the transitions, then be killed. Verified by reading reliability_gate.cpp:244-246 and node_death_detector.cpp:333-355 against the row.

[mfaferek93]

_poll_stable_tracked_count crashes when the first sample is None: last_count and streak_start start as None, watchdog_detector_status() returns None on any RequestException, slow first GET or non-200, so on that poll count != last_count is False and the elif evaluates now - streak_start with streak_start still None - TypeError. The docstring promises "None on every poll counts as stable at None"; the code does the opposite. On a loaded TSan runner one slow first GET turns ros2cli_ignored (and the suppression file's prune row, same helper at test_node_death_suppression_e2e.test.py:458-466) into an ERROR unrelated to the detector. Seeding streak_start = time.monotonic() before the loop fixes both copies.

[mfaferek93]

These two are the only give-up budgets in the file not multiplied by TIME_SCALE (the module says at :199-204 that every one is), while test_lifecycle_expectation_e2e.test.py:1111,1120 scales the identical restart drive. Under the x3 sanitizer rewrite the re-arm after a full gateway respawn gets a third of the budget the first arming got. Related race in the boundary file: B2_GRACE = 15 and B6_GRACE = 15 (test_node_death_boundary_e2e.test.py:207,210) give a 16-tick = 3.2 s nominal window from DEACTIVATE to maturity, and the row spends the label poll, the tracked-nodes poll, _fault_present_now and the ~1.1 s post-kill snapshot tail inside it. Lose it in B2 and assert_fault_absent_throughout(INACTIVE) goes red blaming the detector; lose it in B6 and INACTIVE raises via the ordinary present path, so the row is green without the !ever_armed absence branch ever running. Scale the two timeouts and raise the two graces (40-50, or * TIME_SCALE). Reading only - I did not run the e2e locally.

[mfaferek93]

Related to the hold this block explains: the once-per-episode withheld-clear warning at :569 (outside this diff) still says "that hold releases as soon as the node reads active or its streak passes grace". For an ever-armed node that has DEPARTED neither can happen until it returns (README:724 and tracker.hpp:664-666 say the hold is unbounded), so the only diagnostic an operator gets tells them to wait for something that is not coming. In my run the same line also promised a GRAPH_NODE_NOT_MANAGED report "after 60 ticks", which the absence path zeroes (tracker.hpp:636), so it never fires either. The tracker already knows is_departed; saying "until the node returns" for that case would be honest.

[mfaferek93]

This doc contradicts the detector. Lines 142-144 say the flag "answers true again once the watcher has stopped asking"; the detector deliberately passes == kEarned only and explains at lifecycle_expectation_detector.cpp:339-345 why kProvisional must not latch. Lines 147-151 say defaulting to true "keeps the already-shipped absence handling" - the shipped handling on main was advance-on-absence, and true selects the new HOLD path, so an unwired caller silently loses a reporting path rather than keeping one. Fix the text, and consider defaulting to false (the direction that never drops a report).

[mfaferek93]

The per-detector keys now read int64 before the range check - good - but the plugin-scope keys at :154-171 (tick_interval_ms, warmup_cycles, prune_grace) still get<int>() before validating, the exact narrow-then-validate wrap this PR fixes below. prune_grace: 4294967296 becomes 0, resolve_node_death_prune_grace() faithfully injects 0, both sides agree, nothing warns. Consistency is preserved, the operator's value is not. Same int64 read as here would close it.

[mfaferek93]

This row (and the one at :734) polls only for an empty label and then kills, relying on the comment's tick-ordering argument that the GetState budget has been spent; the provisional row in the same file reads measurement_pending=False off the route first. If the gateway ever delivers the App one refresh before its service endpoints, the kill lands on kUnclaimed, the key is never admitted, and the red reads as a detector regression. Passing measurement_pending=False to _poll_watchdog_entity at both sites makes the precondition explicit.

[mfaferek93]

MISS_GRACE = 20 is explained here as chosen to sit past the wall-clock floor, but it is injected only in prune_no_false_heal (:318); the other four scenarios run on the default 2, floored to 14 with a startup WARN, so the timing arithmetic in their docstrings is computed from a grace the gateway does not use. Moving it into the base detector_params like test_node_death_e2e.test.py:314 does keeps the docstrings honest.