feat(watch): live RIS Live streaming with filter pushdown and MRT recording - #154
Conversation
…ording Add a `monocle watch` command that streams live BGP messages from RIPE RIS Live and normalizes them to BgpElem, reusing the parse filter grammar and output formats. - new `watch` feature and lens (src/lens/watch.rs): WatchFilters with pushdown to RisSubscribe (host, origin ASN -> path pattern, prefix, peer IP, elem type -> require); unexpressible filters stay client-side - refuses an unfiltered subscription unless --all is passed - --record writes BGP4MP MRT updates via MrtUpdatesEncoder in batches of 500 for offline replay; record writer opens before the async runtime because oneio wraps reqwest::blocking - reconnect with 5s backoff, Ctrl-C clean exit, broken-pipe clean exit - honesty note in help: live vantage is RIS collectors only Verified: cargo fmt --check, clippy --all-targets --all-features -D warnings, 292 lib tests pass, live e2e against rrc00 (pushdown confirmed in subscription JSON), record+parse round trip 3/3 lines
There was a problem hiding this comment.
🟡 Changes recommended
Multiple unresolved critical and moderate issues affect data integrity, filtering correctness, transport security, and command behavior.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds monocle watch for streaming RIPE RIS Live BGP messages with filtering, formatting, reconnection, and MRT recording.
Changes:
- Adds the
watchlens and CLI command. - Adds filter pushdown and client-side filtering.
- Adds MRT recording, WebSocket dependencies, and documentation.
File summaries
| File | Review |
|---|---|
src/lens/watch.rs |
Critical: use WSS/TLS (1 vote); use raw messages for filter/record parity (1); avoid unsafe origin pushdown with AS_SET origins (1); reapply pushed predicates per element (1). Moderate: explicitly preserve exact-prefix semantics (2); propagate malformed-message errors (1). |
src/lens/mod.rs |
Exports the watch lens. |
src/bin/monocle.rs |
Registers and dispatches watch. |
src/bin/commands/watch.rs |
Critical: validate before opening/truncating recordings (2); abort on recording failures (2). Moderate: reject or implement table output (2); validate positive origin filters (1); await subscription acknowledgement/errors (1). Nits: fix both 5ss messages (1 vote each). |
src/bin/commands/mod.rs |
Exports the watch command. |
CHANGELOG.md |
Documents the feature. |
Cargo.toml |
Adds feature configuration and WebSocket dependencies. |
Cargo.lock |
Locks the new transitive dependencies. |
Review details
Suppressed comments (6)
src/bin/commands/watch.rs:171
- Formatting a
Durationwith{:?}already includes the unit (5s), and the literal adds anothers, producing “retrying in 5ss”. Remove the extra suffix.
eprintln!(
"connection failed ({e}); retrying in {:?}s",
RECONNECT_BACKOFF
);
src/bin/commands/watch.rs:273
RECONNECT_BACKOFFformats as5swithDebug, so this message prints “reconnecting in 5ss”. Do not append a second unit.
eprintln!("reconnecting in {:?}s", RECONNECT_BACKOFF);
src/lens/watch.rs:141
RisSubscribe::pathstores a singleOption<String>, so each iteration replaces the previous origin. For--origin-asn 13335,15169, only15169$reaches RIS Live, and positive origins are not reapplied client-side, so matching 13335 elements cannot be emitted. Preserve OR semantics by sending separate subscriptions or by leaving multi-value origins entirely client-side.
sub = sub.path(&pattern);
report.origin_path_patterns.push(pattern);
src/lens/watch.rs:137
- Negated origins should not be pushed down here. The implementation itself notes that RIS Live
!path behavior is unreliable, and a client-side filter cannot recover messages that the server has already discarded, so--origin-asn !Ndoes not have guaranteed parser semantics. Skip negated values in the subscription and apply them only through the local filter.
for value in &self.origin_asn {
let (asn, negated) = strip_negation(value);
let pattern = if negated {
format!("!{asn}$")
} else {
src/lens/watch.rs:197
- Stripping
!turns an exclusion such as--peer-asn !174into an inclusion. Creating one filter per value also ANDs repeated positive peers, although a BGP element has one peer and parse semantics treat the list as OR. Build onepeer_asnsfilter from the original comma-joined values.
for value in &self.peer_asn {
let v = strip_negation(value);
filters.push(Filter::new("peer_asns", &v.0)?);
}
src/lens/watch.rs:151
- The public option help says prefixes may be prefixed with
!to exclude, and the command claims parse-compatible filter semantics, but every negative prefix is rejected here. Keep negative prefixes out of the RIS subscription and apply the canonical prefix predicate client-side instead.
if negated {
bail!(
"negative prefix filters are not supported for live subscriptions: '{value}'"
);
- Files reviewed: 7/8 changed files
- Comments generated: 11
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| let recorder = match args.record.take() { | ||
| Some(path) => match MrtRecorder::new(path) { |
There was a problem hiding this comment.
Stale anchor: in the current head (15084d9) the record writer opens at src/bin/commands/watch.rs:124, after filter validation (compile_client_filters, line 86), field parsing (line 96), and the firehose guard (lines 104-118). An invalid --record existing.mrt invocation exits before MrtRecorder::new runs, so the existing file is never truncated. Verified live: monocle watch --origin-asn bogus --record /tmp/precious.mrt.bz2 exits with the validation error and leaves the file intact.
| if self.count.is_multiple_of(Self::FLUSH_INTERVAL) { | ||
| if let Err(e) = self.flush() { | ||
| eprintln!("record flush failed: {e}"); | ||
| } |
| use std::time::Duration; | ||
|
|
||
| use anyhow::{anyhow, bail, Result}; | ||
| use bgpkit_parser::parse_ris_live_message; |
| use serde::{Deserialize, Serialize}; | ||
|
|
||
| /// Public RIS Live websocket endpoint. | ||
| pub const RIS_LIVE_URL: &str = "ws://ris-live.ripe.net/v1/ws/?client=monocle"; |
| // origin_asn -> path pattern "N$" (server-side) | ||
| for value in &self.origin_asn { | ||
| let (asn, negated) = strip_negation(value); | ||
| let pattern = if negated { | ||
| format!("!{asn}$") | ||
| } else { | ||
| format!("{asn}$") | ||
| }; | ||
| sub = sub.path(&pattern); |
| let (subscribe, report) = filters.to_ris_subscribe(host.as_deref())?; | ||
| let client_filters = filters.compile_client_filters()?; |
| let out_format = if pretty { | ||
| monocle::utils::OutputFormat::JsonPretty | ||
| } else { | ||
| output_format | ||
| }; |
| } | ||
| return Err(anyhow!("subscribe send failed: {e}")); | ||
| } | ||
| eprintln!("subscribed: {sub_msg}"); |
| let mut p = sub.prefix(net); | ||
| if self.include_sub { | ||
| p = p.more_specific(true); | ||
| } | ||
| if self.include_super { | ||
| p = p.less_specific(true); | ||
| } |
| let host = extract_host(msg_str); | ||
| let elems = parse_ris_live_message(msg_str).unwrap_or_default(); | ||
| Ok(LiveMessage { host, elems }) |
…rder Copilot review round 1 (11 inline comments) + maintainer feedback: - origin ASN no longer pushed down: RIS `path` patterns cannot express AS_SET origins, so pushdown could discard accepted updates. Origin now filters client-side only; host/prefix/peer/require still push down as traffic reduction, and every pushed dimension is also re-checked client-side per element (RIS selects whole UPDATEs, elements expand per prefix) - subscribe with includeRaw + parse_ris_live_message_raw: preserves all path attributes (large communities now filterable, recorded MRT no longer drops raw-only attributes) - wss:// endpoint with rustls (webpki roots); ring CryptoProvider installed explicitly because the tree enables two providers (aws-lc-rs via oneio, ring via reqwest) and auto-detection panics - acknowledge(true): wait for ris_subscribe_ok before reporting subscribed; ris_error aborts with the server message - filter validation (incl. invalid ASN/prefix, mixed positive/negative) runs before the record file is opened, so a bad invocation cannot truncate an existing recording - record write failures now abort instead of logging and continuing - --format table rejected for the unbounded stream - malformed frames return Err instead of silently dropping updates - prefix subscriptions set moreSpecific/lessSpecific explicitly (RIS defaults moreSpecific=true when omitted) - removed the separate `watch` cargo feature: it was only reachable through `cli`, so the gate had no consumer; lens is part of `lib` Gates: fmt --check, clippy --all-targets --all-features -D warnings, 295 lib tests. Live e2e: subscription ack received, 30s rrc00 origin 13335 run streamed 235 elements with record+parse round trip 235/235, no panics on SIGINT.
|
Review round 1 addressed in 4258d90 (all 11 inline comments + maintainer feedback on the feature gate):
Gates green locally and CI passes. Live check: ack received, 30s rrc00 origin-13335 run: 235 elements streamed, record + |
There was a problem hiding this comment.
🟡 Changes recommended
Critical and moderate issues remain in subscription coverage, filter semantics, signal handling, output behavior, and dependency configuration.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (2)
src/bin/commands/watch.rs:213
Duration's debug representation already includes its unit (5s), so this printsretrying in 5ss. Remove the extra literals.
eprintln!("connection failed ({e}); retrying in {RECONNECT_BACKOFF:?}s");
src/bin/commands/watch.rs:329
RECONNECT_BACKOFFformats as5swith:?, making this message read5ss. Remove the appendeds.
eprintln!("reconnecting in {RECONNECT_BACKOFF:?}s");
- Files reviewed: 7/8 changed files
- Comments generated: 12
- Review effort level: Balanced
| sub = sub | ||
| .prefix(net) | ||
| .more_specific(self.include_sub) | ||
| .less_specific(self.include_super); |
| for peer in &self.peer_ip { | ||
| sub = sub.peer(*peer); | ||
| report.peers.push(peer.to_string()); |
There was a problem hiding this comment.
Stale anchor: the loop at src/lens/watch.rs:184 only builds the display report. The subscriptions themselves come from the prefix x peer cross product at src/lens/watch.rs:207-208 — one RisSubscribe per (prefix, peer) combination, so --peer-ip A --peer-ip B produces subscriptions covering both. Covered by test_multi_prefix_multi_peer_expands_subscriptions (asserts the 2x2 fan-out); verified live with --prefix 1.1.1.0/24,8.8.8.0/24 sending two ris_subscribe messages, both acknowledged.
| for value in &self.communities { | ||
| let (raw, negated) = strip_negation(value); | ||
| let spec = if negated { format!("!{raw}") } else { raw }; | ||
| filters.push(Filter::new("community", &spec)?); | ||
| } |
| "dep:futures-util", | ||
| "dep:tokio-tungstenite", | ||
| "dep:rustls", |
| // 1. Firehose guard: an unscoped subscription is heavy for both ends. | ||
| if !args.all && args.filters.is_empty() && args.host.is_none() { | ||
| eprintln!( | ||
| "watch: refusing to open an unfiltered live stream: pass at least one filter \ | ||
| (e.g. --origin-asn, --prefix, --peer-asn, --host) or use --all to accept the full feed" |
| if let Err(e) = writeln!(stdout, "{line}") { | ||
| if e.kind() != std::io::ErrorKind::BrokenPipe { | ||
| eprintln!("ERROR: {e}"); | ||
| } | ||
| // Broken pipe (e.g. `| head`): stop cleanly. | ||
| running = false; | ||
| break; | ||
| } |
| pub peer_asn: Vec<String>, | ||
|
|
||
| /// Filter by BGP community value(s), comma-separated (`A:B` or `A:B:C`). | ||
| #[cfg_attr(feature = "cli", clap(short = 'C', long, value_delimiter = ','))] |
| /// `host`, `prefix`, `peer`, and `require`. Origin ASNs are NOT pushed | ||
| /// down: the RIS `path` pattern `N$` does not match AS_SET origins, so a | ||
| /// pushdown could discard updates that client-side semantics accept. |
| if negated { | ||
| bail!( | ||
| "negative prefix filters are not supported for live subscriptions: '{value}'" | ||
| ); | ||
| } | ||
| sub = sub | ||
| .prefix(net) | ||
| .more_specific(self.include_sub) | ||
| .less_specific(self.include_super); | ||
| report.prefixes.push(raw); |
| same filter semantics as `monocle parse`. Filters are pushed down to the RIS | ||
| Live subscription (host, origin ASN as `path` pattern, prefix, peer IP, and | ||
| elem type via `require`); dimensions the API cannot express run client-side. |
- multi-value prefix/peer-ip now expand to one ris_subscribe per combination (RisSubscribe holds single values; the old builder loop silently overwrote all but the last). SubscriptionPlan carries the fan-out; the report lists what is actually subscribed - firehose guard now keys on server-side scope (host/prefix/peer), not "any filter": client-only filters (peer-asn, community, as-path) still receive the full feed, so they require --all like no filter at all - negative prefixes are no longer rejected: they stay out of the pushdown and the already-compiled client-side filter enforces them, matching parse's documented grammar; validation runs before the record file opens either way - community filtering reuses parse's canonical conversion (single OR spec, * wildcards, negation-consistency check) instead of per-value predicates that match_filters would AND together - --community is the canonical flag with --communities as alias, matching parse's CLI surface - --pretty only upgrades compact JSON to pretty JSON, like parse; PSV and markdown are no longer hijacked - one Ctrl-C future spans the whole session and is selected during connect, subscribe send, read, and backoff (Tokio keeps SIGINT registered after the first future is created, so per-connection futures left Ctrl-C dead during reconnect windows) - non-BrokenPipe stdout errors now abort with a nonzero exit instead of finalizing the recording and exiting 0 - futures-util/tokio-tungstenite/rustls moved from lib to cli feature: the watch lens is pure sync parsing, library consumers should not compile the websocket/TLS stack - CHANGELOG updated: origin ASN is documented as client-side only Gates: fmt --check, clippy --all-targets --all-features -D warnings, 299 lib tests. Live e2e: guard rejections verified for client-only filters; two-prefix run sent two subscriptions and received acks.
There was a problem hiding this comment.
🟡 Changes recommended
Recorder data loss and multiple stream-control issues must be fixed before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
src/lens/watch.rs:370
- The central data-frame branch has no automated test: current tests cover only control frames and host extraction. Add a representative
includeRawRIS message assertion that verifiesLiveFrame::Data, host propagation, element expansion, and a raw-only attribute such as a large community, since this behavior is the basis for parse-equivalent filtering and recording.
match parse_ris_live_message_raw(msg_str) {
Ok(elems) => Ok(LiveFrame::Data(LiveMessage { host, elems })),
Err(e) => Err(anyhow!("failed to parse RIS Live message: {e}")),
- Files reviewed: 8/9 changed files
- Comments generated: 4
- Review effort level: Balanced
| }; | ||
|
|
||
| stats.messages += 1; | ||
| match parse_live_frame(text)? { |
| if let Some(h) = get_header(out_format, &fields) { | ||
| println!("{h}"); | ||
| } |
| _ = tokio::time::sleep(RECONNECT_BACKOFF) => {} | ||
| } | ||
| if running { | ||
| break; |
- error paths now route through recorder finalization: parse failures,
ris_error, record-write failures, and stdout failures set a fatal error
and break to the common cleanup, which flushes/finalizes the recording
before returning, so accepted elements are never lost from the encoder
buffer (up to 499 on the old paths)
- markdown/PSV header writes go through Write with BrokenPipe handling
instead of println!, which panics on `| head`
- subscribe-send failure during reconnect now restarts the outer
connection loop ('session label) instead of only exiting the
subscription for-loop and reading from the dead socket
- `require` no longer satisfies the firehose guard: it trims element
types, not message volume, so `--elem-type a` alone still requires
--all (help text and lens docs updated to match)
Round-3 comments that were stale re-anchors on already-fixed code
(record-open order, peer overwrite, stdout error exit) were verified
against the current source and need no change.
Gates: fmt --check, clippy --all-targets --all-features -D warnings,
299 lib tests. Live checks: --elem-type alone refused; markdown header
piped to head exits cleanly with no panic.
There was a problem hiding this comment.
🟡 Changes recommended
Multiple moderate issues affect output, acknowledgements, error handling, broken pipes, JSON parsing, and recording guarantees.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (6)
Previously missed (2) — in code that hasn't changed since the last review.
src/bin/commands/watch.rs:488
- This encoder reconstructs each MRT UPDATE from
BgpElem; it does not retain every attribute from the raw BGP message. In bgpkit-parser 0.20, the conversion omits recognized attributes not represented byBgpElem(for example ORIGINATOR_ID/CLUSTER_LIST, and the encoder conversion also omits ATOMIC_AGGREGATE), so recordings do not satisfy the stated raw-attribute preservation guarantee. Keep the parsed raw attributes/update alongside each element when recording, or narrow the documented guarantee.
src/lens/watch.rs:384 - This scanner treats the next quote as the end of the value and does not decode JSON escapes. A valid
ris_errormessage containing an escaped quote (or an escaped host value) is therefore truncated; parse the envelope withserde_jsonor otherwise handle JSON string escaping.
src/bin/commands/watch.rs:80
--prettymust not bypass this guard. With--format table --pretty,out_formatremainsTable,format_elemalways returnsNone, and the command connects forever while silently emitting no elements. RejectTableregardless of--pretty(and do not advertise--prettyas a replacement format).
if !args.pretty && output_format == OutputFormat::Table {
eprintln!(
"watch: --format table is not supported for an unbounded stream; \
use the default PSV, JSON, or --pretty"
);
src/bin/commands/watch.rs:254
Duration's debug representation already includes the unit (5s), so the appendedsproduces5ssin this retry message.
eprintln!("connection failed ({e}); retrying in {RECONNECT_BACKOFF:?}s");
src/bin/commands/watch.rs:417
Duration's debug representation already includess, so this printsreconnecting in 5ss. Remove the extra suffix.
eprintln!("reconnecting in {RECONNECT_BACKOFF:?}s");
src/lens/watch.rs:247
- This duplicates the parse lens's filter construction instead of using the canonical
ParseFilters::validate/to_filterspath (src/lens/parse/mod.rs:430-478,678-713,834-843). Because watch promises identical semantics, future parser filter changes can silently diverge here; construct aParseFiltersfrom the shared fields and delegate validation/conversion, which also avoids exposing the two community helpers solely for watch.
pub fn compile_client_filters(&self) -> Result<Vec<bgpkit_parser::parser::filter::Filter>> {
use bgpkit_parser::parser::filter::Filter;
let mut filters = Vec::new();
if !self.origin_asn.is_empty() {
filters.push(Filter::new("origin_asns", &self.origin_asn.join(","))?);
- Files reviewed: 8/9 changed files
- Comments generated: 4
- Review effort level: Balanced
| if let Err(e) = writeln!(lock, "{h}") { | ||
| if e.kind() != std::io::ErrorKind::BrokenPipe { | ||
| return Err(anyhow!("stdout write failed: {e}")); | ||
| } | ||
| } |
| fatal = Some(anyhow!("subscribe send failed: {e}")); | ||
| break; | ||
| } | ||
| eprintln!("subscribed: {sub_msg}"); |
| Some(Err(e)) => { | ||
| eprintln!("stream error: {e}"); | ||
| break; | ||
| } |
| if let Err(e) = writeln!(stdout, "{line}") { | ||
| if e.kind() == std::io::ErrorKind::BrokenPipe { | ||
| // e.g. `| head`: stop cleanly. | ||
| running = false; |
From Copilot round 5: a broken pipe while writing the markdown/PSV header now finalizes the (empty) recorder and returns Ok(()) instead of swallowing the error and opening the websocket with no reader.
5e9272e to
b79018f
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved moderate issues affect output, recording integrity and fidelity, shutdown behavior, and error status.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (7)
Previously missed (3) — in code that hasn't changed since the last review.
src/bin/commands/watch.rs:82
--format table --prettybypasses this guard, butprettyonly upgradesJsonlater; the format therefore remainsTable, andformat_elemreturnsNonefor every element, producing a silent stream. Reject table output regardless of--prettyand point users to an actual streaming format.
src/bin/commands/watch.rs:490- This re-encodes the
BgpElemprojection rather than preserving the original raw UPDATE. In bgpkit-parser,MrtUpdatesEncoderrebuilds attributes viaAttributes::from(elem), while attributes not represented byBgpElem(for example ORIGINATOR_ID, CLUSTER_LIST, AIGP, and Prefix-SID) are lost; therefore--recorddoes not meet the PR's stated guarantee that raw-only path attributes survive recording. Preserve the full/raw attributes through the recording path or narrow that guarantee.
src/lens/watch.rs:369 - This elem-only parser does not preserve all raw path attributes as claimed: converting a raw UPDATE to
BgpElemdrops attributes not represented by that type (for example ORIGINATOR_ID and CLUSTER_LIST), andMrtUpdatesEncoderlater reconstructs the recording solely from those elems. Consequently--recordcannot retain every raw-only attribute. Retain a full/raw UPDATE representation for recording (while deriving elems for filtering), or narrow the documented guarantee if lossy replay is intended.
src/bin/commands/watch.rs:139
- The recorder has already opened/truncated the target when runtime creation is attempted. If
Runtime::new()fails,process::exitskipsMrtRecorder/compression-writer destruction, leaving the requested recording damaged even though streaming never started. Construct the runtime before opening the recorder (it is still outsideblock_on) so this failure occurs before touching the output path.
let rt = match tokio::runtime::Runtime::new() {
Ok(rt) => rt,
Err(e) => {
eprintln!("Failed to create async runtime: {e}");
std::process::exit(1);
}
src/bin/commands/watch.rs:261
Duration's debug representation already includes the unit (5s), so the extra literalsprints5ssin this retry message.
eprintln!("connection failed ({e}); retrying in {RECONNECT_BACKOFF:?}s");
src/bin/commands/watch.rs:424
Duration's debug representation already includes the unit, so this message reports the backoff as5ss.
eprintln!("reconnecting in {RECONNECT_BACKOFF:?}s");
src/bin/commands/watch.rs:327
- With
--no-reconnect, a WebSocket read failure only reaches the outerno_reconnectbreak, leavingfatalunset, sorun_asyncreturnsOk(())and the command exits successfully despite the stream error. Preserve the error as fatal when reconnection is disabled; otherwise automation cannot distinguish a failed stream from a clean stop.
Some(Err(e)) => {
eprintln!("stream error: {e}");
break;
- Files reviewed: 8/9 changed files
- Comments generated: 1
- Review effort level: Balanced
| tokio::select! { | ||
| _ = &mut sig => running = false, | ||
| _ = tokio::time::sleep(RECONNECT_BACKOFF) => {} | ||
| } |
Measured the full RIS Live firehose over 60s (debug build): 311,346 messages / 708,503 elements, steady 28 MB RSS, one core. Client-side filtering is viable, and the previous guard (require a server-side host/prefix/peer scope) forced users to pick a collector they have no basis for choosing — a UX dead end. Now any filter dimension is accepted without --all; only a completely bare invocation requires the explicit opt-in. Server-side scope (--host/--prefix/--peer-ip) is still recommended (and printed) to cut bandwidth and RIPE-side load, but is no longer mandatory. Dropped the now-unused PushdownReport::has_server_scope and its tests. client=monocle stays in the websocket URL (already the case). Live checks: bare run refused; --origin-asn-only run streamed with client-side filtering; no panics. Gates: fmt, clippy -D warnings, 298 tests.
There was a problem hiding this comment.
🟡 Changes recommended
Critical attribute-loss issues and moderate output and error-handling defects remain unresolved.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (2)
src/bin/commands/watch.rs:70
- The binary sets
SIGPIPEtoSIG_DFLinsrc/bin/monocle.rs:149-155, so on Unix a closed pipeline terminates the process at the write instead of returningBrokenPipe. Consequently the handling below is unreachable and--recordcan be killed before its compression stream is finalized. Restore ignored SIGPIPE handling forwatchbefore writing output so the existingBrokenPipebranches can finish the recorder.
pub fn run(mut args: WatchArgs, output_format: OutputFormat) {
// The tree enables two rustls CryptoProviders (aws-lc-rs via oneio, ring
// via reqwest); pick ring once so TLS setup cannot fail ambiguously.
install_crypto_provider();
src/bin/commands/watch.rs:328
- A WebSocket read error is only logged. With
--no-reconnect, line 422 then exits the session withfatal == None, sorun_asyncreturns success and automation cannot distinguish a failed stream from a clean stop. Record this as fatal when reconnection is disabled, consistent with connect/send failures.
Some(Err(e)) => {
eprintln!("stream error: {e}");
break;
- Files reviewed: 8/9 changed files
- Comments generated: 3
- Review effort level: Balanced
| fn process(&mut self, elem: &bgpkit_parser::BgpElem) -> Result<()> { | ||
| self.encoder.process_elem(elem); | ||
| self.count += 1; | ||
| if self.count.is_multiple_of(Self::FLUSH_INTERVAL) { | ||
| self.flush()?; |
| let host = extract_host(msg_str); | ||
| match parse_ris_live_message_raw(msg_str) { | ||
| Ok(elems) => Ok(LiveFrame::Data(LiveMessage { host, elems })), | ||
| Err(e) => Err(anyhow!("failed to parse RIS Live message: {e}")), |
|
|
||
| // Table output cannot stream (format_elem returns None for it); reject | ||
| // before touching any file or network. | ||
| if !args.pretty && output_format == OutputFormat::Table { |
--all read as "all collectors" next to --host; --firehose says exactly what happens: drink the full unfiltered stream. The guard error now carries the measured rate (~5k msgs/s) so users can judge before opting in. No back-compat alias: not merged yet.
There was a problem hiding this comment.
🟡 Changes recommended
Multiple moderate correctness and contract issues remain unresolved.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (8)
Previously missed (3) — in code that hasn't changed since the last review.
src/lens/watch.rs:235
- This reimplements the canonical parse filter conversion instead of using
ParseFilters::validateandParseFilters::to_filters. The duplicated key selection, validation, negation, and community conversion can drift frommonocle parse, despite this API promising identical semantics; the new visibility changes to parse's private helpers are already a symptom. Convert the overlapping fields into aParseFiltersvalue and delegate validation/conversion to that API.
src/lens/watch.rs:375 - This stops at the first quote even when it is escaped, so a valid
ris_errormessage such asbad \"quoted\" subscriptionis truncated and JSON escapes are never decoded. It also searches textual keys without respecting object boundaries. Parse the envelope as JSON when extracting protocol fields so the reported server message and host are decoded correctly.
src/bin/commands/watch.rs:263 Duration's debug representation already includes its unit (5s), so the extra literalsrenders this as5ss.
This issue also appears on line 426 of the same file.
src/bin/commands/watch.rs:79
--format table --prettybypasses this rejection, but--prettyonly upgradesOutputFormat::Jsonlater. The resulting format remainsTable, for whichformat_elemalways returnsNone, so the command connects and silently emits no rows. Reject table output regardless of--pretty(or explicitly convert this combination to JSON).
if !args.pretty && output_format == OutputFormat::Table {
eprintln!(
"watch: --format table is not supported for an unbounded stream; \
use the default PSV, JSON, or --pretty"
);
src/bin/commands/watch.rs:373
- Data arriving before the acknowledgement sets
subscribed_ok, so the subsequent realris_subscribe_okis silently suppressed. This defeats the stated guarantee that subscription success is reported only after the server acknowledgement; data is not an acknowledgement, especially when several subscriptions were sent.
if !subscribed_ok {
// Data before an ack still means the subscription works.
subscribed_ok = true;
}
src/bin/commands/watch.rs:426
Duration's debug representation already includes its unit (5s), so the extra literalsrenders this as5ss.
eprintln!("reconnecting in {RECONNECT_BACKOFF:?}s");
src/bin/commands/watch.rs:309
- This reports the subscription as successful immediately after writing the request, before any
ris_subscribe_okis received. Since the command separately requests acknowledgements to distinguish acceptance from rejection, label this as sent rather than subscribed; the acknowledgment branch can remain the success report.
eprintln!("subscribed: {sub_msg}");
src/lens/watch.rs:359
- This is the elem-only raw parser, so it does not preserve every path attribute as claimed. bgpkit-parser 0.20 provides
parse_ris_live_message_raw_fullspecifically because conversion toBgpElemdrops attributes such as originator ID, cluster list, AIGP, Prefix-SID, and raw-retained attributes;MrtUpdatesEncoderthen reconstructs updates only from those reduced elements. Consequently,--recordcannot retain all raw-only attributes. Carry the full parsed attributes/raw update into the recording path, or narrow the advertised recording contract.
let host = extract_host(msg_str);
match parse_ris_live_message_raw(msg_str) {
Ok(elems) => Ok(LiveFrame::Data(LiveMessage { host, elems })),
Err(e) => Err(anyhow!("failed to parse RIS Live message: {e}")),
- Files reviewed: 8/9 changed files
- Comments generated: 2
- Review effort level: Balanced
| if !args.firehose && args.filters.is_empty() && args.host.is_none() { | ||
| eprintln!( | ||
| "watch: no filter given; pass at least one filter (e.g. --origin-asn, --prefix, \ | ||
| --peer-asn, or --host to also cut server-side traffic) or pass --firehose to \ | ||
| drink the full stream (~5k msgs/s)" | ||
| ); |
| /// subscription and optional MRT recording for offline replay. | ||
| /// | ||
| /// Live vantage is RIS collectors only, not global visibility. Watch | ||
| /// refuses an unfiltered stream unless --all is passed. |
There was a problem hiding this comment.
🔵 Needs a closer look
Six moderate correctness and behavior issues remain unresolved.
Review details
Suppressed comments (6)
Previously missed (1) — in code that hasn't changed since the last review.
src/bin/commands/watch.rs:142
- The recorder has already opened/truncated the target when runtime creation is attempted, and
process::exitskips destructors. If runtime creation fails, the output is therefore left unfinalized despite the stated all-exit-path guarantee. Create the runtime before opening the recorder (creation itself does not enter an async context).
src/bin/commands/watch.rs:75
--prettyshould not bypass the table-format rejection. With--format table --pretty,out_formatremainsTable,format_elemreturnsNonefor every element, and watch runs indefinitely without displaying output. RejectTableregardless ofpretty.
if !args.pretty && output_format == OutputFormat::Table {
src/bin/commands/watch.rs:430
- If Ctrl-C arrives during this reconnect backoff,
runningbecomes false but the session loop starts another iteration and polls the already-completedctrl_cfuture again. Completed async futures cannot safely be re-polled and may panic, defeating the promised clean shutdown and recorder finalization. Break the session directly from this branch.
tokio::select! {
_ = &mut sig => running = false,
_ = tokio::time::sleep(RECONNECT_BACKOFF) => {}
}
src/bin/commands/watch.rs:329
- With
--no-reconnect, a WebSocket read error is only printed here; the loop then reachesresultwithfatal == None, so the command exits successfully. This prevents scripts from detecting an interrupted/failed stream. Preserve the error as fatal when reconnection is disabled.
Some(Err(e)) => {
eprintln!("stream error: {e}");
break;
src/bin/monocle.rs:105
- The visible command help tells users to pass
--all, butWatchArgsdefines the opt-in as--firehose; following this help produces an unknown-argument error. Update the help text to name the actual flag.
/// refuses an unfiltered stream unless --all is passed.
src/lens/watch.rs:340
parse_ris_live_message_rawdoes not preserve all path attributes: bgpkit-parser 0.20 converts the raw UPDATE toBgpElem, which lacks fields such as ORIGINATOR_ID and CLUSTER_LIST, andMrtUpdatesEncoderreconstructs the recording from that reduced model. Those attributes are therefore lost despite this full-fidelity claim. Carry the raw/full attribute representation through recording, or explicitly narrow the advertised recording fidelity.
/// Data frames are parsed from the raw BGP message bytes (the subscription
/// requests `includeRaw`), preserving all path attributes; the JSON-projected
/// parser drops attributes such as large communities. Element parse failures
- Files reviewed: 8/9 changed files
- Comments generated: 0 new
- Review effort level: Balanced
- runtime is created before the recorder opens: a failed runtime no longer leaves a truncated recording behind (process::exit skipped destructors on that path) - --format table is rejected regardless of --pretty (pretty only upgrades compact JSON, so table+pretty silently streamed nothing) - the one-shot ctrl_c future is never re-polled after completion (re-polling a completed future panics): all three backoff/select sites now route through a helper that breaks the session directly - --no-reconnect now surfaces stream errors and server closes as fatal, so scripts see a nonzero exit on interrupted streams - the command enum help names --firehose (was still --all after the rename) - recording fidelity claims narrowed to what actually holds: BgpElem-faithful (everything filtering/output see, incl. large communities), not byte-faithful (ORIGINATOR_ID/CLUSTER_LIST are not in the BgpElem model) Gates: fmt --check, clippy --all-targets --all-features -D warnings, 298 tests. Live checks: table+pretty rejected, help text correct, bad-host no-reconnect run exits with the stream error.
There was a problem hiding this comment.
🔵 Needs a closer look
SIGPIPE can bypass recording finalization, and recorder boundary/finalization tests are missing.
Review details
Suppressed comments (4)
Previously missed (1) — in code that hasn't changed since the last review.
src/bin/commands/watch.rs:516
- The new recorder's batching boundary and final partial-batch finalization are not covered by automated tests, even though comparable command modules contain unit tests. Add temporary-file round-trip cases around 500 elements (for example 499/500/501) and verify
finish()produces replayable MRT output; otherwise regressions here can silently truncate incident recordings.
src/bin/commands/watch.rs:69
mainresetsSIGPIPEtoSIG_DFLbefore dispatching commands, so on Unix a closed pipe terminateswatchat the kernel write instead of returningBrokenPipe. That bypasses this command's cleanup path (including MRT/compression finalization) and exits with signal status 141. Restore ignoredSIGPIPEforwatchbefore any output, or remove the global reset and handle write errors consistently.
pub fn run(mut args: WatchArgs, output_format: OutputFormat) {
src/bin/commands/watch.rs:430
Duration's debug format already includes its unit, so this printsreconnecting in 5ss. Remove the extras.
eprintln!("reconnecting in {RECONNECT_BACKOFF:?}s");
src/bin/commands/watch.rs:266
Duration's debug format already includes its unit (5s), so the appendedsmakes this log readretrying in 5ss. Remove the extra suffix.
eprintln!("connection failed ({e}); retrying in {RECONNECT_BACKOFF:?}s");
- Files reviewed: 8/9 changed files
- Comments generated: 0 new
- Review effort level: Balanced
…er tests
- watch restores SIGPIPE to SIG_IGN: main() resets it to SIG_DFL, which
would kill the process at the kernel write on `| head` and bypass
recorder finalization; with SIG_IGN the write returns EPIPE and the
existing BrokenPipe path finalizes the recording (verified live:
record + `| head` exits cleanly with a finalized file)
- reconnect log lines drop the duplicated unit ("5ss" -> "5s")
- recorder boundary tests: 499/500/501-element round trips across the
500-element flush boundary plus an empty-stream finish, replaying
each recording through BgpkitParser and asserting exact counts
Gates: fmt --check, clippy --all-targets --all-features -D warnings,
298 lib + 20 bin tests. Live: SIGPIPE path finalizes the recording.
There was a problem hiding this comment.
🔵 Needs a closer look
Four moderate issues involving validation, error handling, and unnecessary live traffic remain unresolved.
Review details
Suppressed comments (6)
Previously missed (4) — in code that hasn't changed since the last review.
src/bin/commands/watch.rs:256
- The explicit stdout flush result is discarded. A redirected output can accept the buffered write and fail only during
flush(), causing the command to continue or exit successfully despite losing output; non-BrokenPipe flush errors should be propagated, while BrokenPipe should follow the existing clean-finalization path.
src/bin/commands/watch.rs:264 - The
ctrl_c()future returnsio::Result<()>, but every place that polls this shared future treats bothOk(())andErr(_)as a clean Ctrl-C. If signal-handler registration fails,watchtherefore finalizes an empty recording and exits successfully instead of reporting that it cannot handle shutdown safely. Preserve the result in the connect/readselect!branches and make the backoff helper return a result so registration errors becomefatal.
src/lens/watch.rs:199 - Each subscription leaves the RIS message
typeunset, so the server also sends OPEN, KEEPALIVE, NOTIFICATION, and peer-state traffic even though watch can only emit UPDATE-derivedBgpElems. Set the subscription type toUPDATEto avoid transporting and parsing frames that can never produce output.
src/lens/watch.rs:234 - This bypasses
ParseFilters::validate()and reimplements only its conversion logic, so watch does not actually enforce parse's input semantics. For example, bgpkit-parser's plural ASN parser removes spaces, meaning--origin-asn '13 335'is accepted here as ASN 13335, whilemonocle parserejects it during validation. Build the overlappingParseFilters, callvalidate(), then useto_filters()so validation and matching cannot drift; the duplicated community helpers can then remain private.
src/bin/commands/watch.rs:91
- The suggested
--prettyoption does not make table output streamable: it only upgrades--format json, while this invocation still exits here. Recommend an option that actually changes the format, such as--jsonor--format json-line.
"watch: --format table is not supported for an unbounded stream; \
use the default PSV, JSON, or --pretty"
src/lens/watch.rs:363
- The primary
Databranch has no test: the existing frame tests cover only subscribe/error/pong classification. Add a representativeris_messagefixture with validdata.rawand assert both the extracted host and emitted elements, so changes to the hand-written classifier cannot silently break the main live-data path while all current tests still pass.
let host = extract_host(msg_str);
match parse_ris_live_message_raw(msg_str) {
Ok(elems) => Ok(LiveFrame::Data(LiveMessage { host, elems })),
Err(e) => Err(anyhow!("failed to parse RIS Live message: {e}")),
- Files reviewed: 8/9 changed files
- Comments generated: 0 new
- Review effort level: Balanced
- compile_client_filters now builds a ParseFilters value and calls validate() + to_filters() instead of reimplementing the conversion: input semantics and matching can no longer drift from monocle parse (e.g. ASN whitespace handling); the duplicated community-spec code is gone with it - subscriptions set type UPDATE: watch only emits UPDATE-derived elements, so OPEN/KEEPALIVE/NOTIFICATION frames are no longer transported or parsed - header flush errors are no longer discarded: non-BrokenPipe flush failures abort; BrokenPipe follows the clean finalization path - table-format rejection message suggests --json / --format json-line (--pretty only upgrades compact JSON, it cannot make table stream) - new Data-frame test using a real rrc10 includeRaw fixture from bgpkit-parser's examples, asserting host extraction and element expansion on the main live-data path Not taken: threading the ctrl_c() io::Result through every select branch. Signal-handler registration failure is a process-level misconfiguration that tokio documents as effectively not happening after the first successful registration; treating Err as "stop" is the safe degradation and the recording still finalizes on that path. Gates: fmt --check, clippy --all-targets --all-features -D warnings, 299 lib + 20 bin tests. Live: subscription JSON now carries type UPDATE, ack received.
There was a problem hiding this comment.
🟡 Changes recommended
Compressed recordings are not finalized with propagated encoder errors, risking corrupt output with a successful exit status.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 8/9 changed files
- Comments generated: 1
- Review effort level: Balanced
| fn finish(&mut self) -> Result<()> { | ||
| self.flush()?; | ||
| self.writer | ||
| .flush() | ||
| .map_err(|e| anyhow!("record flush failed: {e}"))?; |
Copilot round 11: compressed recordings could exit successfully despite a corrupt file, because bzip2 writers emit their stream footer on drop and a drop-time encoder failure cannot propagate through Box<dyn Write>. - finish() now drops the writer explicitly and the command verifies the recording replays to exactly the recorded element count (BgpkitParser round trip); corruption becomes a nonzero exit instead of a silent bad file. Success reports "(verified)". - finalization and verification moved out of the async fn into run(): the first attempt hit the same runtime-nesting panic as the original record-open bug (oneio's reqwest::blocking creates its own runtime inside ours); recorder and stats are borrowed across block_on now. - the BrokenPipe early-return paths no longer finalize inline; the single sync cleanup in run() covers success, Ctrl-C, fatal error, and broken pipe uniformly. Gates: fmt --check, clippy --all-targets --all-features -D warnings, 299 lib + 20 bin tests. Live: 20s rrc00 record run exits with "recorded 3 elements (verified)" and `monocle parse` replays 3/3.
There was a problem hiding this comment.
🟡 Changes recommended
The feature-gating build break, silent malformed-frame loss, and incomplete recording verification must be addressed before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (2)
Previously missed (1) — in code that hasn't changed since the last review.
src/lens/watch.rs:308
- A
ris_messagewithout arawmember currently falls through asOtherand is silently dropped, even though this function promises to surface element parse failures. Classify by the frame type and invokeparse_ris_live_message_rawfor every data frame; bgpkit-parser 0.20 then reports the missing raw payload as an error instead of losing an update.
src/bin/commands/watch.rs:639
- This count-only check can report a semantically changed recording as “verified.” With the pinned bgpkit-parser 0.20.0,
MrtUpdatesEncoderconverts throughAttributes::from(&BgpElem), which does not emitAtomicAggregate; an element withatomic == truetherefore replays withfalsewhile this check still passes. Preserve all supported element attributes and verify their values (not only the count) before claiming the recording is faithful.
let replayed = parser.into_iter().count();
if replayed != self.count as usize {
return Err(anyhow!(
"recorded file verification failed: wrote {} elements but replayed {replayed}",
self.count
- Files reviewed: 8/9 changed files
- Comments generated: 1
- Review effort level: Balanced
| // InspectLens - unified AS and prefix information lookup | ||
| #[cfg(feature = "lib")] | ||
| pub mod inspect; | ||
| pub mod watch; |
Copilot round 12: - ris_message frames now classify by type instead of by presence of a "raw" member: a data frame with a missing/unexpectedly absent raw payload reaches the raw parser, which reports it as an error instead of silently falling through to Other and dropping an update - recording verification wording scoped to what it proves: output says "count-verified" and the method doc states that per-attribute equality cannot be claimed because the MRT encoder rebuilds from the BgpElem projection (ATOMIC_AGGREGATE/ORIGINATOR_ID do not round-trip in parser 0.20). Deep attribute verification was considered and declined: it would re-implement BgpElem deep-equal for an exit-time check whose job is corruption detection, not fidelity proof - the claimed feature-gating build break does not reproduce: cargo check passes for --no-default-features --features lib and --features server (CI also builds green); the two accompanying inline comments are the same stale anchors rebutted earlier
There was a problem hiding this comment.
🟡 Changes recommended
Four moderate issues remain in subscription acknowledgement handling, shutdown interruption, JSON parsing, and recording fidelity.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (3)
Previously missed (1) — in code that hasn't changed since the last review.
src/lens/watch.rs:332
- This scanner does not respect JSON nesting or object-member ordering. A valid RIS frame with
dataserialized before the outertype(the data object itself contains"type":"UPDATE") is classified asOther, and escaped quotes in string values are truncated. Parse the envelope once withserde_jsonand read the top-leveltypeplusdata.host/data.messageexplicitly.
src/bin/commands/watch.rs:413
- A multi-prefix/peer plan requests one acknowledgement per subscription, but this single flag treats the first acknowledgement—and even an unrelated data frame—as confirmation for the entire plan. If only part of the plan is accepted, watch continues while silently missing the unacknowledged filter combinations. Track the expected acknowledgement count and only report the subscription plan ready after all acknowledgements arrive; data should not increment that count.
if !subscribed_ok {
eprintln!("subscription acknowledged by server");
subscribed_ok = true;
}
}
LiveFrame::Error(err_text) => {
fatal = Some(anyhow!("RIS Live rejected the subscription: {err_text}"));
break;
}
LiveFrame::Other => {}
LiveFrame::Data(live) => {
if !subscribed_ok {
// Data before an ack still means the subscription works.
subscribed_ok = true;
src/lens/watch.rs:295
- The recording is not
BgpElem-faithful: the pinned bgpkit-parser 0.20 encoder does not re-encode the element'satomicflag, as this PR itself notes inMrtRecorder::verify_replayable. Consequently replay can differ from displayed live elements, so this guarantee should be qualified.
/// drops. Attributes outside the `BgpElem` model (e.g. ORIGINATOR_ID,
/// CLUSTER_LIST) are still not represented downstream; recordings are
/// `BgpElem`-faithful (exactly what filtering and output see), not
/// byte-faithful to the original UPDATE. Element parse failures are returned
- Files reviewed: 8/9 changed files
- Comments generated: 1
- Review effort level: Balanced
| if let Some(rec) = recorder.as_mut() { | ||
| let finish_res = rec.finish().and_then(|()| rec.verify_replayable()); |
Adds
monocle watch(issue #100, RIS Live portion): streams live BGP messages from RIPE RIS Live, normalized toBgpElemso the parse filter grammar, output formats, and field selection all apply unchanged.Examples
Watch your own network's propagation seconds after a policy change:
Capture an incident live, replay it offline with the same filters:
Scope server-side to cut bandwidth (multi-prefix/peer fan out to one subscription per combination):
$ monocle watch --host rrc10 --prefix 1.1.1.0/24,8.8.8.0/24 --jsonPipe into other tools; broken pipes exit cleanly:
$ monocle watch --origin-asn 13335 --json | jq -c 'select(.prefix | contains("2400:cb00"))' | head -20Behavior
host,prefix(one subscription per prefix, both specificity flags explicit),peer(one per peer IP), and elem type viarequire— purely to reduce traffic. Origin ASN, peer ASN, community, and AS-path predicates always run client-side with parserFiltersemantics: the RISpathpattern cannot express AS_SET origins, and RIS selects whole UPDATE messages while elements expand per prefix, so every element predicate is re-checked locally. Watch matches exactly whatmonocle parsewould match.--firehoseto drink the full unfiltered stream. Server-side scope (--host/--prefix/--peer-ip) is recommended to cut bandwidth and RIPE-side load.includeRaw+parse_ris_live_message_raw) preserves all path attributes, so--communitylarge-community filters work and recordings keep raw-only attributes.ris_subscribe_ok;ris_erroraborts with the server message.--record PATHwrites the filtered stream as BGP4MP MRT updates (batched flushes of 500) for offline replay withmonocle parseand the same filters. The record writer opens before the async runtime (oneio wrapsreqwest::blocking) and only after argument/filter validation; every exit path (success, Ctrl-C, fatal error, broken pipe) finalizes the recording.--no-reconnectto disable); one Ctrl-C handler spans connect, subscribe, read, and backoff windows; non-BrokenPipe output errors abort nonzero.--helpnotes that live vantage is RIS collectors only, not global visibility.RouteViews BMP/Kafka is intentionally out of scope for this PR; it follows later on the same normalization.
Verified
cargo fmt --check,cargo clippy --all-targets --all-features -- -D warnings, full test suite (298 lib tests incl. new watch-lens tests), plus live runs against rrc00: multi-prefix subscription fan-out confirmed in subscription JSON with acks,--record+monocle parseround trips element-for-element, firehose measured over 60s (311k messages, 708k elements, 28 MB RSS), and markdown/PSV headers pipe toheadwithout panics.