stress: locate panics per thread, and say when a panic is in a dependency

CHANGELOG.md

@@ -21,6 +21,24 @@
 - Race messages name the threads (`by thread T5 (ftm-refill)`), the text summary labels
   each stack with its thread, and JSON stacks carry a `thread` field.
 - A race whose access is in PyO3's own source says so, with the PyO3 version and line.
+- **`stress/panic` is located per call, not per message.** The driver records the native
+  thread of each panic and Rust prints the same id in its panic line, so two sites that
+  panic with the same message are two findings, each naming the callables that panicked
+  there with their own counts. When Rust prints no thread id, a finding lists every
+  site that shares the message.
+- **A panic inside a dependency says so.** The message names the dependency and its
+  version (and PyO3's argument conversion when the site is there), and the headline
+  counts it as "inside a dependency, reached from your extension". When the log holds a
+  Rust backtrace, the first frame in the crate's own code is the location.
+- A `stress/panic` message says when mutators were running.
+
+### JSON
+
+- `stress.panics`: every distinct panic message per callable, with a count.
+- `stress.panic_threads`: per native thread id, the panicking calls in order, as
+  `[callable, message, count]` runs.
+- A `stress/panic` finding inside a dependency carries `dependency` (for example
+  `"pyo3 0.29.2"`).
 
 ## 0.1.0 — 2026-09-25
 

docs/limitations.md

@@ -47,15 +47,19 @@ Known gaps, each still open:
   invisible to it.
 - **Exceptions raised only under concurrency**, other than Rust panics, are counted in
   the JSON output (`stress.exceptions`) but not surfaced in the text summary.
-- **A panic raised inside a dependency's code** is located at the dependency's source
-  line, not at the frame of yours that led there. The finding's symbol is the only pointer
-  to your callable.
-- **Panic locations are matched by message text.** When two sites panic with the same
-  message, a finding can be filed at the other site, and the second site may not appear
-  as a finding of its own.
+- **A panic raised inside a dependency's code** is labelled with the dependency and
+  located at its source line. The frame of yours that led there is found only when the
+  log holds a Rust backtrace: replay with `RUST_BACKTRACE=1` set. Backtraces are off by
+  default because printing one for every panic slows each panicking call about tenfold,
+  which changes the schedule under test.
+- **Panic sites are told apart by thread id**, which current Rust prints in its panic
+  line. With a toolchain that prints none, a panic is located by its message, and when
+  two sites share the message the finding lists both without saying which call panicked
+  where.
 - **A panic on input a mutator made invalid reads as a concurrency panic**, because the
-  single-threaded baseline never sees the mutator's transient state. A mutator must keep
-  the shared inputs valid at every instant.
+  single-threaded baseline never sees the mutator's transient state. The finding says
+  when mutators were running, but does not check whether one caused it. A mutator must
+  keep the shared inputs valid at every instant.
 - **Some dependency-internal TSan reports still fail runs.** Beyond the suppressed
   crossbeam-deque race, reports inside dependencies' fence-based synchronisation (which
   TSan does not model) and glibc's thread-local teardown can be filed as "in your

docs/stress.md

@@ -134,7 +134,7 @@ was, so that shows mutators reach this class of bug, not that they find it unaid
 
 A mutator must keep the shared inputs **valid at every instant**. The baseline never sees
 a mutator's transient state, so a call that panics on input the mutator made invalid is
-reported as a panic under concurrency.
+reported as a panic under concurrency. The finding says that mutators were running.
 
 A mutator that only **rewrites a buffer's contents** in place (a slice assignment into a
 `bytearray` the extension is reading, or a numpy array refilled with `arr[:] = ...`) races
@@ -199,10 +199,15 @@ across Python threads, so they are neither driven nor counted as a coverage hole
 
 **Panics under contention are findings.** A Rust panic (`PanicException`) that a
 callable raises under concurrency but never in its baseline is
-reported as `stress/panic`, with the count, the first panic message and — from Rust's
+reported as `stress/panic`, with the count, the panic message and — from Rust's
 own panic output — the source line that panicked. Callables panicking at the same line are
-one finding. The line is matched by panic message, so two sites panicking with the same
-message can be filed under one of them (see Limits). It is not a data
+one finding, with each callable's own count. The driver records the thread each panic
+happened on, and Rust prints the same thread id in its panic line, so two sites panicking
+with the same message are two findings. A panic inside a dependency (PyO3, another crate,
+the standard library) says so and carries a `dependency` field, and the headline counts it
+as inside a dependency rather than in your extension. When the log holds a Rust
+backtrace (replay with `RUST_BACKTRACE=1` set), the first frame in your own code becomes
+the location. It is not a data
 race and TSan may see nothing, but it is contention the code does not handle — and since
 `PanicException` is a `BaseException`, callers' `except Exception` will not catch it.
 A panic the baseline also raised is how the method treats those arguments, and
@@ -289,9 +294,9 @@ relative to that directory. Every run prints the complete command that replays i
 - **Wrong results are not checked.** A call that returns a wrong value under concurrency
   without raising is invisible; exceptions other than panics that occur only under
   concurrency are counted in the JSON output but not surfaced in the summary.
-- **Panic attribution is coarse.** A panic inside a dependency is located at the
-  dependency's source line, not at your frame that led there; panic sites are matched by
-  message text, so one site can hide another with the same message.
+- **A panic inside a dependency is located at the dependency's line** unless the log holds
+  a backtrace (replay with `RUST_BACKTRACE=1`). When the toolchain prints no thread
+  id in its panic lines, sites that share a message are listed together.
 - **Mutators must keep inputs valid** (above), or their transient state reads as a
-  concurrency panic.
+  concurrency panic; the finding says mutators were running, nothing more.
 - See [limitations.md](limitations.md#what-stress-does-not-report-yet) for the full list.

fixtures/racy/stress-panic-two-sites/Cargo.toml

@@ -0,0 +1,12 @@
+[package]
+name = "stress-panic-two-sites"
+version = "0.0.0"
+edition = "2021"
+publish = false
+
+[lib]
+name = "stress_panic_two_sites"
+crate-type = ["cdylib", "rlib"]
+
+[dependencies]
+pyo3 = { version = "0.29", features = ["extension-module"] }

fixtures/racy/stress-panic-two-sites/expected.toml

@@ -0,0 +1,5 @@
+race = false
+rules = []
+stress_rules = ["stress/panic"]
+symbols = ["head", "tail"]
+justification = "Two `try_lock().expect()` sites with the same message on one shared instance: neither panics on one thread, each panics at its own line once two threads overlap. Not a data race (a real mutex guards the data), so TSan must stay silent. `stress` must report two stress/panic findings, one per line, each naming only the method that panicked there. Matching panics to sites by message text filed both under one line."

fixtures/racy/stress-panic-two-sites/src/lib.rs

@@ -0,0 +1,43 @@
+//! Two panic sites with one message — not a data race.
+//!
+//! `head` and `tail` each take the same exclusive claim with `try_lock` and
+//! `expect` it, with the same message. On one thread neither panics; as soon
+//! as two threads overlap, each panics at its own line. The two sites print
+//! identical messages, so a tool that locates panics by message alone files
+//! both under one line and loses the other. Each must be reported at its own
+//! line, naming the method that panicked there.
+
+use pyo3::prelude::*;
+use std::sync::Mutex;
+
+#[pyclass]
+struct Queue {
+    items: Mutex<Vec<u32>>,
+}
+
+#[pymethods]
+impl Queue {
+    #[new]
+    fn new() -> Self {
+        Queue {
+            items: Mutex::new((0..64).collect()),
+        }
+    }
+
+    fn head(&self) -> u32 {
+        let guard = self.items.try_lock().expect("queue is busy");
+        std::thread::sleep(std::time::Duration::from_micros(50));
+        guard[0]
+    }
+
+    fn tail(&self) -> u32 {
+        let guard = self.items.try_lock().expect("queue is busy");
+        std::thread::sleep(std::time::Duration::from_micros(50));
+        guard[guard.len() - 1]
+    }
+}
+
+#[pymodule]
+fn stress_panic_two_sites(m: &Bound<'_, PyModule>) -> PyResult<()> {
+    m.add_class::<Queue>()
+}

fixtures/racy/stress-panic-two-sites/tests/test_threads.py

@@ -0,0 +1,10 @@
+"""Each test builds its own object, so under `ftcheck ci` no instance is
+shared and neither site panics."""
+import stress_panic_two_sites as m
+
+
+def test_head_and_tail():
+    queue = m.Queue()
+    for _ in range(100):
+        assert queue.head() == 0
+        assert queue.tail() == 63

python/ftcheck/ci/__init__.py

@@ -24,16 +24,28 @@ def describe_findings(findings: list[dict]) -> str:
         ("stress/panic", "panic under concurrency", "panics under concurrency"),
         ("stress/hang", "hang under concurrency", "hangs under concurrency"),
     ]
+    # A panic inside a dependency's code is reached through the extension's
+    # surface, but calling it "in your extension" sent users to the wrong code.
+    dependency = sum(1 for f in findings if f["rule"] == "stress/panic" and f.get("dependency"))
+    yours = [f for f in findings if not (f["rule"] == "stress/panic" and f.get("dependency"))]
     parts = []
     for prefix, one, many in kinds:
-        n = sum(1 for f in findings if f["rule"].startswith(prefix))
+        n = sum(1 for f in yours if f["rule"].startswith(prefix))
         if n:
             parts.append(f"{n} {one if n == 1 else many}")
-    other = len(findings) - sum(int(p.split()[0]) for p in parts)
+    other = len(yours) - sum(int(p.split()[0]) for p in parts)
     if other:
         parts.append(f"{other} other finding{'s' if other != 1 else ''}")
-    joined = parts[0] if len(parts) == 1 else ", ".join(parts[:-1]) + " and " + parts[-1]
-    return f"{joined} in your extension"
+    described = []
+    if parts:
+        joined = parts[0] if len(parts) == 1 else ", ".join(parts[:-1]) + " and " + parts[-1]
+        described.append(f"{joined} in your extension")
+    if dependency:
+        noun = "panic" if dependency == 1 else "panics"
+        described.append(
+            f"{dependency} {noun} under concurrency inside a dependency, reached from your extension"
+        )
+    return " and ".join(described)
 
 # pytest exit statuses that mean the suite itself never ran properly.
 _PYTEST_BROKEN = {2: "interrupted", 3: "internal error", 4: "usage error", 5: "no tests collected"}