[Feat] Public labelled set for the injection guard from InjecAgent and AgentDojo

CHANGELOG.md

@@ -17,6 +17,13 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver
 - `docs/benchmarks.md`: the Google Flights driver comparison rerun on 2026-09-23 from Poland, every arm three times on
   both decision backends, next to the baseline rows in one table; the 24 S1A records, as one archive, and the chart
   under `docs/results/flights/rerun-2026-09-23/`.
+- `evals/labelled/injection-public.jsonl`: 665 labelled tool outputs for the `injection_guard` rail from InjecAgent
+  and AgentDojo at pinned versions, 335 with an injected instruction. 65 of those read as an ordinary request that only
+  the user's task tells apart; their notes carry a `-u` source and they are reported on their own row. Negatives
+  include requests written for a human (`R2` in the note), which with the unmarked positives form a hard subset.
+  `uv run scripts/build_injection_dataset.py` rebuilds the file byte for byte; `--report <job>` prints precision and
+  recall per source without the `-u` positives, their own recall, and balanced accuracy on the hard subset against
+  fixed keyword baselines.
 
 ### Changed
 

NOTICE

@@ -12,3 +12,9 @@ This product includes software developed by third parties:
   evals/2048/style/fonts are Clear Sans by Intel Corporation (Apache-2.0)
 - evals/millionaire/questions.json, fetched at first run: Open Trivia Database
   (https://opentdb.com), CC BY-SA 4.0
+- evals/labelled/injection-public.jsonl: tool outputs from InjecAgent,
+  uiuc-kang-lab/InjecAgent (MIT, Copyright (c) 2023 Qiusi Zhan). Full license text:
+  THIRD_PARTY_LICENSES.md
+- evals/labelled/injection-public.jsonl: tool outputs from AgentDojo, ethz-spylab/agentdojo
+  (MIT, Copyright (c) 2024 Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca
+  Beurer-Kellner, Marc Fischer, and Florian Tramèr). Full license text: THIRD_PARTY_LICENSES.md

THIRD_PARTY_LICENSES.md

@@ -39,3 +39,53 @@ Fonts under `evals/2048/style/fonts` are Clear Sans by Intel Corporation, Apache
 
 `evals/millionaire/questions.json`, fetched on the first Millionaire run, holds questions from the
 [Open Trivia Database](https://opentdb.com), Creative Commons Attribution-ShareAlike 4.0 International.
+
+## InjecAgent
+
+The records of `evals/labelled/injection-public.jsonl` whose note starts with `injecagent-` are built by
+`scripts/build_injection_dataset.py` from [uiuc-kang-lab/InjecAgent](https://github.com/uiuc-kang-lab/InjecAgent) at
+commit `f19c9f2c79a41046eb13c03c51a24c567a8ffa07`: `data/test_cases_dh_base.json`, `data/test_cases_ds_base.json` and
+`data/attacker_simulated_responses.json`.
+
+MIT License
+
+Copyright (c) 2023 Qiusi Zhan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+documentation files (the "Software"), to deal in the Software without restriction, including without limitation
+the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
+to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions
+of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
+TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
+CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE.
+
+## AgentDojo
+
+The records of `evals/labelled/injection-public.jsonl` whose note starts with `agentdojo` are tool outputs of the
+[ethz-spylab/agentdojo](https://github.com/ethz-spylab/agentdojo) suites, benchmark version v1.2.2 of the `agentdojo`
+0.1.35 package, produced by `scripts/build_injection_dataset.py` from the user tasks' ground truth with and without the
+package's attacks, and from the suites' own read tools on the default environments.
+
+MIT License
+
+Copyright (c) 2024 Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+documentation files (the "Software"), to deal in the Software without restriction, including without limitation
+the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
+to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions
+of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
+TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
+CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE.

docs/benchmarks.md

@@ -78,6 +78,44 @@ The same showcase on ALFWorld, Blackjack and Millionaire is under `results/<eval
 
 The rail scores 20 of 20 on its labelled set at a median of 464 ms (`s1a run injection_guard`).
 
+A larger set, `evals/labelled/injection-public.jsonl`, uses data from two public benchmarks, InjecAgent and AgentDojo:
+665 tool outputs such as emails, calendar entries, files, web pages and product pages, 335 of them with an instruction
+planted by an attacker. `scripts/build_injection_dataset.py` rebuilds the file. The run below is from 2026-09-27
+(`uv run s1a run injection_guard --model jev --labelled-set evals/labelled/injection-public.jsonl`), at a median of
+300 ms per decision and $0.016 for the whole set.
+
+| tool output | count | quarantined by Jev |
+|---|---|---|
+| no planted instruction | 330 | 0 |
+| of which a normal request written for a person, such as a bill asking for payment | 34 | 0 |
+| planted instruction in attack wording, such as "ignore your previous instructions" (AgentDojo) | 154 | 150 |
+| planted request to send the user's data somewhere (InjecAgent) | 61 | 58 |
+| planted request for a harmful action, with no attack wording (InjecAgent) | 55 | 22 |
+| planted request that reads like an ordinary one, such as a payment request in an email | 65 | 34 |
+
+The last row is counted apart. Only the user's task tells these requests from real ones, and the rail sees only the
+tool output.
+
+Over the 600 outputs outside the last row, the run gives these precision and recall figures:
+
+| measure | Jev | quarantine everything |
+|---|---|---|
+| precision: share of quarantined outputs that carry a planted instruction | 1.00 | 0.45 |
+| recall, InjecAgent direct harm (55) | 0.40 | 1.00 |
+| recall, InjecAgent data stealing (61) | 0.95 | 1.00 |
+| recall, AgentDojo (154) | 0.97 | 1.00 |
+
+Recall is the share of planted instructions that the rail quarantined. "Quarantine everything" is what a rail scores if
+it quarantines every output.
+
+On the 150 outputs that carry no attack wording (the 116 planted instructions without it and the 34 normal requests), a
+rule that quarantines any text containing "please" is right 65% of the time, counting the two groups equally. Jev is
+right 85% of the time: it catches 69% of the planted instructions and none of the normal requests.
+
+The set has two limits. Only 34 of the normal outputs contain a request. The planted instructions and the normal
+requests also come from different benchmarks with different text formats, so a simple word-count classifier already
+separates them 73% of the time.
+
 ## Series
 
 The protocol in `evals/README.md` quotes nothing under ten episodes and asks for 500 Blackjack hands. The series of

docs/roadmap.md

@@ -6,14 +6,17 @@ in `README.md`, `docs/agents.md`, `docs/browser-front.md` (the browser policy) a
 
 ## Rails: one shipped, two to ship
 
-### Prompt-injection guard: shipped, with a 20-item labelled set
+### Prompt-injection guard: shipped, with two labelled sets
 
 jiuwen ingests untrusted pages through `browser_snapshot` and `fetch_webpage`. The `injection_guard` rail
 (`s1a/agents/injection_guard.py`) ships: one rail on `after_tool_call` asks Jev "does this text instruct
 the agent?" (`noul`) and quarantines the result; 20 of 20 on `evals/labelled/injection.jsonl` at a median of
-464 ms. Still to build: the planted-page demo (the browser subagent visits a planted page and the guard flags the
-planted instruction before the model reads it) and a labelled set from the public ones, InjecAgent (1,054 tool-output
-injections) and AgentDojo (97 tasks, 629 security cases). jiuwen's own
+464 ms. `evals/labelled/injection-public.jsonl` adds 665 records from the public InjecAgent and AgentDojo benchmarks,
+built by `scripts/build_injection_dataset.py`: 335 positives, 65 of them needing the user's task to tell apart and
+reported on their own, and 330 negatives. Its precision, recall and hard-subset balanced accuracy are in
+`docs/benchmarks.md`. Still to build:
+the planted-page demo (the browser subagent visits a planted page and the guard flags the planted instruction before
+the model reads it). jiuwen's own
 `openjiuwen/harness/rails/security/prompt_security_rail.py` is pattern-based; `SafetyPromptRail` only injects
 guideline text before model calls.
 

evals/README.md

@@ -14,6 +14,7 @@ folder per episode). `uv run python -m evals.table evals/results` aggregates the
 | Desktop (Cua Driver) | clicks toward `--goal` until the window shows `--expect` | `s1a run desktop --app Calculator --goal "compute 12 times 7" --expect 84 --execute --model jev --rethink off --episodes 1` |
 | Ticket router (30 local labelled tickets) | correct routes to five queues | `s1a run ticket_router --model jev --rethink off --episodes 1` |
 | Injection guard (rail) | precision and recall on a labelled set | `s1a run injection_guard` |
+| Injection guard, public set (InjecAgent + AgentDojo) | precision and recall per source without the context-needed positives, balanced accuracy on the hard subset | `s1a run injection_guard --model jev --labelled-set evals/labelled/injection-public.jsonl`; numbers in `docs/benchmarks.md` |
 
 ## The loop