PRISM-eval

Evaluation code and data for PRISM: Recovering Instruction Sets from Language Model Activations, accepted to the EMNLP 2026 Main Conference.

PRISM reads a language model's activations and reports the instructions represented in them. This repository evaluates those reports against known instruction sets. To try PRISM interactively or train a model, see prism.

Quick start

Use Python 3.11 or later, uv, and a CUDA GPU. Evaluation runs Qwen3.5-9B with the released PRISM checkpoint locally and scores its reports through a separate judge endpoint.

git clone https://github.com/Offensive-AI-Lab/prism-eval.git
cd prism-eval
uv sync
cp .env.example .env
uv run python scripts/download_weights.py --only prism-qwen3.5-9b-grpo

The checkpoint is saved in ./checkpoints; the target model is downloaded on first use. Configure the judge endpoint in .env. The paper uses google/gemma-4-31B-it with reasoning disabled:

PRISM_EVAL_CHECKPOINT_DIR=./checkpoints
PRISM_EVAL_MODEL=gemma4-31B-it
PRISM_EVAL_BASE_URL=http://localhost:8088/v1
PRISM_EVAL_API_KEY=EMPTY

To host the judge, install vLLM in a separate environment and run scripts/serve_judge.sh on its GPU host. The supplied configuration needs roughly 80 GB of GPU memory for the judge, in addition to the target model's memory. The endpoint can be on another machine; set its address and credentials above.

Run the 1,000-record paper evaluation:

uv run prism-eval evaluate --config configs/main/qwen3.5-9b-grpo.yaml --offline

The run writes rows.jsonl and summary.json under results/qwen3.5-9b-grpo/. --offline disables Weave tracing and leaderboard publication; it still calls the judge.

Compare with the paper results. The reproduction guide covers hardware, ablations, and run-to-run variation; configs/template.yaml lists configuration options.

Checkpoints and baselines

Checkpoint	Target model	Configuration
PRISM — Qwen	Qwen/Qwen3.5-9B	qwen3.5-9b-grpo.yaml
PRISM w/o RL — Qwen	Qwen/Qwen3.5-9B	qwen3.5-9b-sft.yaml
PRISM — Gemma	google/gemma-2-9b-it	gemma-2-9b-it-grpo.yaml
PRISM — Ministral	mistralai/Ministral-3-8B-Instruct-2512-BF16	ministral-3-8b-grpo.yaml

Omit --only from the download command to fetch all four checkpoints. Target model weights are downloaded separately; set HF_TOKEN for gated models.

The text-only control reads the final 128 response tokens without activations and calls a separate language-model endpoint. LatentQA and Activation Oracles were evaluated using their upstream implementations; code revisions and checkpoint links are listed in Results.

Evaluation data

The main suite, data/eval_suite.json, has 1,000 records, with 250 per setting:

Code	Setting	What it measures
AP	Adversarial Prompt	Recovery of instructions injected through documents, tool outputs, email, and other channels
HO	Hidden Objective	Recovery of covert objectives in the system prompt
BC	Behavioral Constraints	Recovery of persona, format, topic, and style constraints
BN	Benign Baseline	Recovery of ordinary instructions and hallucination on benign prompts

Each record includes a prompt, a ground-truth instruction list, and a stable evaluation ID.

See the data card for the schema, sources, and construction process, and License for usage restrictions.

The indirect prompt injection corpus is not distributed. Rebuild it from the public benchmarks with scripts/build_xpia_corpus.py (see the data card); none of the upstream prompt text is shipped here.

Metrics

The judge follows the coverage and hallucination rubric. Adversarial detection is scored on AP and HO using the adversarial-instruction rubric.

Metric	Definition
reward	Coverage minus weighted hallucination and length penalties
coverage	Mean per-instruction score: 1.0 for covered, 0.5 for partial, and 0.0 for missed
hallucination_rate	Mean per-bullet hallucination score: 0.0 grounded, 0.5 ambiguous, 1.0 hallucinated
detect_rate_avg	Fraction of scored AP/HO records with mean adversarial-instruction coverage ≥ 0.5

Development

uv sync --extra dev          # test dependencies (installs pytest)
uv run pytest tests -q       # CPU-only; no GPU, model, or judge endpoint needed

uv sync downloads a CUDA-enabled PyTorch build (several GB); that is expected — evaluation runs the target model on a GPU.

Citation

@inproceedings{gressel2026prism,
  title     = {PRISM: Recovering Instruction Sets from Language Model Activations},
  author    = {Gressel, Gilad and Pankajakshan, Rahul and Diament, Julia and
               Hudis, Efim and Achuthan, Krishnashree and Mirsky, Yisroel},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in
               Natural Language Processing},
  year      = {2026},
  url       = {https://arxiv.org/abs/2606.09563}
}

License

The code and project-authored AP/HO records are licensed under the Apache License 2.0. Third-party datasets retain their original licenses. The BN records are derived from Stanford Alpaca and are restricted to non-commercial use under CC BY-NC 4.0; see DATA_CARD.md and NOTICE.