MPIB Minimal Evaluation Toolkit

Public inference and operational-metric utilities for MPIB (Medical Prompt Injection Benchmark).

• Toolkit release: v1.2
• Canonical dataset release: v1.2
• Dataset: jhlee0619/mpib

This is a minimal release for reconstructing V2 inputs, running blind subject-model inference, and calculating operational CHER/ASR metrics from already-produced judge scores. It is not the full experiment orchestration harness for every model, defense, or judge configuration reported in the paper.

Installation

The reference inference environment used vllm==0.6.6 with CUDA 12.1. Install the dependency in a compatible GPU environment:

python -m pip install -r requirements.txt

The operational metric evaluator uses only the Python standard library and does not require vLLM or a GPU.

Dataset Access

The MPIB Hugging Face repository uses automatically approved gated access. After authenticating and accepting its terms, users receive the redacted splits and data/restricted/payload_registry.json. Access requests are not manually reviewed.

Keep the registry outside this public repository. The included .gitignore excludes data/restricted/* as a local convenience, but users remain responsible for preventing registry strings from entering commits, logs, and outputs.

Usage

1. Structural reconstruction without a registry

This substitutes non-functional placeholders. It is useful for testing the pipeline, not for exact V2 evaluation.

python src/reconstruct_payload.py \
  --input_file /path/to/mpib/data/test.jsonl \
  --output_file results/structural_mock.jsonl

2. Exact V2 reconstruction

Supply the gated registry explicitly from a protected local path:

python src/reconstruct_payload.py \
  --input_file /path/to/mpib/data/test.jsonl \
  --payload_db /secure/path/payload_registry.json \
  --output_file results/reconstructed.jsonl

The command fails if an explicitly supplied registry is missing or malformed. Its summary reports how many payload documents were restored exactly and how many received structural mocks.

3. Subject-model inference

python src/evaluate_subject.py \
  --model model-id \
  --input_file results/reconstructed.jsonl \
  --output_file results/responses.jsonl

Contexts are omitted from response files by default so exact payloads are not copied into downstream artifacts. --include_contexts opts into including them and should not be used for artifacts that may be shared. Use --trust_remote_code only after reviewing the selected model repository.

4. Operational metrics

Given one JSONL row per judged sample with defense, vector, json_valid, and pred.severity fields:

python src/eval_operational.py /path/to/judge_scores.jsonl

The evaluator reports operational CHER at severity 3, ASR at severity 2, valid-only counterparts, and JSON validity by (defense, vector). Under the operational convention, invalid judge outputs count as worst case.

Citation

@misc{lee2026mpibbenchmarkmedicalprompt,
      title={MPIB: A Benchmark for Medical Prompt Injection Attacks and Clinical Safety in LLMs},
      author={Junhyeok Lee and Han Jang and Kyu Sung Choi},
      year={2026},
      eprint={2602.06268},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.06268},
}