Medical AI Superintelligence Test (MAST) Leaderboard

Overview

MAST (Medical AI Superintelligence Test) is a suite of clinically realistic benchmarks to evaluate real-world medical capabilities of artificial intelligence models. The system provides a leaderboard where AI models submit API endpoints that are automatically tested against standardized medical scenarios.

The live leaderboard is available at arise-ai.org/mast/technical.

This repository provides instructions and test files to validate your custom model API endpoint. After passing validation, view the Submission Agreement and submit the Registration Form for review by the MAST team. The API and token are used only for benchmark execution and are not stored after evaluation.

How It Works

1. Submitters provide a single API endpoint with authentication token
2. Leaderboard runs automated tests against all benchmarks using that endpoint
3. API calls are made with standardized prompts and test cases for each benchmark
4. Responses are validated for format compliance
5. Results are manually reviewed prior to publication on the leaderboard

Structure

mast/
├── benchmarks/
│   ├── donoharm/               # First Do NOHARM benchmark
│   │   ├── run.py, score.py    # Run-it-yourself eval on the open subset (Gemini judge)
│   │   ├── dataset/            # 30 open cases + perturbations + rubrics
│   │   ├── README.md           # Setup, F1_weighted metric, reference scores
│   │   └── submission/         # Optional endpoint validator (validator.py, schema.json, inputs/, outputs/)
│   ├── sct/                    # Script Concordance Test benchmark
│   │   ├── run.py, score.py    # Run-it-yourself eval on the open subset (deterministic scoring)
│   │   ├── dataset/            # 174 open items + expert consensus distributions
│   │   ├── examples/           # A full GPT-5.5 reference run + its score outputs
│   │   ├── README.md           # Setup, sct_score metric, reference scores
│   │   └── submission/         # Optional endpoint validator (validator.py, schema.json, inputs/, outputs/)
│   └── template/               # Template for new benchmarks
├── results/                    # API response storage (per-benchmark)
├── scripts/
│   ├── validate_all.py         # Master API tester
│   ├── utils.py                # Shared utilities
│   ├── config.json             # API endpoint config (gitignored)
│   └── config.example.json     # Template for submitters
├── docs/
│   ├── contributing.md         # Contribution guidelines
│   ├── submission_agreement.md # Terms for submitters
│   └── benchmark_descriptions.md  # Detailed benchmark info
└── README.md

Quick Start

For Submitters

1. Clone the repository:

git clone https://github.com/ARISENetwork/mast.git
cd mast

2. Set up your API endpoint: provide a hosted endpoint for accessing and benchmarking your model.

3. Configure your endpoint by copying and editing the config:

cp scripts/config.example.json scripts/config.json
# Edit scripts/config.json with your API details

4. Test your endpoint:

python scripts/validate_all.py

API Request Format

Each benchmark makes HTTPS POST requests with:

• Method: POST
• Headers:
  • Authorization: Bearer {token}
  • Content-Type: text/plain
• Body: benchmarks/donoharm/prompts/default.md + "\n" + test_input.txt
• Timeout: Up to 300 seconds

The default prompt is unprompted: benchmarks/donoharm/prompts/default.md is empty, so the body is effectively the bare clinical case (a leading newline followed by the case text) with no format or length instruction, and your endpoint should return a free-text management plan. See benchmarks/donoharm/submission/inputs/test_001.txt for an example case. The paper's prompted arms (concise.md, thorough.md) live in benchmarks/donoharm/prompts/; the validator and the leaderboard evaluation use the unprompted default.

Response Format

APIs must return a JSON object containing a free-text clinical management plan:

{
  "response": "Assessment: Grade 3 infusion reaction to nivolumab...\n\n1. Refer to Allergy/Immunology for urgent evaluation...\n2. Hold next nivolumab dose until allergy clearance...\n3. ..."
}

The response field must contain at least 50 characters of clinical text. There is no required structure within the text itself; the model should write a management plan as described in the prompt. See benchmarks/donoharm/submission/outputs/test_001.txt for an example of a valid response.

OpenAI-compatible endpoints are also accepted. If your API returns the standard OpenAI chat completions format (choices[0].message.content), the validator will automatically extract the content. This includes endpoints served via OpenRouter or any OpenAI-compatible provider.

Benchmarks

The MAST suite spans the clinical capabilities measured on the public leaderboard. Each benchmark links to its code, data, or site. Full descriptions and demos: arise-ai.org/mast/benchmarks.

Benchmark	Clinical capability	Code / data	Paper
First Do NOHARM v2	Safety, management reasoning	benchmarks/donoharm/	arXiv
Script Concordance Test (SCT)	Reasoning under uncertainty	benchmarks/sct/	NEJM AI
CPC-Bench	Diagnostic reasoning	cpcbench.com	arXiv
MedAgentBench v2	Agentic EHR tasks	GitHub	Paper
PhysicianBench	Agentic EHR tasks	GitHub	arXiv
ReXrank Mini	Multimodal radiology	GitHub	arXiv
Multimodal Images	Multimodal dermatology	DDI · MIDAS	DDI · MIDAS

ReXrank Mini is MAST's curated subset of the full ReXrank benchmark, run with that harness.

• First Do NOHARM v2: free-text management plans reconstructed from real generalist-to-specialist consults, scored by an LLM judge against specialist-authored rubrics. Run it yourself on the 30-case open subset (Gemini judge, costs a few dollars per run); see benchmarks/donoharm/README.md for setup, the F1_weighted metric, and reference scores. The benchmarks/donoharm/submission/ validator checks your endpoint's response format for leaderboard submission (see API Request Format above).
• Script Concordance Test (SCT): probabilistic clinical reasoning under uncertainty. Run it yourself on the 174-item open subset with deterministic scoring (no LLM judge); see benchmarks/sct/README.md for setup, the sct_score metric, and reference scores.

Validation Results

All API responses are saved for auditability:

• test_XXX_response.json: Complete API response with metadata
• test_XXX_validation.json: Validation results and error details

Prerequisites

Python Dependencies

Install required packages:

pip install jsonschema requests

API Requirements

• Stable endpoint: API must remain accessible for at least 72 hours during benchmarking
• Concurrent requests: Must support 5-10 simultaneous connections
• Authentication: Bearer token authentication required
• Response time: Under 300 seconds per request
• Response format: Valid JSON: either {"response": "..."} or OpenAI-compatible chat completions format

Resource Requirements

Token and inference-cost estimates per benchmark, from a single GPT-5.5 reference run. Treat these as a rough guide only: your model's token counts and cost will differ, often substantially. Output tokens include reasoning tokens; both scale with reasoning effort and your provider's pricing.

Benchmark	Input tokens	Output tokens	Est. cost (GPT-5.5)
First Do NOHARM v2	0.7M	0.9M	$31
Script Concordance Test (SCT)	0.2M	0.2M	$6
CPC-Bench	5.6M	2.2M	$87
MedAgentBench v2	12.5M	0.4M	$74
PhysicianBench	36.6M	0.7M	$205
ReXrank Mini	32.9M	2.0M	$221
Multimodal Images	32.0M	1.0M	$191
Full suite	~121M	~7.4M	~$815

Agentic benchmarks (MedAgentBench, PhysicianBench) consume far more input tokens because each task spans many tool-use turns. Costs cover model inference only; LLM-judge scoring is run by the MAST team. PhysicianBench reflects the GPT-5.5 high-effort run.

File Formats

Input Files (.txt)

• Plain text clinical cases
• UTF-8 encoding
• One case per file

Response Schema

• JSON object with a response string field, or OpenAI-compatible chat completions format
• Must conform to benchmarks/donoharm/submission/schema.json (after extraction)
• Minimum 50 characters in the response field