PPI Advisor

An AI-powered agentic system that helps researchers — especially clinicians and applied scientists without a statistics background — determine which Prediction-Powered Inference (PPI) method is appropriate for their study.

Built on the decision framework from:

"Demystifying Prediction-Powered Inference" Song, Kluger, Parikh, & Gu (2026). arXiv:2601.20819v2

---

Table of Contents

• What is PPI?
• What This System Does
• Prerequisites
• Setup — Windows
• Setup — macOS / Linux
• Running the App
• Using the App
• Project Structure
• Architecture Details
• PPI Methods Covered
• Troubleshooting

---

What is PPI?

Prediction-Powered Inference is a statistical framework for studies where:

• A gold-standard outcome (Y) is expensive, invasive, or logistically burdensome to measure for every individual (e.g., expert chart review, imaging phenotypes, field surveys).
• A machine learning model can predict Y (producing Ŷ) for a much larger sample using readily available covariates.
• A small labeled subset has both the true outcome Y and the prediction Ŷ; a large unlabeled subset has only Ŷ.

PPI uses the ML predictions to boost statistical efficiency (tighter confidence intervals) while explicitly correcting for prediction error using the labeled subset. The key insight: inferences remain valid regardless of prediction quality — poor predictions merely fail to improve efficiency rather than compromising validity.

What This System Does

The PPI Advisor guides users through a 3-step workflow:

1. Understand Your Data — Upload a CSV file, describe your study goal. The system helps you identify which columns are the outcome (Y), predictions (Ŷ), label indicator (S), and covariates (X).

2. Verify Assumptions — The system automatically runs statistical diagnostics to check PPI's three core assumptions:

   • A1 (Distribution Comparability): Are your labeled and unlabeled samples from the same population? Tested via Standardized Mean Differences, Kolmogorov-Smirnov tests, and energy distance.
   • A2 (Model Independence): Was your prediction model trained on data completely separate from your inference data? Assessed through structured questions about model provenance (this cannot be tested statistically).
   • A3 (Complete Covariates): Are all covariates fully observed for every unit? Tested via missingness pattern analysis.

3. Recommend the Right PPI Method — Based on which assumptions hold or are violated, the system recommends the most appropriate PPI variant(s) following the paper's decision flowchart (Figure 2), with plain-language explanations.

---

Prerequisites

• Python 3.13+ (the project was developed with Python 3.13)
• uv — a fast Python package manager (install instructions)
• OpenAI API key — the system uses GPT models for the agentic reasoning

---

Setup — Windows

1. Install uv (if not already installed)

Open PowerShell and run:

pip install uv

Or use the official installer:

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

2. Clone or download this project

Place the project folder wherever you like. For example:

F:\Projects\ppi-advisor\

3. Create the virtual environment and install dependencies

Open a terminal in the project folder:

cd "F:\Projects\ppi-advisor"
uv venv
uv sync

This reads pyproject.toml and uv.lock to install exact dependency versions.

4. Set your OpenAI API key

Option A — Create a .env file in the project root:

Copy-Item .env.example .env

Then edit .env and replace the placeholder:

OPENAI_API_KEY=sk-your-actual-key-here

Option B — Set it as an environment variable:

$env:OPENAI_API_KEY = "sk-your-actual-key-here"

5. Generate the sample dataset (optional)

A sample clinical study CSV is included, but you can regenerate it:

uv run python generate_sample_data.py

6. Launch the app

uv run streamlit run app.py

The app opens in your browser at http://localhost:8501.

---

Setup — macOS / Linux

1. Install uv (if not already installed)

# macOS (Homebrew)
brew install uv

# macOS / Linux (official installer)
curl -LsSf https://astral.sh/uv/install.sh | sh

After installing, restart your terminal or run source ~/.bashrc / source ~/.zshrc.

2. Clone or download this project

cd ~/Projects
# If using git:
git clone <repo-url> ppi-advisor
cd ppi-advisor

# Or just unzip / copy the folder and cd into it

3. Create the virtual environment and install dependencies

uv venv
uv sync

If you see an error about Python 3.13 not being found, install it via uv:

uv python install 3.13
uv venv
uv sync

4. Set your OpenAI API key

Option A — Create a .env file:

cp .env.example .env

Edit it with your favorite editor:

nano .env
# or
open .env    # macOS

Set:

OPENAI_API_KEY=sk-your-actual-key-here

Option B — Export as environment variable:

export OPENAI_API_KEY="sk-your-actual-key-here"

To make it permanent, add the export line to your ~/.zshrc or ~/.bashrc.

5. Generate the sample dataset (optional)

uv run python generate_sample_data.py

6. Launch the app

uv run streamlit run app.py

The app opens in your browser at http://localhost:8501.

---

Running the App

Once launched, the Streamlit UI provides:

• Sidebar — File upload, assumption status tracker, quick-start prompts, settings, activity log
• Chat interface — Describe your study in plain language; the agent walks you through the workflow
• Agent working steps — Expandable panel showing what the agent did behind the scenes

Quick test with sample data

The sample dataset (sample_clinical_study.csv) is auto-detected. Click the first quick-start prompt in the sidebar to run the full end-to-end workflow. It will:

1. Load and describe the dataset
2. Verify all three PPI assumptions
3. Recommend PPI++ as the appropriate method (since all assumptions are satisfied in the sample data)

Using your own data

1. Upload your CSV via the sidebar file uploader

2. Tell the agent about your study — for example:

   "I want to estimate regression coefficients for the association between patient age, BMI, and blood pressure with a disease severity score. The outcome column is severity_score, predictions are in predicted_score, and has_label indicates which patients have gold-standard measurements. The ML model was trained at a different hospital with no patient overlap."

3. The agent will guide you from there, asking follow-up questions if needed.

What information the agent will ask for

Question	What to provide	Example answer
Analysis goal	What you want to estimate	"Regression coefficients" or "Population mean"
Column roles	Which columns are Y, Ŷ, S, and X	"Outcome is score, prediction is pred_score, label is is_labeled, covariates are age, bmi, bp"
Model provenance	How the ML model was trained	"Trained on external hospital data, no overlap" or "Trained on our own cohort" or "Commercial model, training data unknown"

---

Project Structure

ppi-advisor/
│
├── app.py                          # Streamlit frontend (chat UI)
├── config.py                       # Centralized paths (UPLOAD_DIR, LOG_DIR)
├── llm_client.py                   # OpenAI API client with function-calling
├── generate_sample_data.py         # Generates sample_clinical_study.csv
│
├── agents/                         # Multi-agent system
│   ├── __init__.py                 # Exports OrchestratorAgent
│   ├── base.py                     # Base agent class with tool-calling loop
│   ├── orchestrator.py             # Main coordinator — routes to specialists
│   ├── data_agent.py               # Data intake specialist
│   ├── diagnostics_agent.py        # Assumption verification specialist
│   └── recommendation_agent.py     # PPI method recommendation specialist
│
├── tools/                          # Tool implementations and schemas
│   ├── __init__.py
│   ├── data_tools.py               # load_and_describe_data, validate_ppi_structure, etc.
│   ├── assumption_tools.py         # assess_assumption_a1/a2/a3 (SMD, KS, energy distance)
│   ├── recommendation.py           # Decision flowchart logic (Figure 2 from the paper)
│   └── dispatch.py                 # Tool schemas (OpenAI format) and dispatcher
│
├── uploads/                        # User-uploaded CSV files
│   └── sample_clinical_study.csv   # Pre-generated sample dataset
├── logs/                           # Agent activity logs
│
├── pyproject.toml                  # Project metadata and dependencies
├── uv.lock                         # Locked dependency versions
├── .env.example                    # Template for API key
├── .gitignore
├── .python-version                 # Python 3.13
└── 2601.20819v2.pdf                # The PPI paper

---

Architecture Details

Multi-Agent System

The system follows a hierarchical multi-agent pattern using OpenAI function calling:

User (Streamlit chat)
        │
        ▼
┌──────────────────┐
│   Orchestrator    │   Coordinates workflow, delegates tasks
│     Agent         │   Has 3 delegation tools
└────────┬─────────┘
         │
    ┌────┼──────────────┐
    ▼    ▼              ▼
 Data   Diagnostics   Recommendation
 Agent  Agent         Agent
   │       │              │
   ▼       ▼              ▼
 3 tools  4 tools       1 tool

Agent Loop

Each agent follows the same loop (defined in agents/base.py):

1. Receive user/delegated message
2. Send to LLM with available tool schemas
3. If LLM requests tool calls → execute tools, append results, loop back to step 2
4. If LLM returns text → return final answer
5. Maximum 12 rounds to prevent infinite loops

Tools (7 total)

Tool	Purpose
load_and_describe_data	Load CSV, return shape, column types, statistics, sample rows
get_column_statistics	Detailed statistics for a single column
validate_ppi_structure	Check that Y, Ŷ, S, X columns are correct
assess_assumption_a1	SMD, KS tests (Bonferroni-corrected), energy distance
assess_assumption_a2	Process model provenance answers
assess_assumption_a3	Missingness rates, patterns, differential missingness
get_ppi_recommendation	Decision flowchart → recommend PPI variant(s)

---

PPI Methods Covered

Based on the paper's Table 3 and Figure 2:

Method	When to Use	Key Idea
PPI	All assumptions satisfied	Original estimator: predictions + bias correction
PPI++	All assumptions satisfied	Adds tuning parameter λ for guaranteed efficiency
Re-PPI	All assumptions satisfied	Optimal within the PPI class via function-valued tuning
Cross-PPI	A2 violated (no external model)	K-fold cross-fitting on internal data
Cross-PPBoot	A2 violated	Cross-fitting + bootstrap confidence intervals
Tuned-CPPI	A2 violated	Cross-fitting + adaptive tuning
MAR-S	A1 violated (non-random labeling)	Semiparametric/doubly robust under MAR
MAR-SS	A1 violated	Estimated propensity scores, decaying overlap
IPI	A3 violated (missing covariates)	Imputation-powered estimating equations
Predict-then-debias	A3 violated	ML-predicted covariates + bootstrap CIs
Chen et al.	Multiple violations	General missingness patterns under MAR

---

Troubleshooting

"Port 8501 is not available"

Another Streamlit instance is running. Either use a different port:

uv run streamlit run app.py --server.port 8510

Or stop the existing process:

# macOS / Linux
pkill -f streamlit

# Windows PowerShell
Stop-Process -Name "streamlit" -Force

"File not found: sample_clinical_study.csv"

Regenerate the sample data:

uv run python generate_sample_data.py

"OPENAI_API_KEY not set" or authentication errors

Make sure your .env file exists in the project root with a valid key:

OPENAI_API_KEY=sk-your-actual-key-here

Or set it as an environment variable before launching.

Python version issues on macOS

If uv sync fails because Python 3.13 isn't found:

uv python install 3.13
uv venv --python 3.13
uv sync

"ModuleNotFoundError" when running

Make sure you're using the virtual environment:

# Use uv run to automatically activate the venv
uv run streamlit run app.py

# Or activate manually first
# macOS / Linux:
source .venv/bin/activate
streamlit run app.py

# Windows:
.venv\Scripts\activate
streamlit run app.py

---

References

• Angelopoulos et al. (2023). Prediction-Powered Inference. Science, 382(6671).
• Angelopoulos, Duchi, & Zrnic (2023). PPI++: Efficient Prediction-Powered Inference.
• Zrnic & Candès (2024). Cross-Prediction-Powered Inference. PNAS, 121(15).
• Song, Kluger, Parikh, & Gu (2026). Demystifying Prediction-Powered Inference. arXiv:2601.20819v2.

See the full reference list in the paper (2601.20819v2.pdf).