agent-ablation

Leave-one-out ablation testing, backward elimination, and ROI evaluation for multi-agent systems — available in both TypeScript / JavaScript and Python with zero runtime dependencies.

You have a set of per-agent findings (scores, confidences, telemetry) and a function or model that turns those findings into a verdict. agent-ablation answers the key production questions:

1. Load-Bearing Influence: Which agents' findings actually changed the verdict, and which were along for the ride?
2. Cost & Token ROI: How many dollars and tokens did each specialist burn per verdict flip? Does that small accuracy bump justify the API bill?
3. Correlated Agents & Pruning: Which redundant agents can be safely pruned via greedy backward elimination without breaking the final verdict?
4. Protective vs. Harmful Signals: When ground truth is provided, did an agent's presence prevent an error (protective), or did it cause a hallucination/false positive (harmful)?
5. Stochastic & Async LLM Judges: Handles async decisions and repeated sampling with majority voting to filter out LLM temperature variance.

---

📦 Installation

TypeScript / Node.js

npm install agent-ablation

Python

pip install agent-ablation

---

🚀 Quickstart Comparison

Python

from agent_ablation import Finding, run_ablation

def decide(findings):
    # Noisy-OR combination: 1 - Π(1 - score/100)
    survival = 1.0
    for f in findings:
        survival *= (1.0 - f.score / 100.0)
    risk = 1.0 - survival
    return "decline" if risk >= 0.7 else "approve" if risk <= 0.3 else "escalate"

findings = [
    Finding(agent_id="transaction_pattern", score=25),
    Finding(agent_id="identity_signal", score=90),
    Finding(agent_id="network_analysis", score=20),
]

result = run_ablation(findings, decide)

print("Baseline:", result.baseline)                  # "decline"
print("Load-Bearing Ratio:", result.load_bearing_ratio) # 0.33 (1 of 3 flipped verdict)
for p in result.per_agent:
    print(f"Without {p.removed_agent_id}: {p.verdict_without} {'(load-bearing)' if p.changed else ''}")

TypeScript

import { runAblation, type Finding } from "agent-ablation";

function decide(findings: Finding[]): "approve" | "decline" | "escalate" {
  const survival = findings.reduce((p, f) => p * (1 - f.score / 100), 1);
  const risk = 1 - survival;
  return risk >= 0.7 ? "decline" : risk <= 0.3 ? "approve" : "escalate";
}

const findings: Finding[] = [
  { agentId: "transaction_pattern", score: 25 },
  { agentId: "identity_signal", score: 90 },
  { agentId: "network_analysis", score: 20 },
];

const result = runAblation(findings, decide);

console.log("Baseline:", result.baseline);                  // "decline"
console.log("Load-Bearing Ratio:", result.load_bearing_ratio); // 0.33
for (const p of result.perAgent) {
  console.log(`Without ${p.removedAgentId}: ${p.verdictWithout} ${p.changed ? "(load-bearing)" : ""}`);
}

---

📊 Core Features

1. Cost & Token ROI Analysis ("Cost per Verdict Flip")

Pass telemetry (cost, tokens, latencyMs / latency_ms) inside findings. batchAblation / batch_ablation computes exact ROI metrics and generates pruning advice:

# Python
from agent_ablation import Finding, batch_ablation, format_markdown_report

cases = [
    [
        Finding(agent_id="expensive_reasoner", score=90, cost=0.15, tokens=3000),
        Finding(agent_id="cheap_heuristic", score=10, cost=0.002, tokens=50),
    ],
    [
        Finding(agent_id="expensive_reasoner", score=20, cost=0.15, tokens=3000),
        Finding(agent_id="cheap_heuristic", score=85, cost=0.002, tokens=50),
    ],
]

results, summary = batch_ablation(cases, decide)
print(format_markdown_report(summary))

---

2. Async & Stochastic Decision Functions (LLM-as-a-Judge)

When decisions involve async LLM calls with temperature, use run_ablation_async / runAblationAsync. Set samples=5 to apply majority-vote consensus:

# Python
result = await run_ablation_async(
    findings,
    async_llm_judge,
    samples=5,  # 5 samples per decision to filter temperature noise
)

---

3. Greedy Backward Elimination & Minimal Viable Panel

Identify the minimal viable panel by iteratively pruning non-essential agents until removing any further agent flips the verdict:

# Python
from agent_ablation import run_backward_elimination

result = run_backward_elimination(all_specialists, decide)
print(result.minimal_agent_ids)   # e.g. ["critic_1", "security_auditor"]
print(result.eliminated_agent_ids) # e.g. ["critic_2", "critic_3", "scout_noisy"]

For detecting 2nd-order joint dependencies, run_pairwise_ablation / runPairwiseAblation tests all pairs $(A, B)$ where neither agent alone is load-bearing, but removing both together flips the outcome.

---

4. Ground-Truth & Net Accuracy Impact ("Protective vs. Harmful")

Supply ground-truth labels in batch_ablation / batchAblation to measure whether an agent's load-bearing presence improved accuracy or caused hallucinations:

# Python
results, summary = batch_ablation(cases, decide, ground_truth=["approve", "decline"])

print(summary.per_agent_stats["hallucinating_agent"].role)               # "Harmful"
print(summary.per_agent_stats["hallucinating_agent"].net_accuracy_impact) # -0.25

---

🔌 Framework Adapters

Zero-dependency adapters to map telemetry and messages from popular frameworks directly into Finding:

Framework	TypeScript	Python
LangGraph / LangChain	fromLangGraphMessages(state.messages, { ... })	from_langgraph_messages(state["messages"], score_of=...)
CrewAI	fromCrewAITasks(crewOutput.tasks_output, { ... })	from_crewai_tasks(crew_output.tasks_output, score_of=...)
AutoGen	fromAutoGenMessages(chatHistory, { ... })	from_autogen_messages(chat_history, score_of=...)
Vercel AI SDK / Steps	fromAISDKSteps(steps, { ... })	from_ai_sdk_steps(steps, score_of=...)
Generic Records	fromRecords(customRecords, { ... })	from_records(custom_records, agent_id=..., score_of=...)

---

🧪 SentryMesh 33% Multi-Signal Replication

SentryMesh is a four-specialist multi-agent fraud investigation system. Its eval harness runs an ablation over its 23-case bank and reports: of 9 cases auto-resolved without human escalation, only 3 survive removal of their single loudest specialist — 6 collapse to escalate.

Both the TypeScript and Python test suites reproduce all 6 cases verbatim.

---

📚 API Reference

Feature	TypeScript API	Python API
Single Case Ablation	runAblation(findings, decide, equals?)	run_ablation(findings, decide, equals=None)
Async / Stochastic Ablation	runAblationAsync(findings, decide, options?)	run_ablation_async(findings, decide, ...)
Batch Ablation + ROI	batchAblation(cases, decide, options?)	batch_ablation(cases, decide, ...)
Async Batch Ablation	batchAblationAsync(cases, decide, options?)	batch_ablation_async(cases, decide, ...)
Backward Elimination	runBackwardElimination(findings, decide, ...)	run_backward_elimination(findings, decide, ...)
Pairwise Ablation	runPairwiseAblation(findings, decide, ...)	run_pairwise_ablation(findings, decide, ...)
Markdown Reporter	formatMarkdownReport(summary, options?)	format_markdown_report(summary, ...)
ASCII Reporter	formatAsciiTable(summary)	format_ascii_table(summary)

---

📄 License

MIT © Ayush Verma — [email protected]