Bot review

[LouisP96]

This PR adds functionality to more easily inspect and diagnose recent bot performance, specifically aimed at finding reasoning errors rather than technical issues.

Only a Metaculus token needed, no LLM (although I have also drafted a SKILL in the template bot repo to leverage these tools to pinpoint reasoning errors).

Specifically, bot-review review pulls scores from the Metaculus API, summarising performance and highlighting the best and worst performing questions (over the course of n days or some specified tournament).

E.g.

bot-review review --tournament minibench-2026-06-29 --top 3 \
    --output review.json --summary review.md

produces a json with the table of tournament questions and relevant forecast information, scores etc, and a summary in markdown:

Example summary output

# Review: MiniBench - 2026-06-29

user 303267 — generated 2026-07-28T16:13:00.800706+00:00

## Overall

- Rank: **17 / 134** (spot_peer_tournament)
- Score: **348.50**
- Medal: none
- Mean coverage: 0.60 (over 36 scored questions)

## Participation

- Questions: 37
- Forecasted: 37
- Scored: 36
- Forecasted but unscored: 1 (1 resolved without a score)
  - [What will SK hynix's closing share price (KRW) be on July 10, 2026?](https://www.metaculus.com/questions/44311/) — resolution: annulled
- Traced: 37 (0 ran only after spot scoring time)

### By tournament

- MiniBench - 2026-06-29: 37 questions (36 scored)

## Best 3 (by spot peer score)

1. +37.6 spot peer · +35.7 baseline · numeric · [What will be the total U.S. wildfire acres burned year-to-date reported by NIFC on July 9, 2026?](https://www.metaculus.com/questions/44332/)
2. +29.1 spot peer · +9.7 baseline · numeric · [How many cumulative confirmed Ebola deaths will the DRC have reported as of the July 2026 resolution date?](https://www.metaculus.com/questions/44329/)
3. +28.9 spot peer · +56.6 baseline · binary · [Will Russia announce an extension of its gasoline export ban beyond July 31, 2026, before July 10, 2026?](https://www.metaculus.com/questions/44337/)

## Worst 3 (by spot peer score)

1. -32.7 spot peer · -3.0 baseline · numeric · [What will the official confirmed death toll from the June 2026 Venezuela earthquakes be, as reported by July 10, 2026?](https://www.metaculus.com/questions/44328/)
2. -31.8 spot peer · -103.0 baseline · binary · [Will Justice Kavanaugh be the author of the majority opinion in NRSC v. FEC?](https://www.metaculus.com/questions/44326/)
3. -12.8 spot peer · -51.5 baseline · multiple_choice · [What drought level will the West Vancouver Island basin show on the B.C. Drought Information Portal as of July 10, 2026?](https://www.metaculus.com/questions/44339/)

Then bot-review show prints one section of one report in order to inspect bot reasoning, perhaps on the worst scoring questions. These are pulled from Metaculus comments.

E.g. bot-review show 44328 --section research:

# RESEARCH
## Report 1 Research
### News Articles (AskNews)
#### Intelligence Briefing: June 2026 Venezuela Earthquakes — Confirmed Death Toll by July 10, 2026

#### Bottom line / forecast target
The **official confirmed death toll** for the **24 June 2026 Venezuela earthquakes**, as reflected by the latest available reporting in the research set and consistent with the expected Wikipedia infobox value by **23:59 UTC on July 9, 2026**, is:

etc.

Or e.g. bot-review show 44328 --forecaster R1:F3 to show a single forecaster's reasoning:

## R1: Forecaster 3 Reasoning
Model: openrouter/anthropic/claude-opus-4.8

### PHASE 1: OUTSIDE VIEW

**(1) Source analysis and data anchor**

The most recent authoritative figure is **1,719 confirmed deaths**, announced by National Assembly President Jorge Rodríguez...

etc.

And I've drafted a skill which I imagine should form part of the template bot repo rather than in forecasting tools?

The skill

---
name: review-bot
description: Review how this forecasting bot actually performed on resolved Metaculus questions — build the outcome table, read the reports it posted, work out why the worst questions went wrong, and write a review. Use when asked to review bot performance, analyse scores, or work out why a forecast went wrong.
---

# Review this bot's forecasts

Diagnose *why* the bot scored the way it did on questions that have resolved. Everything here
is read-only Metaculus API traffic. No forecasting, no publishing, no API spend.

## Hard rules

- **Never run the bot.** `python main.py` spends money and publishes comments. This skill
  only reads.
- **Never read a whole report.** Pull one section of one question with `show`.
- **Ask before changing code.** Diagnose first, propose fixes in the review.
- Metaculus rate-limits at roughly 5 requests/second and the client does not retry. A 429
  means rerun the command.

## 1. Build the data

```bash
python -m forecasting_tools.bot_review.cli review --tournament <slug-or-id> \
    --output review.json --summary review-summary.md
```

Also `--post <POST_ID> ...` for specific questions, `--resolved-since <DAYS>` for anything
that resolved recently, `--from-json review.json` to re-render without refetching. Roughly one
request per post.

Read the summary first, then `review.json` for per-question detail.

## 2. Pick questions to investigate

Around ten, fewer once findings repeat. Most of the budget goes on the worst scores. Also:

- **Forecasted but unscored while resolved** — annulled, or something odder.
- **`traces` non-empty but `trace` null** — every run came after `spot_scoring_time`, so the
  forecast was never counted.
- **Fewer forecasters than the bot's usual** — compare `len(trace.forecasters)` across the
  table.
- **`trace.truncated` true** — the comment hit Metaculus's size cap and only its opening
  survives. Report it as a lost trace; there is nothing to diagnose.
- **No trace at all** — the bot never forecast, or never published a comment.

Those four are mechanical: no reading needed. When one of them matches many questions, that is
a single finding with a count, not a dozen questions to read.

Rank on whichever score the leaderboard uses — the summary names it. Peer scores are relative
to other forecasters, so negative means you did worse than the field, and the tournament score
is the sum.

Well-scored questions are not routine reading; they rarely produce a fix. Read one only to
test a hypothesis the losses raised, and only after checking whether the table can test it for
free — a claimed bias is usually answerable by comparing forecasts to resolutions across every
question. A list of losses looks like a systematic bias whether or not one exists, so test the
patterns you think you see.

## 3. Read the report, cheapest first

Stop as soon as the question is answered.

**a. The trace** — free, already in `review.json`. Every forecaster's prediction, the count,
the run time. A tight cluster that all missed points at a shared input or blind spot; a wide
spread that landed badly points at aggregation; one model alone and right is worth naming.

**b. Search the research** — near-free. You know what the question resolved to, so search for
it rather than reading:

```bash
python -m forecasting_tools.bot_review.cli show <POST_ID> --comment <COMMENT_ID> \
    --section research | grep -niE "<the deciding fact>" | head
```

Found → the research had it and the forecasters underused it. Absent → likely a research gap.

**c. One or two rationales** — the outlier, or one from the cluster if they all agreed.

```bash
python -m forecasting_tools.bot_review.cli show <POST_ID> --comment <COMMENT_ID> --forecaster R1:F3
```

**d. The whole research section** — last resort, when the searches came up empty and you need
to know what the bot actually had. A research section dwarfs a rationale, so these are the
reads to ration.

Take `<COMMENT_ID>` from the question's `trace.comment_id`: on a question forecast more than
once it names the run that was standing when the question was scored, which is the run whose
reasoning affected the score. Without it you get the latest run. Forecaster keys
(`R<report>:F<forecaster>`) and their predictions are in `trace.forecasters`; a key identifies
a forecaster across questions, so it is what to use when naming a repeat outlier.

Also worth asking: did the final prediction match where the reasoning pointed, and did the bot
answer the question the resolution criteria actually asked?

## 4. Write the review

Write `review.md`:

- **Summary** — score, rank, and the two or three things that actually moved it.
- **Per question** — title and link, score, and what went wrong in your own words, with the
  quote or absence from the report that shows it. One or two sentences each.
- **Patterns** — causes that recur, and any model repeatedly out on its own.
- **Suggestions** — concrete: a prompt change, a research gap, a scheduling fix. Name the
  questions each would have helped and drop anything you cannot tie to one.

"Well forecast, unlucky outcome" is a real finding. Do not manufacture a process failure for a
question the bot handled well. State plainly where the evidence is thin — three well-supported
findings beat ten guesses.

My example output would be:

Example review written by the skill

# Review — MiniBench 2026-06-29

Rank **17 / 134**, score **348.50** (spot peer, the plain sum of per-question scores).
37 questions forecast, 36 scored, mean coverage 0.60 (cosmetic here — MiniBench is spot
scored, so coverage does not affect the total).

Three things moved the score:

1. **Two questions cost 64 points between them** — Venezuela death toll (−32.7) and Kavanaugh
   authorship (−31.8), against a total of 348.5. In both, the research contained the deciding
   evidence and the forecasters discounted it.
2. **17 of 37 runs used 4 forecasters instead of 6.** A systematic run-level problem, not a
   forecasting one, and invisible in the scores alone.
3. **Aggregate calibration is fine.** The losses are specific, not systemic — see Patterns.

## Questions

### Venezuela earthquake death toll — [44328](https://www.metaculus.com/questions/44328/) · −32.7 · numeric

Resolved **3800**; the bot's median was **2806**. The research led with the official confirmed
toll of **1719** (Jorge Rodríguez, 29 June) and then reported, repeatedly, that 49,600–54,000
people were unaccounted for — the exact reason the confirmed count would keep climbing. All
four forecasters anchored near the confirmed figure and treated the missing-persons reporting
as background rather than as the driver of the eventual total.

Not a research gap. The information was there and was under-weighted.

### Kavanaugh authorship, NRSC v. FEC — [44326](https://www.metaculus.com/questions/44326/) · −31.8 · binary

Resolved **yes**; the ensemble said 11%, 12%, 17%, 25%, 25%, 29% — published at 21%. The
research stated plainly that multiple opinion-assignment analyses named **Kavanaugh as the
expected author of NRSC v. FEC**, with authorship-count reasoning behind it.

Every forecaster landed near the 1-in-9 base rate for "which justice writes this opinion"
while the research was handing them case-specific evidence. The tight cluster is the tell: six
models, all low, means a shared reasoning failure rather than an aggregation problem.

### Russia diesel export ban — [44303](https://www.metaculus.com/questions/44303/) · −12.0 · binary

Resolved **yes**; published 19.5%. Here the research was accurate as of the run: a partial ban
existed, Novak and Putin had said a full ban was under consideration, and the Ministry of
Energy had not recommended one. The bot correctly reported no formal adoption yet and priced
an 11-day window low.

Defensible at the time. The closest thing to genuine bad luck in this set, though it sits in
the same family as Kavanaugh — momentum evidence discounted in favour of the status quo.

### West Vancouver Island drought level — [44339](https://www.metaculus.com/questions/44339/) · −12.8 · multiple choice

Resolved **Level 2**; the bot put 8.5% on it, with 49% on Level 3 and 38% on Level 4. The
research showed Level 3 as of 16 June and emphasised deterioration — snowpack at 64% of
normal, 300+ wildfires. It carried nothing about the *recovery* that produced a downgrade.

The forecasters extrapolated a worsening trend and never priced improvement. Note this is the
one loss in the opposite direction to the others.

### SK hynix share price — [44311](https://www.metaculus.com/questions/44311/) · annulled

Forecast, resolved, no score. Nothing to fix.

## Patterns

**Evidence discounted in favour of priors.** Venezuela and Kavanaugh are the same failure: the
research supplied specific, decision-relevant information and the forecasters reasoned from a
generic prior instead. Together they cost roughly 18% of the tournament score. Russia is a
milder version.

**The forecasters were wrong together, not apart.** On Kavanaugh the spread was 11–29% — a
tight cluster around a wrong answer. Aggregation cannot rescue that; the fix has to be
upstream, in how the forecasters weigh case-specific evidence against base rates.

**Aggregate calibration is sound.** Tested across all 19 resolved binaries: mean p(yes) 0.354
against a 0.368 base rate, and the bot averaged 0.517 on questions that resolved yes versus
0.260 on those that resolved no. The "systematically under-predicts change" story visible in
the worst six is selection bias — that is what a list of losses looks like.

**Half the runs were degraded.** 17 of 37 ran with 4 forecasters against the usual 6, spanning
+37.6 to −32.7, so it is not obviously costly — but it is not intentional either, and it is
the single most actionable finding here.

## Suggestions

1. **Find out why 17 runs lost two forecasters.** The comments cannot say — the error list
   never reaches Metaculus. It needs the run logs or saved reports from that date. Highest
   value because it is a defect rather than a judgement call.
2. **Push the forecaster prompt on specific-evidence-versus-base-rate.** When research names a
   concrete expectation with reasoning behind it, that should move the forecast off the
   reference class. Would have helped 44326 directly and 44328 partially; worth checking
   against a question the bot won before changing anything.
3. **For numeric questions with a lagging official count, prompt for the trajectory.**
   Venezuela was a question about where a number would *reach*, and the forecasters priced
   where it *was*.
4. **Nothing to do about 44303** beyond noting it. A well-reasoned 19.5% that landed is not a
   process failure.

## Where the evidence is thin

Diagnoses rest on the research sections and the forecaster predictions. I did not read full
rationales, so "discounted the evidence" is inferred from the numbers plus what the research
contained, not from a model stating its reasoning. Confirming it needs one or two rationale
reads on 44326.

Please let me know any thoughts!