diff --git a/README.md b/README.md index f1a1632..e287d80 100644 --- a/README.md +++ b/README.md @@ -22,6 +22,7 @@ notebooks stay short and readable. | [`notebooks/01_historical_archive.ipynb`](notebooks/01_historical_archive.ipynb) | Download, cache, and load the historical archive, then reproduce the baseline scoring regressions (Koijen & Levy WP, Table 3) with the competition's exact scoring transform | | [`notebooks/02_earnings_call_facts.ipynb`](notebooks/02_earnings_call_facts.ipynb) | Where each event's ten-fact `disclosure` comes from — the exact prompt, model config, and parsing behind it, and how extraction failures surface | | [`notebooks/03_earnings_previews.ipynb`](notebooks/03_earnings_previews.ipynb) | The **earnings preview** — the result of running [Anthropic's Earnings Preview Skill](https://github.com/anthropics/financial-services/blob/main/plugins/agent-plugins/earnings-reviewer/skills/earnings-preview/SKILL.md) prior to each earnings announcement. Starting Q4 2026, events carry this as a second disclosure item. The notebook covers how it is built, what it looks like, and how it pairs with the facts | +| [`notebooks/04_options_market_data.ipynb`](notebooks/04_options_market_data.ipynb) | The **option-implied statistics** — what the options market priced for each release, why the scorer is blind to a uniform rescaling of predictions, and how the implied move works as an *event-specific* scale for a prediction. Reproduces the effect on the two reference baselines with the exact scoring transform | ### The option-implied statistics @@ -34,11 +35,11 @@ stats = option_implied_stats_from_disclosure(record) # dict, or None wh move = statistic_value(stats, "implied_absolute_earnings_move") # float, or None unless status is "ok" ``` -`examples.option_stats` documents what the three numbers mean and when they are missing. There is no notebook for it yet, and the bundled 2026Q3 sample predates the item, so run against the live archive to see it. +`examples.option_stats` documents what the three numbers mean and when they are missing, and `scale_by_implied_move` turns the move into an event-specific rescaling of a submission's predictions. Notebook 04 walks through both; its regression sections need the live archive (the bundled sample carries the item on its six events but no baseline predictions). If you parse bundles with your own typed model, make sure an item's `content` accepts any JSON type. A model that allows only strings and lists of strings rejects the whole bundle, facts included, the first time it meets this item. `examples.schemas.DisclosureItem` had exactly that annotation until this release. -All four run top-to-bottom. With an API key they hit the live API; **without one they +All five run top-to-bottom. With an API key they hit the live API; **without one they fall back to a small bundled sample** (`data/sample/`), so you can run them — and CI can execute them — with no credentials. diff --git a/data/README.md b/data/README.md index 0528569..000a7cf 100644 --- a/data/README.md +++ b/data/README.md @@ -18,12 +18,16 @@ Just enough data for the notebooks to run top-to-bottom **without an API key** lines may include additional fields. Treat the API as the source of truth. - `archive_EARNINGS_RELEASE_2026Q3.jsonl.gz` — six **real** archive lines (NVDA, WMT, HD, CRM, AVGO, ORCL), copied verbatim from the live 2026Q3 file on - 2026-09-24. Each carries both disclosure items — the ten facts and the - agent-written **earnings preview** — plus the realized `event_returns`, + 2026-09-24. Each carries all three disclosure items — the ten facts, the + agent-written **earnings preview**, and the **option-implied statistics** (the + last spliced in from the same events' archive lines on 2026-09-29, ahead of its + dissemination on the public archive) — plus the realized `event_returns`, `metrics`, and `knowledge_cutoff` the archive adds once an event is scored, so - `notebooks/03_earnings_previews.ipynb` and the preview section of notebook 01 - run offline on genuine content. 2026Q3 was unsealed when this was pulled, so - fields may later change upstream; the API remains the source of truth. + `notebooks/03_earnings_previews.ipynb`, `notebooks/04_options_market_data.ipynb` + and the preview section of notebook 01 run offline on genuine content. The + lines' `baseline_predictions` are empty, so notebook 04's regression sections + skip in sample mode. 2026Q3 was unsealed when this was pulled, so fields may + later change upstream; the API remains the source of truth. - `transcript_sample.json` — a **synthetic** earnings-call transcript for a fictional company (Northwind Logistics). Real transcripts are licensed and are not distributed with this repo; this one exists so diff --git a/data/sample/archive_EARNINGS_RELEASE_2026Q3.jsonl.gz b/data/sample/archive_EARNINGS_RELEASE_2026Q3.jsonl.gz index 88774e4..a6ac1c6 100644 Binary files a/data/sample/archive_EARNINGS_RELEASE_2026Q3.jsonl.gz and b/data/sample/archive_EARNINGS_RELEASE_2026Q3.jsonl.gz differ diff --git a/notebooks/01_historical_archive.ipynb b/notebooks/01_historical_archive.ipynb index 6434f98..e21d16c 100644 --- a/notebooks/01_historical_archive.ipynb +++ b/notebooks/01_historical_archive.ipynb @@ -323,7 +323,7 @@ "2. Convert that reaction into a percentile rank in `[0, 1]` **across the quarter's cross-section** of scored event outcomes \u2014 a rank against all the period's events, not against the asset's own history. That's the target the competition scores; `examples.scoring.percentile_ranks` is the exact transform (see section 6).\n", "3. Score your model's predicted percentile against it offline, and iterate.\n", "\n", - "For the ex ante side of each event \u2014 the earnings preview \u2014 see [`03_earnings_previews.ipynb`](03_earnings_previews.ipynb).\n", + "For the ex ante side of each event \u2014 the earnings preview \u2014 see [`03_earnings_previews.ipynb`](03_earnings_previews.ipynb). For what the options market priced for each event, and why only an *event-specific* rescaling can move the score, see [`04_options_market_data.ipynb`](04_options_market_data.ipynb).\n", "\n", "When your model is ready, move it into a deployed webhook handler (see the starter repos) \u2014 that's what receives live events and submits predictions. These notebooks deliberately stop short of `POST /predictions`." ] diff --git a/notebooks/03_earnings_previews.ipynb b/notebooks/03_earnings_previews.ipynb index aa5ecb7..0098654 100644 --- a/notebooks/03_earnings_previews.ipynb +++ b/notebooks/03_earnings_previews.ipynb @@ -327,7 +327,7 @@ "- The preview is **optional** and arrived in 2026Q3. Match on `id` / `kind`, never on position, and build for its absence.\n", "- It is a **public-sources research note with a fixed brief** \u2014 Anthropic's open `earnings-preview` skill, run by Claude in a sandboxed Claude Code session \u2014 not a model prediction and not a proprietary feed. Its sources and dates are cited inline.\n", "- Going forward it is assembled **before the CAR1 window opens**; the 2026Q3 archive previews are not guaranteed to be, so treat them as illustrative.\n", - "- The facts (notebook 02) are the ex post disclosure; the preview is the ex ante expectation; `car1` is the reaction. `examples.scoring` turns the last into the percentile target the competition scores." + "- The facts (notebook 02) are the ex post disclosure; the preview is the ex ante expectation; `car1` is the reaction. `examples.scoring` turns the last into the percentile target the competition scores.\n- The third item, the **option-implied statistics**, is the subject of [`04_options_market_data.ipynb`](04_options_market_data.ipynb)." ] } ], diff --git a/notebooks/04_options_market_data.ipynb b/notebooks/04_options_market_data.ipynb new file mode 100644 index 0000000..fb4644d --- /dev/null +++ b/notebooks/04_options_market_data.ipynb @@ -0,0 +1,504 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "a20b0ee9", + "metadata": {}, + "source": [ + "# 04 — Options market data: what the options market expected\n", + "\n", + "From 2026Q3 an event may carry a third disclosure item, the **option-implied statistics**: three numbers read off the stock's listed options just before the release. They say how large a move the options market priced, not which way. The item is `id=\"option-implied-stats\"`, `kind=\"stats\"`, `source=\"option_market\"`, with `content` an **object**. It is optional: fewer than half of events have options liquid enough to measure, and no event before 2026Q3 carries it.\n", + "\n", + "This notebook reads the item, then uses it as an **event-specific scale** for a prediction. The score ignores any uniform rescaling of your predictions, so only something that varies across events can move it. We test that on the two reference baselines with the exact scoring transform.\n", + "\n", + "1. What you receive — coverage in the 2026Q3 archive\n", + "2. The three numbers, on six events\n", + "3. Magnitude, not direction\n", + "4. Why the score needs an event-specific transform\n", + "5. Two rescalings: naive and adjusted\n", + "6. The regression table\n", + "7. The competition's own metric\n", + "8. Caveats\n", + "\n", + "With a live API key it reads the real 2026Q3 archive; without one, the six real archive lines in `data/sample/`, which carry the item. Sections 3 and 5–7 need the whole quarter and skip in sample mode." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a5cb3fd2", + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "from pathlib import Path\n", + "\n", + "import matplotlib.pyplot as plt\n", + "import pandas as pd\n", + "from IPython.display import display\n", + "\n", + "from examples import Client, load_config\n", + "from examples.archive import download_archive, read_jsonl_gz\n", + "from examples.option_stats import (\n", + " STATISTICS,\n", + " option_implied_stats_from_disclosure,\n", + " option_stats_frame,\n", + " scale_by_implied_move,\n", + " statistic_value,\n", + ")\n", + "from examples.scoring import (\n", + " BASELINE_LABELS,\n", + " add_percentiles,\n", + " outcomes_frame,\n", + " percentile_ranks,\n", + " score_submission,\n", + ")\n", + "\n", + "REPO = next(p for p in [Path.cwd(), *Path.cwd().parents] if (p / \"data\" / \"sample\").is_dir())\n", + "ARCHIVE_DIR = REPO / \"data\" / \"archive\" # gitignored download cache\n", + "SAMPLE_DIR = REPO / \"data\" / \"sample\"\n", + "\n", + "QUARTER = \"2026Q3\" # the first quarter with option-implied statistics\n", + "TICKERS = [\"NVDA\", \"WMT\", \"HD\", \"CRM\", \"AVGO\", \"ORCL\"] # notebook 03's six, also bundled\n", + "MOVE = \"implied_absolute_earnings_move\"\n", + "REFRESH = False # True re-downloads the quarter even when a copy is cached (section 1)\n", + "MIN_EVENTS = 100 # fewest covered events worth regressing on\n", + "\n", + "config = load_config()\n", + "print(\"Mode:\", \"LIVE\" if config.is_live else \"SAMPLE (no EM_API_KEY — using bundled data)\")" + ] + }, + { + "cell_type": "markdown", + "id": "e17dc48f", + "metadata": {}, + "source": [ + "## 1. What you receive\n", + "\n", + "`option_stats_frame` gives one row per event: the three statistics as float columns (`NaN` when unavailable), `as_of`, `methodology`, and `skew_status`, which says why the skew is missing when the volatility is present.\n", + "\n", + "The live branch downloads only when the quarter is not cached or `REFRESH = True`. The quarter is unsealed, so a cached copy goes stale; a covered count of zero means your copy predates the item." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8aa0efbf", + "metadata": {}, + "outputs": [], + "source": [ + "path = ARCHIVE_DIR / f\"EARNINGS_RELEASE_{QUARTER}.jsonl.gz\"\n", + "if not config.is_live:\n", + " path = SAMPLE_DIR / f\"archive_EARNINGS_RELEASE_{QUARTER}.jsonl.gz\"\n", + " print(f\"Sample mode: reading {path.name}\")\n", + "elif REFRESH or not path.exists():\n", + " with Client.from_env() as client:\n", + " manifest = client.archive_manifest()\n", + " (path,) = download_archive(\n", + " manifest,\n", + " ARCHIVE_DIR,\n", + " client=client,\n", + " only=lambda f: f.event_type == \"EARNINGS_RELEASE\" and f.quarter == QUARTER,\n", + " )\n", + "else:\n", + " print(f\"Using the cached {path.name}; set REFRESH = True to re-download\")\n", + "\n", + "records = list(read_jsonl_gz(path))\n", + "stats = option_stats_frame(records)\n", + "covered = stats[stats[\"has_option_stats\"]]\n", + "n_covered = len(covered)\n", + "\n", + "print(f\"{len(records):,} events in {QUARTER}; {n_covered:,} carry option-implied statistics\")\n", + "if n_covered:\n", + " print(f\" share covered : {n_covered / len(records):.0%}\")\n", + " print(f\" methodology : {sorted(covered['methodology'].dropna().unique())}\")\n", + " print(f\" skew available : {covered['skew_25_delta'].notna().mean():.0%} of covered\")\n", + " print(f\" as_of before cutoff : {bool((covered['as_of'] < covered['knowledge_cutoff']).all())}\")\n", + "else:\n", + " print(\" (none: this copy of the archive predates the item's dissemination)\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0e1d393c", + "metadata": {}, + "outputs": [], + "source": [ + "outcomes = outcomes_frame(records) # one row per (event, asset) that has a car1\n", + "models = list(BASELINE_LABELS.values()) # [\"Gemini 2.5 Flash-Lite\", \"GPT-5 nano\"]\n", + "\n", + "have_scoring_data = (\n", + " n_covered >= MIN_EVENTS\n", + " and not outcomes.empty\n", + " and set(models) <= set(outcomes.columns)\n", + " and int(outcomes[\"surprise\"].notna().sum()) >= MIN_EVENTS\n", + ")\n", + "if have_scoring_data:\n", + " print(\n", + " f\"Regression sections on: {len(outcomes):,} scored outcomes, {n_covered:,} covered events\"\n", + " )\n", + "elif not config.is_live:\n", + " print(\"Sample mode: sections 3 and 5 to 7 need the live archive. Skipping.\")\n", + "else:\n", + " print(\n", + " f\"Live archive, but not enough to regress on ({n_covered:,} covered events; baseline \"\n", + " f\"predictions {'present' if set(models) <= set(outcomes.columns) else 'absent'}). \"\n", + " \"Sections 3 and 5 to 7 skip.\"\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "383cefc6", + "metadata": {}, + "source": [ + "## 2. The three numbers, on six events\n", + "\n", + "Each statistic is a `{value, status}` pair; `value` is a decimal (`0.0676` is 6.76%) and `null` unless `status` is `\"ok\"`.\n", + "\n", + "- `implied_earnings_volatility` — the risk-neutral standard deviation of the jump on the release, backed out of at-the-money implied volatility on the first two expirations after it (Dubinsky, Johannes, Kaeck and Seeger, *RFS* 2019).\n", + "- `implied_absolute_earnings_move` — the risk-neutral expected absolute return from that jump, `4·Φ(J/2) − 2 ≈ 0.8·J`. Same information, in return units.\n", + "- `skew_25_delta` — 25-delta call minus 25-delta put implied volatility on the nearest expiration. Negative means puts are richer than calls. Missing on its own when the wings are thin.\n", + "\n", + "`as_of` always precedes `knowledge_cutoff`, so the item is a legitimate ex ante input. Below: the raw object for one of notebook 03's six events, then all six." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a929e19d", + "metadata": {}, + "outputs": [], + "source": [ + "def latest_record(ticker: str) -> dict | None:\n", + " hits = [r for r in records if r[\"focal_assets\"][0][\"identifier_value\"] == ticker]\n", + " return max(hits, key=lambda r: r[\"event_datetime\"]) if hits else None\n", + "\n", + "\n", + "chosen = {t: r for t in TICKERS if (r := latest_record(t)) is not None}\n", + "if not chosen: # a different quarter, say: fall back to the first six covered events\n", + " ids = set(covered[\"event_id\"].head(6))\n", + " chosen = {r[\"focal_assets\"][0][\"identifier_value\"]: r for r in records if r[\"event_id\"] in ids}\n", + "\n", + "first = next(iter(chosen.values()))\n", + "raw = option_implied_stats_from_disclosure(first)\n", + "print(f\"{first['event_id']} — the item's content object:\")\n", + "print(json.dumps(raw, indent=2))\n", + "print()\n", + "for name in STATISTICS:\n", + " print(f\"{name:32s} {statistic_value(raw, name)}\") # None unless status is 'ok'" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "20127bff", + "metadata": {}, + "outputs": [], + "source": [ + "six = option_stats_frame(chosen.values()).set_index(\"tickers\").loc[list(chosen)]\n", + "six[\"move / vol\"] = six[MOVE] / six[\"implied_earnings_volatility\"]\n", + "display(\n", + " six[[\"as_of\", \"knowledge_cutoff\", *STATISTICS, \"skew_status\", \"move / vol\"]].style.format(\n", + " {\n", + " \"as_of\": \"{:%Y-%m-%d %H:%M}\",\n", + " \"knowledge_cutoff\": \"{:%Y-%m-%d %H:%M}\",\n", + " \"implied_earnings_volatility\": \"{:.2%}\",\n", + " MOVE: \"{:.2%}\",\n", + " \"skew_25_delta\": \"{:+.2%}\",\n", + " \"move / vol\": \"{:.3f}\",\n", + " },\n", + " na_rep=\"\\u2014\",\n", + " )\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "d8b9d51a", + "metadata": {}, + "source": [ + "## 3. Magnitude, not direction\n", + "\n", + "The move is about the size of the reaction, not its sign, and it is risk-neutral rather than an unbiased forecast. The skew is an asymmetry in option prices, not a predicted return. Over the quarter the move correlates strongly with the *absolute* realized abnormal return; the skew's correlation with the *signed* return is nil." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "33ee8cf2", + "metadata": {}, + "outputs": [], + "source": [ + "joined = outcomes.merge(stats[[\"event_id\", *STATISTICS]], on=\"event_id\", how=\"left\")\n", + "\n", + "six_ids = {r[\"event_id\"] for r in chosen.values()}\n", + "display(\n", + " joined[joined[\"event_id\"].isin(six_ids)]\n", + " .set_index(\"identifier_value\")[[\"car1\", MOVE, \"skew_25_delta\"]]\n", + " .reindex(list(chosen)) # a very recent live event may have no car1 yet\n", + " .rename(columns={MOVE: \"implied move\", \"skew_25_delta\": \"skew\"})\n", + " .style.format({\"car1\": \"{:+.2%}\", \"implied move\": \"{:.2%}\", \"skew\": \"{:+.2%}\"}, na_rep=\"\\u2014\")\n", + ")\n", + "\n", + "if have_scoring_data:\n", + " cov = joined.dropna(subset=[\"car1\", MOVE])\n", + " with_skew = cov.dropna(subset=[\"skew_25_delta\"])\n", + " print(f\"Covered outcomes: {len(cov):,}\")\n", + " print(f\" corr(|car1|, implied move) = {cov['car1'].abs().corr(cov[MOVE]):+.2f}\")\n", + " print(f\" corr(car1, implied move) = {cov['car1'].corr(cov[MOVE]):+.2f}\")\n", + " print(\n", + " f\" corr(car1, skew) = {with_skew['car1'].corr(with_skew['skew_25_delta']):+.2f}\"\n", + " )\n", + " print(\n", + " f\" |car1| above implied move : {(cov['car1'].abs() > cov[MOVE]).mean():.0%} of outcomes\"\n", + " )\n", + "\n", + " fig, ax = plt.subplots(figsize=(6.5, 4))\n", + " ax.scatter(cov[MOVE] * 100, cov[\"car1\"].abs() * 100, s=9, alpha=0.45, color=\"#0072B2\", lw=0)\n", + " lim = cov[MOVE].max() * 100\n", + " ax.plot([0, lim], [0, lim], color=\"#9AA0A6\", lw=1, ls=\"--\") # realized == implied\n", + " ax.set_xlabel(\"implied absolute earnings move (%)\")\n", + " ax.set_ylabel(\"|car1| realized (%)\")\n", + " ax.set_title(f\"{QUARTER}: the priced move vs. the realized one\")\n", + " ax.grid(color=\"#D7DBE0\", lw=0.6)\n", + " ax.set_axisbelow(True)\n", + " for side in (\"top\", \"right\"):\n", + " ax.spines[side].set_visible(False)\n", + " plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "ed68cbc5", + "metadata": {}, + "source": [ + "## 4. Why the score needs an event-specific transform\n", + "\n", + "The competition fits `y = α + β₁·ĝ + β₂·s` within a period (`y` the percentile rank of `car1`, `s` that of the surprise, `ĝ` your predicted percentile) and credits the R² gained over the surprise-only fit. That gain is `β₁²·s₁₁.₂ / s_yy` (see `examples.scoring`), so replacing every `p` with `a + b·p` changes nothing: `β₁` absorbs `b`, `α` absorbs `a`. Neither the direction nor the spread of your numbers is scored.\n", + "\n", + "Only a factor that differs across events can move the score. The implied move is one, known before the cutoff. The question is what to multiply, and by how much." + ] + }, + { + "cell_type": "markdown", + "id": "60631c7f", + "metadata": {}, + "source": [ + "## 5. Two rescalings: naive and adjusted\n", + "\n", + "A prediction is a centre plus a deviation. A stock priced for a 3% move cannot reach the tails of the cross-section, so its deviation should count for less; one priced for 12% can. Ranks are bounded, so the effect saturates: `E[y | ĝ, m] ≈ centre + (ĝ − centre)·g(m)`, `g` increasing and concave.\n", + "\n", + "The **naive** rescale, `0.5 + (ĝ − 0.5)·m/m̄`, gets both wrong. The baselines centre near 0.7, not 0.5, so pivoting at 0.5 pushes every low-move event down and every high-move event up whatever the model said. And a linear factor of up to three or four clips about one event in seven at 0 or 1.\n", + "\n", + "The **adjusted** rescale pivots at the submission's own mean and takes a square root: `μ + (ĝ − μ)·√(m/m̄)`. These are the defaults of `scale_by_implied_move`; the naive rule is `pivot=0.5, power=1.0`. Events without the item keep their prediction. The fit sample is the quarter's covered outcomes with a surprise and both baselines, ranked over the **whole** quarter as the scorer does. First the mechanism: the slope of `y` on `ĝ` by tercile of the implied move." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "741c16b7", + "metadata": {}, + "outputs": [], + "source": [ + "if have_scoring_data:\n", + " # Rank over the WHOLE quarter first (the scorer's percentiles), then restrict.\n", + " scored = add_percentiles(outcomes).merge(\n", + " stats[[\"event_id\", *STATISTICS]], on=\"event_id\", how=\"left\"\n", + " )\n", + " fit_df = scored.dropna(subset=[\"y\", \"surprise_pct\", *models, MOVE]).copy()\n", + "\n", + " def ranked(values: pd.Series) -> pd.Series:\n", + " return pd.Series(percentile_ranks(values.tolist()), index=values.index)\n", + "\n", + " fit_df[\"move_pct\"] = ranked(fit_df[MOVE])\n", + " n_no_skew = int(fit_df[\"skew_25_delta\"].isna().sum())\n", + " fit_df[\"skew_pct\"] = ranked(fit_df[\"skew_25_delta\"].fillna(fit_df[\"skew_25_delta\"].mean()))\n", + "\n", + " for m in models:\n", + " fit_df[f\"{m} naive\"] = scale_by_implied_move(fit_df[m], fit_df[MOVE], pivot=0.5, power=1.0)\n", + " fit_df[f\"{m} adjusted\"] = scale_by_implied_move(fit_df[m], fit_df[MOVE])\n", + "\n", + " def share_clipped(column: str) -> float:\n", + " return float(((fit_df[column] <= 0) | (fit_df[column] >= 1)).mean())\n", + "\n", + " print(f\"Fit sample: N = {len(fit_df):,} covered outcomes with a surprise and both baselines\")\n", + " print(\n", + " f\" mean implied move = {fit_df[MOVE].mean():.2%}; skew mean-filled on {n_no_skew:,} rows\"\n", + " )\n", + " for m in models:\n", + " print(\n", + " f\" {m}: mean ĝ = {fit_df[m].mean():.3f} over {fit_df[m].nunique()} distinct values; \"\n", + " f\"clipped {share_clipped(f'{m} naive'):.1%} naive, \"\n", + " f\"{share_clipped(f'{m} adjusted'):.1%} adjusted\"\n", + " )" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "01c9b576", + "metadata": {}, + "outputs": [], + "source": [ + "if have_scoring_data:\n", + " import statsmodels.api as sm\n", + "\n", + " fit_df[\"move tercile\"] = pd.qcut(fit_df[MOVE], 3, labels=[\"low\", \"mid\", \"high\"])\n", + " rows = []\n", + " for tercile, group in fit_df.groupby(\"move tercile\", observed=True):\n", + " row = {\"tercile\": tercile, \"n\": len(group), \"mean move\": group[MOVE].mean()}\n", + " row[\"sd(y)\"] = group[\"y\"].std()\n", + " for m in models:\n", + " X = sm.add_constant(group[[\"surprise_pct\", m]])\n", + " row[f\"slope on ĝ — {m}\"] = sm.OLS(group[\"y\"], X).fit().params[m]\n", + " rows.append(row)\n", + " display(\n", + " pd.DataFrame(rows)\n", + " .set_index(\"tercile\")\n", + " .style.format({\"mean move\": \"{:.1%}\", \"sd(y)\": \"{:.3f}\"}, precision=3)\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "4e0fe301", + "metadata": {}, + "source": [ + "## 6. The regression table\n", + "\n", + "Eight columns on one fit sample: the surprise alone; the surprise plus the two option statistics as plain regressors (percentile-ranked, skew mean-filled); then for each baseline, the surprise with `ĝ` as submitted, naively scaled, and adjusted. Read the R² row.\n", + "\n", + "As of early October 2026: the option statistics add about 0.003 as regressors; the naive rescale costs Gemini about 0.007 and GPT-5 nano about 0.015; the adjusted rescale adds about 0.005 for Gemini and nothing for GPT-5 nano. Modest, but in the right direction, and a freely fitted interaction does no better. The numbers will drift as the quarter fills in." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "97af8216", + "metadata": {}, + "outputs": [], + "source": [ + "if have_scoring_data:\n", + " import statsmodels.api as sm\n", + " from stargazer.stargazer import Stargazer\n", + "\n", + " gemini, gpt = models # display names, in BASELINE_LABELS order\n", + " S, RM, RK = \"s (surprise pctile)\", \"implied move (pctile)\", \"skew (pctile)\"\n", + " G, GN, GA = \"ĝ (model pctile)\", \"ĝ naively scaled\", \"ĝ adjusted\"\n", + "\n", + " def fit(cols):\n", + " # cols: {display_name -> source column}; shared names let stargazer stack rows.\n", + " X = sm.add_constant(pd.DataFrame({name: fit_df[src] for name, src in cols.items()}))\n", + " return sm.OLS(fit_df[\"y\"].to_numpy(), X).fit()\n", + "\n", + " fits = [\n", + " fit({S: \"surprise_pct\"}), # (1) surprise only\n", + " fit({S: \"surprise_pct\", RM: \"move_pct\", RK: \"skew_pct\"}), # (2) surprise + options\n", + " ]\n", + " for m in (gemini, gpt):\n", + " fits += [\n", + " fit({S: \"surprise_pct\", G: m}), # surprise + model\n", + " fit({S: \"surprise_pct\", GN: f\"{m} naive\"}), # surprise + model, naively scaled\n", + " fit({S: \"surprise_pct\", GA: f\"{m} adjusted\"}), # surprise + model, adjusted\n", + " ]\n", + "\n", + " star = Stargazer(fits)\n", + " star.title(f\"Rescaling the reference baselines by the option-implied move — {QUARTER}\")\n", + " star.custom_columns([\"Benchmarks\", gemini, gpt], [2, 3, 3])\n", + " star.covariate_order([\"const\", S, RM, RK, G, GN, GA])\n", + " star.rename_covariates({\"const\": \"Intercept\"})\n", + " star.dependent_variable_name(\"r_abn (percentiled CAR1)\")\n", + " star.add_line(\n", + " \"ĝ transform\",\n", + " [\"\", \"\", \"as submitted\", \"naive\", \"adjusted\", \"as submitted\", \"naive\", \"adjusted\"],\n", + " )\n", + " star.show_model_numbers(True)\n", + " display(star)" + ] + }, + { + "cell_type": "markdown", + "id": "6d9fac44", + "metadata": {}, + "source": [ + "## 7. The competition's own metric\n", + "\n", + "Section 6 used a common fit sample. Here each transform is applied as you would submit it, over every scored outcome in the quarter with uncovered events keeping `ĝ`, and `score_submission` reports `delta_r_squared`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3ac93475", + "metadata": {}, + "outputs": [], + "source": [ + "if have_scoring_data:\n", + " full = scored.copy() # every scored outcome in the quarter; NaN move -> ĝ unchanged\n", + " rows = []\n", + " for m in models:\n", + " full[f\"{m} naive\"] = scale_by_implied_move(full[m], full[MOVE], pivot=0.5, power=1.0)\n", + " full[f\"{m} adjusted\"] = scale_by_implied_move(full[m], full[MOVE])\n", + " for label, column in (\n", + " (\"as submitted\", m),\n", + " (\"naive\", f\"{m} naive\"),\n", + " (\"adjusted\", f\"{m} adjusted\"),\n", + " ):\n", + " result = score_submission(full, column)\n", + " rows.append(\n", + " {\n", + " \"baseline\": m,\n", + " \"ĝ transform\": label,\n", + " \"n_obs\": result[\"n_obs\"],\n", + " \"R² surprise\": result[\"r_squared_surprise\"],\n", + " \"R²\": result[\"r_squared\"],\n", + " \"ΔR²\": result[\"delta_r_squared\"],\n", + " }\n", + " )\n", + " display(\n", + " pd.DataFrame(rows)\n", + " .set_index([\"baseline\", \"ĝ transform\"])\n", + " .style.format({\"R² surprise\": \"{:.4f}\", \"R²\": \"{:.4f}\", \"ΔR²\": \"{:+.4f}\"})\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "f5a1c982", + "metadata": {}, + "source": [ + "## 8. Caveats\n", + "\n", + "- The move is a **risk-neutral magnitude**: no direction, and the realized absolute return exceeds it more than a third of the time.\n", + "- **Coverage is partial.** Fewer than half of events carry the item, mostly larger companies, so most predictions are untouched; hence section 7's gain is smaller than section 6's.\n", + "- **One quarter, in sample.** A few thousandths of R², with the pivot and reference scale computed on the same data. The lesson is the shape of the result, not its size.\n", + "- A multi-asset event carries one item, attached above to each focal asset.\n", + "\n", + "## What to take away\n", + "\n", + "- The item is optional. Match on `id` and `kind`, read each `status`, and build for its absence.\n", + "- The move is about `0.8` times the volatility, so use one of the two. The skew is not a direction.\n", + "- The score ignores any uniform remap of your predictions; a per-event input helps only through how it differs across events.\n", + "- Pivot at your own centre and let the factor saturate. `scale_by_implied_move` does both by default; `pivot=0.5, power=1.0` shows what the naive version costs.\n", + "\n", + "## Where to go from here\n", + "\n", + "The item reaches a live event before its `knowledge_cutoff`, so a deployed handler can apply `scale_by_implied_move` to its own prediction before `POST /predictions`, with its typical prediction as the pivot and a running mean of past moves as the scale. For the scoring transform see section 6 of [`01_historical_archive.ipynb`](01_historical_archive.ipynb); for the preview beside this item, [`03_earnings_previews.ipynb`](03_earnings_previews.ipynb)." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/src/examples/__init__.py b/src/examples/__init__.py index 6802faa..eb8e07f 100644 --- a/src/examples/__init__.py +++ b/src/examples/__init__.py @@ -14,6 +14,8 @@ scoring — the competition's exact percentile scoring transform summary — the exact 10-fact extraction behind each event's disclosure preview — the agent-written earnings preview attached from 2026Q3 onward + option_stats — the option-implied statistics attached from 2026Q3 onward, and + the implied-move rescaling of a submission's predictions """ from __future__ import annotations diff --git a/src/examples/option_stats.py b/src/examples/option_stats.py index 3c9d9bb..6b9b4ec 100644 --- a/src/examples/option_stats.py +++ b/src/examples/option_stats.py @@ -47,6 +47,12 @@ thinly to measure, which is most smaller companies. Fewer than half of 2026Q3 events carry it. The skew can be unavailable on its own when only the out-of-the-money options are thin. Events before 2026Q3 never carry the item. + +Using the move: the competition's score is invariant to any uniform affine remap of +a submission's predictions (see :mod:`examples.scoring`), so a number that is the +same for every event cannot change it. The implied move differs across events, and +:func:`scale_by_implied_move` uses it to stretch or shrink each prediction's +distance from the submission's own centre. Notebook 04 shows the effect. """ from __future__ import annotations @@ -157,3 +163,48 @@ def option_stats_frame(records: Iterable[dict]) -> pd.DataFrame: df[column] = pd.to_datetime(df[column], utc=True, format="ISO8601") df = df.sort_values("event_datetime").reset_index(drop=True) return df + + +def scale_by_implied_move( + prediction: pd.Series, + move: pd.Series, + *, + pivot: float | None = None, + power: float = 0.5, +) -> pd.Series: + """Rescale predicted percentiles event by event with the option-implied move. + + Returns ``pivot + (prediction - pivot) * (move / move.mean()) ** power``, clipped + to ``[0, 1]`` and indexed like ``prediction``. ``move`` is aligned to + ``prediction`` by index label first, so pass two columns of the same frame or + two series sharing an index. An event with no move (``NaN``) keeps its + prediction unchanged, and if no event has a usable move the input comes back + as it is. + + Why this shape: the competition scores how well predictions explain the + *percentile rank* of the one-day abnormal return. A rank cannot grow without + bound, so the effect of a larger expected magnitude saturates, and a + submission's own forecasts are rarely centred at 0.5. Hence the two defaults: + + ``pivot=None`` + Pivot at the submission's own mean prediction. Stretching away from 0.5 + instead pushes every low-move event down and every high-move event up + whatever the model thought, which adds the move as a signed regressor the + data do not support. + ``power=0.5`` + A square-root factor, so a move four times the average doubles the + deviation rather than quadrupling it. On 2026Q3 this parameter-free rule + captures essentially all of the gain a freely fitted interaction would. + + ``pivot=0.5, power=1.0`` gives the "naive" rescale, ``0.5 + (p - 0.5) * m / mean(m)``, + which lowers the score for the reference baselines. The reference scale + ``move.mean()`` matters only through the clip: dividing by any other positive + constant is an affine remap, which the score ignores. + """ + move = pd.to_numeric(move.reindex(prediction.index), errors="coerce").astype("float64") + reference = float(move.mean()) + if not reference > 0: + return prediction.copy() + centre = float(prediction.mean()) if pivot is None else pivot + factor = (move / reference).pow(power).where(move.notna(), 1.0) + return (centre + (prediction - centre) * factor).clip(0.0, 1.0) diff --git a/tests/test_option_stats.py b/tests/test_option_stats.py index 7495e6e..b8fe2db 100644 --- a/tests/test_option_stats.py +++ b/tests/test_option_stats.py @@ -4,19 +4,24 @@ import math +import pandas as pd import pytest +from examples.archive import read_jsonl_gz from examples.option_stats import ( OPTION_STATS_ID, OPTION_STATS_SOURCE, + STATISTICS, option_implied_stats_from_disclosure, option_stats_frame, + scale_by_implied_move, statistic_status, statistic_value, ) from examples.preview import earnings_preview_from_disclosure from examples.schemas import Disclosure from examples.summary import facts_from_disclosure +from tests.conftest import SAMPLE_ARCHIVE_2026Q3 from tests.test_preview import FACTS_ITEM, PREVIEW_ITEM STATS = { @@ -140,3 +145,80 @@ def test_option_stats_frame() -> None: assert with_stats["as_of"] < with_stats["knowledge_cutoff"] assert math.isnan(without["implied_earnings_volatility"]) and without["tickers"] == "NKE" assert option_stats_frame([]).empty + + +# ----- scale_by_implied_move --------------------------------------------------- + + +def test_scale_leaves_uncovered_events_unchanged() -> None: + prediction = pd.Series([0.9, 0.3, 0.6, 0.75]) + move = pd.Series([0.05, float("nan"), 0.20, float("nan")]) + out = scale_by_implied_move(prediction, move) + assert out.index.equals(prediction.index) + assert out[1] == prediction[1] and out[3] == prediction[3] + assert out[0] != prediction[0] and out[2] != prediction[2] + # No usable move at all: the input comes back as it is. + nothing = pd.Series([float("nan")] * 4) + pd.testing.assert_series_equal(scale_by_implied_move(prediction, nothing), prediction) + + +def test_scale_pivots_at_the_prediction_mean_by_default() -> None: + prediction = pd.Series([0.9, 0.3, 0.6]) # mean 0.6 + move = pd.Series([0.05, 0.05, 0.20]) # mean 0.10 + out = scale_by_implied_move(prediction, move) + # Below-average move shrinks the deviation from the mean by sqrt(1/2)... + assert out[0] == pytest.approx(0.6 + 0.3 * math.sqrt(0.5)) + assert out[1] == pytest.approx(0.6 - 0.3 * math.sqrt(0.5)) + # ...and a prediction at the pivot is untouched whatever the move. + assert out[2] == pytest.approx(0.6) + # A move equal to the average leaves the prediction exactly as it was. + same = scale_by_implied_move(prediction, pd.Series([0.1, 0.1, 0.1])) + pd.testing.assert_series_equal(same, prediction) + # A move twice the average stretches the deviation by sqrt(2). + assert scale_by_implied_move(prediction, pd.Series([0.1, 0.1, 0.4]))[2] == pytest.approx(0.6) + stretched = scale_by_implied_move(pd.Series([0.8, 0.4]), pd.Series([0.2, 0.05])) + assert stretched[0] == pytest.approx(0.6 + 0.2 * math.sqrt(0.2 / 0.125)) + + +def test_scale_clips_to_the_unit_interval() -> None: + out = scale_by_implied_move( + pd.Series([0.05, 0.95, 0.5]), pd.Series([1.0, 1.0, 0.0001]), pivot=0.5, power=1.0 + ) + assert list(out) == [0.0, 1.0, 0.5] + + +def test_naive_rescale_is_pivot_half_power_one() -> None: + prediction = pd.Series([0.82, 0.55, 0.9, 0.5, 0.2]) + move = pd.Series([0.03, 0.08, 0.15, 0.06, float("nan")]) + naive = (0.5 + (prediction - 0.5) * move / move.mean()).clip(0, 1) + naive[4] = prediction[4] # the uncovered event keeps its prediction + out = scale_by_implied_move(prediction, move, pivot=0.5, power=1.0) + pd.testing.assert_series_equal(out, naive) + + +def test_scale_aligns_on_index() -> None: + prediction = pd.Series([0.9, 0.3, 0.6], index=["a", "b", "c"]) + move = pd.Series([0.05, 0.20, 0.05], index=["b", "c", "a"]) # shuffled labels + out = scale_by_implied_move(prediction, move) + expected = scale_by_implied_move(prediction, move.reindex(prediction.index)) + pd.testing.assert_series_equal(out, expected) + assert list(out.index) == ["a", "b", "c"] + + +def test_bundled_2026q3_sample_carries_option_stats_for_every_event() -> None: + records = list(read_jsonl_gz(SAMPLE_ARCHIVE_2026Q3)) + assert len(records) == 6 + frame = option_stats_frame(records) + assert frame["has_option_stats"].all() + assert (frame["as_of"] < frame["knowledge_cutoff"]).all() + assert frame[list(STATISTICS)].notna().all().all() # all six have liquid wings + for record in records: + stats = option_implied_stats_from_disclosure(record) + j = statistic_value(stats, "implied_earnings_volatility") + phi = 0.5 * (1 + math.erf((j / 2) / math.sqrt(2))) + assert statistic_value(stats, "implied_absolute_earnings_move") == pytest.approx( + 4 * phi - 2, abs=1e-4 + ) + # The other two items are still there and still parse. + assert len(facts_from_disclosure(record)) == 10 + assert earnings_preview_from_disclosure(record)