diff --git a/README.md b/README.md index b58806e..0ca9a34 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,9 @@ [![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE) Notebook examples for exploring the [Explaining Markets](https://explainingmarkets.ai/) -competition API and its historical data archive. Notebook-first, with the repeated +competition API and its historical data archive — including the two disclosure items +each earnings event carries, the ten call facts and (from 2026Q3) the earnings +preview. Notebook-first, with the repeated API / parsing / plotting boilerplate factored into a small `examples` package so the notebooks stay short and readable. @@ -18,8 +20,9 @@ notebooks stay short and readable. | [`notebooks/00_api_quickstart.ipynb`](notebooks/00_api_quickstart.ipynb) | The read endpoints — events calendar (table + chart), submission health, archive manifest, webhook self-test | | [`notebooks/01_historical_archive.ipynb`](notebooks/01_historical_archive.ipynb) | Download, cache, and load the historical archive, then reproduce the baseline scoring regressions (Koijen & Levy WP, Table 3) with the competition's exact scoring transform | | [`notebooks/02_earnings_call_facts.ipynb`](notebooks/02_earnings_call_facts.ipynb) | Where each event's ten-fact `disclosure` comes from — the exact prompt, model config, and parsing behind it, and how extraction failures surface | +| [`notebooks/03_earnings_previews.ipynb`](notebooks/03_earnings_previews.ipynb) | The **earnings preview** — the agent-written pre-release research note that recent events carry as a second disclosure item: how it is built (at a high level), what it looks like, and how it pairs with the facts and the realized reaction | -All three run top-to-bottom. With an API key they hit the live API; **without one they +All four run top-to-bottom. With an API key they hit the live API; **without one they fall back to a small bundled sample** (`data/sample/`), so you can run them — and CI can execute them — with no credentials. @@ -60,7 +63,7 @@ events_frame(events).head() ```text notebooks/ the numbered, narrative examples -src/examples/ helper package: config · client · schemas · frames · plotting · archive · scoring · summary +src/examples/ helper package: config · client · schemas · frames · plotting · archive · scoring · summary · preview scripts/ small runnable scripts (e.g. headless archive download) tests/ offline unit tests (mocked HTTP + bundled sample data) data/ diff --git a/data/README.md b/data/README.md index f13f538..0528569 100644 --- a/data/README.md +++ b/data/README.md @@ -16,6 +16,14 @@ Just enough data for the notebooks to run top-to-bottom **without an API key** documented event payload with an inlined disclosure (`facts`). It is a **stand-in for exposition** — the real archive is served by the API and its lines may include additional fields. Treat the API as the source of truth. +- `archive_EARNINGS_RELEASE_2026Q3.jsonl.gz` — six **real** archive lines (NVDA, + WMT, HD, CRM, AVGO, ORCL), copied verbatim from the live 2026Q3 file on + 2026-09-24. Each carries both disclosure items — the ten facts and the + agent-written **earnings preview** — plus the realized `event_returns`, + `metrics`, and `knowledge_cutoff` the archive adds once an event is scored, so + `notebooks/03_earnings_previews.ipynb` and the preview section of notebook 01 + run offline on genuine content. 2026Q3 was unsealed when this was pulled, so + fields may later change upstream; the API remains the source of truth. - `transcript_sample.json` — a **synthetic** earnings-call transcript for a fictional company (Northwind Logistics). Real transcripts are licensed and are not distributed with this repo; this one exists so diff --git a/data/sample/archive_EARNINGS_RELEASE_2026Q3.jsonl.gz b/data/sample/archive_EARNINGS_RELEASE_2026Q3.jsonl.gz new file mode 100644 index 0000000..88774e4 Binary files /dev/null and b/data/sample/archive_EARNINGS_RELEASE_2026Q3.jsonl.gz differ diff --git a/notebooks/00_api_quickstart.ipynb b/notebooks/00_api_quickstart.ipynb index 1d24718..b535ea2 100644 --- a/notebooks/00_api_quickstart.ipynb +++ b/notebooks/00_api_quickstart.ipynb @@ -5,23 +5,18 @@ "id": "0", "metadata": {}, "source": [ - "# 00 — API quickstart\n", + "# 00 \u2014 API quickstart\n", "\n", - "A guided tour of the endpoints you *read from* while exploring the\n", - "[Explaining Markets](https://explainingmarkets.ai/) competition:\n", + "A guided tour of the endpoints you *read from* while exploring the [Explaining Markets](https://explainingmarkets.ai/) competition:\n", "\n", - "- **`GET /events`** — the events calendar, shown as a clean table and a chart\n", - "- **`GET /archive`** — a peek at the historical-data manifest\n", - "- **`POST /webhook/test`** — fire a synthetic delivery at your webhook\n", - "- **`GET /health`** — your submission's rolling delivery + prediction counters\n", + "- **`GET /events`** \u2014 the events calendar, shown as a clean table and a chart\n", + "- **`GET /archive`** \u2014 a peek at the historical-data manifest\n", + "- **`POST /webhook/test`** \u2014 fire a synthetic delivery at your webhook\n", + "- **`GET /health`** \u2014 your submission's rolling delivery + prediction counters\n", "\n", - "Everything runs top-to-bottom. With an API key in `.env` you hit the live API;\n", - "without one, the calendar section falls back to a small bundled sample so you can\n", - "still see the shape of things.\n", + "Everything runs top-to-bottom. With an API key in `.env` you hit the live API; without one, the calendar section falls back to a small bundled sample so you can still see the shape of things.\n", "\n", - "> **Not covered here:** submitting predictions (`POST /predictions`). That places a\n", - "> real, scored entry and belongs in a deployed webhook handler — see the starter\n", - "> repos, not these examples." + "> **Not covered here:** submitting predictions (`POST /predictions`). That places a real, scored entry and belongs in a deployed webhook handler \u2014 see the starter repos, not these examples." ] }, { @@ -47,7 +42,7 @@ "SAMPLE = REPO / \"data\" / \"sample\"\n", "\n", "config = load_config()\n", - "print(\"Mode: \", \"LIVE\" if config.is_live else \"SAMPLE (no EM_API_KEY — using bundled data)\")\n", + "print(\"Mode: \", \"LIVE\" if config.is_live else \"SAMPLE (no EM_API_KEY \u2014 using bundled data)\")\n", "print(\"API base:\", config.api_base_url)" ] }, @@ -56,12 +51,9 @@ "id": "2", "metadata": {}, "source": [ - "## 1. Events calendar — `GET /events`\n", + "## 1. Events calendar \u2014 `GET /events`\n", "\n", - "The calendar lists scheduled and realized events. Each event has a type — an open\n", - "string set; currently only `EARNINGS_RELEASE` is emitted, though new types may be\n", - "added over time — a timestamp, and its focal assets. We flatten it into a tidy,\n", - "sorted table." + "The calendar lists scheduled and realized events. Each event has a type \u2014 an open string set; currently only `EARNINGS_RELEASE` is emitted, though new types may be added over time \u2014 a timestamp, and its focal assets. We flatten it into a tidy, sorted table." ] }, { @@ -88,7 +80,7 @@ "id": "4", "metadata": {}, "source": [ - "A stacked daily bar makes the shape of the calendar obvious at a glance — how many events land each day, split by type." + "A stacked daily bar makes the shape of the calendar obvious at a glance \u2014 how many events land each day, split by type." ] }, { @@ -107,10 +99,9 @@ "id": "6", "metadata": {}, "source": [ - "## 2. Historical archive — `GET /archive`\n", + "## 2. Historical archive \u2014 `GET /archive`\n", "\n", - "The archive is how you get historical events for offline backtesting. Here we just\n", - "glance at the manifest; **notebook 01** downloads, caches, and loads the data." + "The archive is how you get historical events for offline backtesting. Here we just glance at the manifest; **notebook 01** downloads, caches, and loads the data." ] }, { @@ -133,13 +124,9 @@ "id": "8", "metadata": {}, "source": [ - "## 3. Webhook self-test — `POST /webhook/test`\n", + "## 3. Webhook self-test \u2014 `POST /webhook/test`\n", "\n", - "Fires a synthetic `event_type=TEST` delivery at the webhook URL registered for\n", - "your submission. If it's enqueued you get a `202`; the delivery should reach your\n", - "endpoint within a few seconds and show up in the health counters below. A failure\n", - "here usually means no webhook URL is set, or credentials aren't initialized in the\n", - "portal yet." + "Fires a synthetic `event_type=TEST` delivery at the webhook URL registered for your submission. If it's enqueued you get a `202`; the delivery should reach your endpoint within a few seconds and show up in the health counters below. A failure here usually means no webhook URL is set, or credentials aren't initialized in the portal yet." ] }, { @@ -155,10 +142,10 @@ " try:\n", " with Client.from_env() as client:\n", " status = client.send_test_event()\n", - " print(f\"✓ Test event enqueued (HTTP {status}). It should reach your webhook\")\n", - " print(\" within a few seconds — watch the health counters in the next section.\")\n", + " print(f\"\u2713 Test event enqueued (HTTP {status}). It should reach your webhook\")\n", + " print(\" within a few seconds \u2014 watch the health counters in the next section.\")\n", " except ApiError as exc:\n", - " print(f\"✗ Test event failed (HTTP {exc.status_code}).\")\n", + " print(f\"\u2717 Test event failed (HTTP {exc.status_code}).\")\n", " print(\" Check that your submission has a webhook URL and initialized credentials.\")\n", " print(f\" Details: {exc}\")" ] @@ -168,11 +155,9 @@ "id": "10", "metadata": {}, "source": [ - "## 4. Submission health — `GET /health`\n", + "## 4. Submission health \u2014 `GET /health`\n", "\n", - "Rolling 24-hour counters for outbound webhook deliveries and inbound predictions.\n", - "Right after the self-test above, you should see `webhook_n_2xx` (and\n", - "`webhook_last_delivery_at`) tick up — proof the round trip works end to end." + "Rolling 24-hour counters for outbound webhook deliveries and inbound predictions. Right after the self-test above, you should see `webhook_n_2xx` (and `webhook_last_delivery_at`) tick up \u2014 proof the round trip works end to end." ] }, { @@ -197,10 +182,8 @@ "source": [ "## Next steps\n", "\n", - "- **`01_historical_archive.ipynb`** — download, cache, and load the historical\n", - " archive for offline experiments.\n", - "- Ready to compete? Deploy a webhook handler with one of the starter repos and\n", - " submit predictions from there." + "- **`01_historical_archive.ipynb`** \u2014 download, cache, and load the historical archive for offline experiments.\n", + "- Ready to compete? Deploy a webhook handler with one of the starter repos and submit predictions from there." ] } ], diff --git a/notebooks/01_historical_archive.ipynb b/notebooks/01_historical_archive.ipynb index 16e97f1..42ac9ca 100644 --- a/notebooks/01_historical_archive.ipynb +++ b/notebooks/01_historical_archive.ipynb @@ -5,23 +5,20 @@ "id": "0", "metadata": {}, "source": [ - "# 01 — Historical archive\n", + "# 01 \u2014 Historical archive\n", "\n", - "The competition exposes a historical archive so you can experiment **offline**:\n", - "one gzip-JSONL file per `event_type` × `quarter`, each line a past event with its\n", - "disclosure. This notebook:\n", + "The competition exposes a historical archive so you can experiment **offline**: one gzip-JSONL file per `event_type` \u00d7 `quarter`, each line a past event with its disclosure. This notebook:\n", "\n", "1. Browses the archive manifest (`GET /archive`)\n", "2. Downloads and **caches** the files locally (`data/archive/`, gitignored)\n", "3. Loads them into a pandas DataFrame\n", "4. Inspects a single event and its disclosure\n", + "5. Looks at the **earnings preview** \u2014 the second disclosure item recent events carry\n", + "6. Reproduces the baseline scoring regressions\n", "\n", - "With a live key you pull the real archive; without one, you load the small bundled\n", - "sample so the mechanics are still clear.\n", + "With a live key you pull the real archive; without one, you load the small bundled sample so the mechanics are still clear.\n", "\n", - "> Download URLs are short-lived and signed. The helper refreshes an expired URL\n", - "> automatically (via `GET /archive/{event_type}/{quarter}`) when you pass it a\n", - "> client." + "> Download URLs are short-lived and signed. The helper refreshes an expired URL automatically (via `GET /archive/{event_type}/{quarter}`) when you pass it a client." ] }, { @@ -33,18 +30,25 @@ "source": [ "from pathlib import Path\n", "\n", - "from IPython.display import display\n", + "from IPython.display import Markdown, display\n", "\n", "from examples import Client, load_config\n", "from examples.archive import download_archive, load_archive\n", "from examples.frames import manifest_frame\n", + "from examples.preview import (\n", + " earnings_preview_from_disclosure,\n", + " preview_display_markdown,\n", + " preview_title,\n", + " previews_frame,\n", + ")\n", + "from examples.summary import facts_from_disclosure\n", "\n", "REPO = next(p for p in [Path.cwd(), *Path.cwd().parents] if (p / \"data\" / \"sample\").is_dir())\n", "ARCHIVE_DIR = REPO / \"data\" / \"archive\" # gitignored download cache\n", "SAMPLE_DIR = REPO / \"data\" / \"sample\" # tiny bundled stand-in\n", "\n", "config = load_config()\n", - "print(\"Mode:\", \"LIVE\" if config.is_live else \"SAMPLE (no EM_API_KEY — using bundled data)\")" + "print(\"Mode:\", \"LIVE\" if config.is_live else \"SAMPLE (no EM_API_KEY \u2014 using bundled data)\")" ] }, { @@ -54,9 +58,7 @@ "source": [ "## 1. Browse the manifest\n", "\n", - "Each row is one downloadable file — what event type and quarter it covers, how\n", - "many events it holds, and how big it is. (The signed URLs are omitted from this\n", - "view.)" + "Each row is one downloadable file \u2014 what event type and quarter it covers, how many events it holds, and how big it is. (The signed URLs are omitted from this view.)" ] }, { @@ -82,9 +84,9 @@ "source": [ "## 2. Download & cache\n", "\n", - "`download_archive` writes each file into `data/archive/`, **skipping** any file\n", - "already cached at the size the manifest reports — so re-running is cheap. Narrow\n", - "the pull with `only=`, e.g. `only=lambda f: f.quarter >= \"2026Q1\"`." + "`download_archive` writes each file into `data/archive/`, **skipping** any file already cached at the size the manifest reports \u2014 so re-running is cheap. Narrow the pull with `only=`, e.g. `only=lambda f: f.quarter >= \"2026Q1\"`.\n", + "\n", + "A quarter whose manifest row says `sealed = False` is still filling in: its file grows as events are scored and disclosures are attached, so its size changes and the cache check re-downloads it. Sealed quarters never change." ] }, { @@ -116,9 +118,7 @@ "source": [ "## 3. Load into pandas\n", "\n", - "`load_archive` reads every `*.jsonl.gz` under a directory (or a single file) into\n", - "one DataFrame, adding `event_type` and `quarter` provenance columns from each\n", - "filename." + "`load_archive` reads every `*.jsonl.gz` under a directory (or a single file) into one DataFrame, adding `event_type` and `quarter` provenance columns from each filename." ] }, { @@ -141,9 +141,9 @@ "source": [ "## 4. Inspect one event\n", "\n", - "Pull a single row and read its focal assets and disclosure facts. (The bundled\n", - "sample inlines disclosures under a `disclosure` field; the live archive may carry\n", - "additional fields — inspect `df.columns` to see what a given dataset provides.)" + "Pull a single row and read its focal assets and disclosure. The bundled sample inlines disclosures under a `disclosure` field; the live archive carries additional fields \u2014 inspect `df.columns` to see what a given dataset provides.\n", + "\n", + "`disclosure.items[]` can hold **two** items: the ten earnings-call facts (`kind=\"facts\"`, the subject of notebook 02) and, for recent events, the earnings preview (`id=\"earnings-preview\"`, `kind=\"text\"`, section 5). `facts_from_disclosure` and `earnings_preview_from_disclosure` look each one up and return `[]` / `None` when it is absent \u2014 never assume a position in the list." ] }, { @@ -154,21 +154,71 @@ "outputs": [], "source": [ "row = df.iloc[0]\n", + "record = row.to_dict()\n", "\n", - "focal = row.get(\"focal_assets\") if \"focal_assets\" in df.columns else None\n", - "tickers = [a[\"identifier_value\"] for a in (focal or [])]\n", + "focal = record.get(\"focal_assets\")\n", + "tickers = [a[\"identifier_value\"] for a in (focal if isinstance(focal, list) else [])]\n", "print(f\"event_type={row['event_type']} quarter={row['quarter']} tickers={tickers}\")\n", - "print(f\"event_datetime={row.get('event_datetime')}\\n\")\n", - "\n", - "disclosure = row.get(\"disclosure\") if \"disclosure\" in df.columns else None\n", - "if isinstance(disclosure, dict):\n", - " for item in disclosure.get(\"items\", []):\n", - " if item.get(\"kind\") == \"facts\":\n", - " print(f\"facts from {item.get('source')!r}:\")\n", - " for fact in item[\"content\"]:\n", - " print(\" -\", fact)\n", + "print(f\"event_datetime={record.get('event_datetime')}\\n\")\n", + "\n", + "for fact in facts_from_disclosure(record):\n", + " print(\" -\", fact)\n", + "\n", + "preview = earnings_preview_from_disclosure(record)\n", + "if preview is None:\n", + " print(\"\\nNo earnings preview on this event (expected before 2026Q3 \u2014 see section 5).\")\n", "else:\n", - " print(\"No inlined 'disclosure' column here — check df.columns for this dataset's fields.\")" + " print(f\"\\nEarnings preview attached: {preview_title(preview)!r}, {len(preview):,} chars\")" + ] + }, + { + "cell_type": "markdown", + "id": "4be0df7f", + "metadata": {}, + "source": [ + "## 5. The earnings preview (2026Q3 onward)\n", + "\n", + "From 2026Q3, an event's disclosure may carry a second item alongside the facts: the **earnings preview**, an agent-written research note assembled from public sources *before* the release \u2014 the market's ex ante information set, in one markdown string. Note that this is optional, so it may not be present for every event.\n", + "\n", + "To create the earnings preview, we run [Anthropic's Earnings Preview Skill](https://github.com/anthropics/financial-services/blob/main/plugins/agent-plugins/earnings-reviewer/skills/earnings-preview/SKILL.md) prior to the start of the CAR1 measurement window used for the competition. Notebook 03 covers how the preview is built and reads several in full; here we just check coverage and render one." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e1d0528e", + "metadata": {}, + "outputs": [], + "source": [ + "records = df.to_dict(\"records\")\n", + "coverage = previews_frame(records)\n", + "\n", + "by_quarter = coverage.groupby(\"quarter\")[\"has_preview\"].agg(events=\"size\", with_preview=\"sum\")\n", + "display(by_quarter)\n", + "\n", + "with_preview = coverage[coverage[\"has_preview\"]]\n", + "if with_preview.empty:\n", + " print(\"No previews in this dataset.\")\n", + "else:\n", + " print(f\"Earliest event with a preview: {with_preview['event_datetime'].min()}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "246cd47e", + "metadata": {}, + "outputs": [], + "source": [ + "if not with_preview.empty:\n", + " # Prefer NVDA when present; otherwise the most recent event with a preview.\n", + " pick = with_preview[with_preview[\"tickers\"] == \"NVDA\"]\n", + " pick = (pick if not pick.empty else with_preview).sort_values(\"event_datetime\").iloc[-1]\n", + " record = next(r for r in records if r[\"event_id\"] == pick[\"event_id\"])\n", + " preview = earnings_preview_from_disclosure(record)\n", + " print(f\"{pick['tickers']} {pick['event_id']} event={pick['event_datetime']}\")\n", + " print(f\"knowledge_cutoff={pick['knowledge_cutoff']} {len(preview or ''):,} chars\\n\")\n", + " display(Markdown(preview_display_markdown(preview or \"\")))" ] }, { @@ -176,29 +226,19 @@ "id": "10", "metadata": {}, "source": [ - "## 5. Reproduce the baseline scoring regressions\n", - "\n", - "`examples.scoring` ports the competition's **exact** scoring transform, so you can\n", - "(i) reproduce the leaderboard's numbers offline and (ii) use it in your own\n", - "analyses and optimizations. Scoring runs *within a period* (a quarter): the\n", - "realized one-day cumulative abnormal return (`car1`) of every outcome is\n", - "percentile-ranked into the dependent variable `y = r_abn`, and the earnings\n", - "surprise is percentile-ranked into the `s` regressor. Each reference baseline\n", - "already submitted a predicted percentile `ĝ`.\n", - "\n", - "We reproduce **Table 3** of Koijen & Levy (WP) on the disseminated data: `y`\n", - "regressed on five specifications —\n", - "\n", - "1. surprise only (`s`) — the naive benchmark,\n", - "2. Gemini only (`ĝ`),\n", - "3. GPT-5 nano only (`ĝ`),\n", + "## 6. Reproduce the baseline scoring regressions\n", + "\n", + "`examples.scoring` ports the competition's **exact** scoring transform, so you can (i) reproduce the leaderboard's numbers offline and (ii) use it in your own analyses and optimizations. Scoring runs *within a period* (a quarter): the realized one-day cumulative abnormal return (`car1`) of every outcome is percentile-ranked into the dependent variable `y = r_abn`, and the earnings surprise is percentile-ranked into the `s` regressor. Each reference baseline already submitted a predicted percentile `\u011d`.\n", + "\n", + "We reproduce **Table 3** of Koijen & Levy (WP) on the disseminated data: `y` regressed on five specifications \u2014\n", + "\n", + "1. surprise only (`s`) \u2014 the naive benchmark,\n", + "2. Gemini only (`\u011d`),\n", + "3. GPT-5 nano only (`\u011d`),\n", "4. surprise + Gemini,\n", "5. surprise + GPT-5 nano.\n", "\n", - "This section needs the **live** archive (the bundled sample carries no returns or\n", - "baseline predictions). We use `2025Q4`, the sealed quarter behind the paper's\n", - "table. (The paper also reports bootstrapped R² intervals; we show point estimates\n", - "and standard errors.)" + "This section needs the **live** archive (the bundled sample carries no returns or baseline predictions). We use `2025Q4`, the sealed quarter behind the paper's table. (The paper also reports bootstrapped R\u00b2 intervals; we show point estimates and standard errors.)" ] }, { @@ -213,7 +253,12 @@ "PERIOD = \"2025Q4\" # sealed quarter matching Koijen & Levy (WP), Table 3\n", "\n", "needed = {\"event_returns\", \"metrics\", \"baseline_predictions\"}\n", - "have_scoring_data = needed <= set(df.columns) and (df[\"quarter\"] == PERIOD).any()\n", + "in_period = df[\"quarter\"] == PERIOD\n", + "have_scoring_data = (\n", + " needed <= set(df.columns)\n", + " and in_period.any()\n", + " and df.loc[in_period, \"event_returns\"].notna().any() # the sample's mixed frame has NaN here\n", + ")\n", "\n", "if have_scoring_data:\n", " # Percentiles are ranked WITHIN the period, so restrict to the quarter first.\n", @@ -224,7 +269,7 @@ " fit_df = scored.dropna(subset=[\"y\", \"surprise_pct\", *models]).copy()\n", " print(f\"{PERIOD}: {len(scored):,} scored outcomes | fit sample N = {len(fit_df):,}\")\n", "else:\n", - " print(f\"Sample mode: section 5 needs the live archive for {PERIOD}. Skipping.\")" + " print(f\"Sample mode: section 6 needs the live archive for {PERIOD}. Skipping.\")" ] }, { @@ -240,7 +285,7 @@ " from stargazer.stargazer import Stargazer\n", "\n", " gemini, gpt = models # display names, in BASELINE_LABELS order\n", - " S, G = \"s (surprise pctile)\", \"ĝ (model pctile)\"\n", + " S, G = \"s (surprise pctile)\", \"\u011d (model pctile)\"\n", "\n", " def fit(cols):\n", " # cols: {display_name -> source column}; shared names let stargazer stack rows.\n", @@ -256,7 +301,7 @@ " ]\n", "\n", " star = Stargazer(fits)\n", - " star.title(f\"Transcript summaries — Koijen & Levy (WP) Table 3, reproduced on {PERIOD}\")\n", + " star.title(f\"Transcript summaries \u2014 Koijen & Levy (WP) Table 3, reproduced on {PERIOD}\")\n", " star.custom_columns([\"Baseline\", gemini, gpt, gemini, gpt], [1, 1, 1, 1, 1])\n", " star.covariate_order([\"const\", S, G])\n", " star.rename_covariates({\"const\": \"Intercept\"})\n", @@ -272,21 +317,15 @@ "source": [ "## Where to go from here\n", "\n", - "The archive gives you the **inputs** — events and their disclosures. To turn this\n", - "into a backtest you'd:\n", + "The archive gives you the **inputs** \u2014 events and their disclosures. To turn this into a backtest you'd:\n", "\n", - "1. Pair each event with the realized next-day reaction for its focal asset, from\n", - " your own market-data source.\n", - "2. Convert that reaction into a percentile rank in `[0, 1]` **across the\n", - " quarter's cross-section** of scored event outcomes — a rank against all the\n", - " period's events, not against the asset's own history. That's the target the\n", - " competition scores; `examples.scoring.percentile_ranks` is the exact\n", - " transform (see section 5).\n", + "1. Pair each event with the realized next-day reaction for its focal asset, from your own market-data source.\n", + "2. Convert that reaction into a percentile rank in `[0, 1]` **across the quarter's cross-section** of scored event outcomes \u2014 a rank against all the period's events, not against the asset's own history. That's the target the competition scores; `examples.scoring.percentile_ranks` is the exact transform (see section 6).\n", "3. Score your model's predicted percentile against it offline, and iterate.\n", "\n", - "When your model is ready, move it into a deployed webhook handler (see the starter\n", - "repos) — that's what receives live events and submits predictions. These notebooks\n", - "deliberately stop short of `POST /predictions`." + "For the ex ante side of each event \u2014 the earnings preview \u2014 see [`03_earnings_previews.ipynb`](03_earnings_previews.ipynb).\n", + "\n", + "When your model is ready, move it into a deployed webhook handler (see the starter repos) \u2014 that's what receives live events and submits predictions. These notebooks deliberately stop short of `POST /predictions`." ] } ], diff --git a/notebooks/02_earnings_call_facts.ipynb b/notebooks/02_earnings_call_facts.ipynb index 2842a45..118a18f 100644 --- a/notebooks/02_earnings_call_facts.ipynb +++ b/notebooks/02_earnings_call_facts.ipynb @@ -7,10 +7,7 @@ "source": [ "# 02 \u2014 Earnings-call facts: where the disclosure comes from\n", "\n", - "Every realized `EARNINGS_RELEASE` event carries a `disclosure` block, and inside it an\n", - "item with `kind=\"facts\"` / `source=\"earnings_call\"` whose `content` is **ten sentences**\n", - "distilled from the company's earnings call. Those ten sentences are the substantive\n", - "input most agents build on, so it is worth knowing exactly how they are produced.\n", + "Every realized `EARNINGS_RELEASE` event carries a `disclosure` block, and inside it an item with `kind=\"facts\"` / `source=\"earnings_call\"` whose `content` is **ten sentences** distilled from the company's earnings call. Those ten sentences are the substantive input most agents build on, so it is worth knowing exactly how they are produced.\n", "\n", "This notebook walks the whole path:\n", "\n", @@ -22,13 +19,9 @@ "6. How extraction failures surface (`parse_note`)\n", "7. Optionally, run the real call yourself\n", "\n", - "`examples.summary` is an exact semantic port of the production summarizer, so the\n", - "request built here is the request the pipeline sends.\n", + "`examples.summary` is an exact semantic port of the production summarizer, so the request built here is the request the pipeline sends.\n", "\n", - "> **You cannot regenerate a competition disclosure with this.** The production input is\n", - "> a licensed full-length transcript that is not distributed with this repo, and extended\n", - "> thinking makes the call nondeterministic even on identical input. Everything below runs\n", - "> on a small **synthetic** transcript for a fictional company." + "> **You cannot regenerate a competition disclosure with this.** The production input is a licensed full-length transcript that is not distributed with this repo, and extended thinking makes the call nondeterministic even on identical input. Everything below runs on a small **synthetic** transcript for a fictional company." ] }, { @@ -73,14 +66,9 @@ "source": [ "## 1. What you receive\n", "\n", - "Start from the delivered side. Each archive line is an event payload; the facts live at\n", - "`disclosure.items[]` under the entry whose `kind` is `facts` and whose `source` is\n", - "`earnings_call`. `facts_from_disclosure` does that lookup and returns `[]` when an event\n", - "has no such item \u2014 a scheduled-but-not-yet-occurred event legitimately has none, and so\n", - "does a call whose extraction failed.\n", + "Start from the delivered side. Each archive line is an event payload; the facts live at `disclosure.items[]` under the entry whose `kind` is `facts` and whose `source` is `earnings_call`. `facts_from_disclosure` does that lookup and returns `[]` when an event has no such item \u2014 a scheduled-but-not-yet-occurred event legitimately has none, and so does a call whose extraction failed.\n", "\n", - "(The bundled sample is illustrative and carries fewer than ten facts; real archive lines\n", - "carry ten.)" + "(The bundled sample is illustrative and carries fewer than ten facts; real archive lines carry ten.)" ] }, { @@ -109,9 +97,7 @@ "source": [ "## 2. The artifact behind the pointer\n", "\n", - "Internally the pipeline publishes a small JSON artifact per event and the competition\n", - "dereferences it. Its shape is worth seeing, because it carries one field the disclosure\n", - "does not: `parse_note`, which records how cleanly the model's output parsed.\n", + "Internally the pipeline publishes a small JSON artifact per event and the competition dereferences it. Its shape is worth seeing, because it carries one field the disclosure does not: `parse_note`, which records how cleanly the model's output parsed.\n", "\n", "`facts` can legitimately be `null` \u2014 see section 6." ] @@ -143,14 +129,9 @@ "\n", "This is the part that most shapes what the facts can possibly contain.\n", "\n", - "`format_transcript` renders the raw transcript to markdown, and the **only** metadata it\n", - "prepends is the transcript's own header title. No ticker, no event date, no fiscal\n", - "period, no consensus estimate, and no market data are injected separately. There is also\n", - "**no truncation, chunking, or token budgeting** \u2014 the entire call goes into a single user\n", - "message.\n", + "`format_transcript` renders the raw transcript to markdown, and the **only** metadata it prepends is the transcript's own header title. No ticker, no event date, no fiscal period, no consensus estimate, and no market data are injected separately. There is also **no truncation, chunking, or token budgeting** \u2014 the entire call goes into a single user message.\n", "\n", - "So the summarizer knows what was said on the call, plus whatever the title reveals, and\n", - "nothing else." + "So the summarizer knows what was said on the call, plus whatever the title reveals, and nothing else." ] }, { @@ -176,10 +157,7 @@ "\n", "Printed in full \u2014 this is the literal text the model receives.\n", "\n", - "Note what the instructions ask for: single-sentence, **quantified**, investor-relevant,\n", - "drawn only from the transcript, and explicitly *not* verbatim quotation. The framing\n", - "target is anticipating the stock's reaction, which is why the facts skew toward numbers,\n", - "guidance, and surprises rather than narrative summary." + "Note what the instructions ask for: single-sentence, **quantified**, investor-relevant, drawn only from the transcript, and explicitly *not* verbatim quotation. The framing target is anticipating the stock's reaction, which is why the facts skew toward numbers, guidance, and surprises rather than narrative summary." ] }, { @@ -215,14 +193,9 @@ "\n", "Two things stand out.\n", "\n", - "**Extended thinking is on.** The model reasons before answering, and on current models\n", - "that is *adaptive* thinking at `effort=\"high\"` \u2014 no fixed token budget. Thinking is a\n", - "large part of why the extraction is nondeterministic.\n", + "**Extended thinking is on.** The model reasons before answering, and on current models that is *adaptive* thinking at `effort=\"high\"` \u2014 no fixed token budget. Thinking is a large part of why the extraction is nondeterministic.\n", "\n", - "**There is no structured-output machinery.** No `tools`, no JSON schema, no\n", - "`response_format`, no `temperature`, no stop sequences. The ten-fact JSON shape is\n", - "requested in prose and then *validated after the fact*. That is a deliberate design\n", - "choice with a visible consequence \u2014 section 6." + "**There is no structured-output machinery.** No `tools`, no JSON schema, no `response_format`, no `temperature`, no stop sequences. The ten-fact JSON shape is requested in prose and then *validated after the fact*. That is a deliberate design choice with a visible consequence \u2014 section 6." ] }, { @@ -246,11 +219,7 @@ "source": [ "## 6. How extraction failures surface\n", "\n", - "Because structure is enforced by prompt text rather than by the API, the response can\n", - "deviate. The pipeline applies only **safe, non-corrupting** repairs \u2014 it never fabricates\n", - "a fact, never splits one, and never pads to reach ten. Anything it cannot recover safely\n", - "becomes a failure, and the published artifact carries `\"facts\": null` rather than a\n", - "partial list.\n", + "Because structure is enforced by prompt text rather than by the API, the response can deviate. The pipeline applies only **safe, non-corrupting** repairs \u2014 it never fabricates a fact, never splits one, and never pads to reach ten. Anything it cannot recover safely becomes a failure, and the published artifact carries `\"facts\": null` rather than a partial list.\n", "\n", "The `parse_note` vocabulary below is exactly what you may observe." ] @@ -299,9 +268,7 @@ "source": [ "## 7. Run it yourself (optional, needs an Anthropic key)\n", "\n", - "This section calls the real API on the synthetic transcript above. It is **not** a\n", - "competition credential and nothing else in this repo needs it \u2014 set `ANTHROPIC_API_KEY`\n", - "in `.env` only if you want to watch the extraction happen.\n", + "This section calls the real API on the synthetic transcript above. It is **not** a competition credential and nothing else in this repo needs it \u2014 set `ANTHROPIC_API_KEY` in `.env` only if you want to watch the extraction happen.\n", "\n", "Skipped automatically when the key is absent, so the notebook still runs top-to-bottom." ] @@ -341,18 +308,12 @@ "source": [ "## What to take away\n", "\n", - "- The facts are a **quantified, investor-framed distillation of the call and nothing\n", - " else** \u2014 no market data, no estimates, no cross-company context. Anything beyond the\n", - " call is yours to add.\n", - "- They are ordered by the model's judgment of relevance, which is why over-length\n", - " responses are trimmed from the tail.\n", - "- Extraction can fail, and a failure is visible rather than silently patched. Build for\n", - " the case where an event's facts are absent.\n", - "- Running this yourself will **not** reproduce a competition disclosure: different\n", - " transcript, and a nondeterministic call.\n", - "\n", - "Next: [`01_historical_archive.ipynb`](01_historical_archive.ipynb) turns these events into\n", - "a scored analysis frame with the competition's exact scoring transform." + "- The facts are a **quantified, investor-framed distillation of the call and nothing else** \u2014 no market data, no estimates, no cross-company context. Anything beyond the call is yours to add.\n", + "- They are ordered by the model's judgment of relevance, which is why over-length responses are trimmed from the tail.\n", + "- Extraction can fail, and a failure is visible rather than silently patched. Build for the case where an event's facts are absent.\n", + "- Running this yourself will **not** reproduce a competition disclosure: different transcript, and a nondeterministic call.\n", + "\n", + "Next: [`01_historical_archive.ipynb`](01_historical_archive.ipynb) turns these events into a scored analysis frame with the competition's exact scoring transform, and [`03_earnings_previews.ipynb`](03_earnings_previews.ipynb) covers the *other* disclosure item \u2014 the pre-release earnings preview." ] } ], diff --git a/notebooks/03_earnings_previews.ipynb b/notebooks/03_earnings_previews.ipynb new file mode 100644 index 0000000..aa5ecb7 --- /dev/null +++ b/notebooks/03_earnings_previews.ipynb @@ -0,0 +1,347 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "ce6d489c", + "metadata": {}, + "source": [ + "# 03 \u2014 Earnings previews: the ex ante information set\n", + "\n", + "Notebook 02 covered the ten **facts** distilled from the earnings call. In addition to that summary of the call, most events will also include an **earnings preview**, an agent-written research note assembled from public sources *before* the release. This is meant to serve as a summary of the market's ex ante expectation about the announcement: consensus, guidance, what moved the stock last time, and what to watch.\n", + "\n", + "Each earnings preview is created by running [Anthropic's Earnings Preview Skill](https://github.com/anthropics/financial-services/blob/main/plugins/agent-plugins/earnings-reviewer/skills/earnings-preview/SKILL.md) prior to the start of the CAR1 measurement window surrounding the event. This is done programmatically in a sandbox such that Claude has full access to the standard Claude Code harness.\n", + "\n", + "Earnings previews sit in `disclosure.items[]` as `id=\"earnings-preview\"`, `kind=\"text\"`, `source=\"claude_code_web_research\"`, `media_type=\"text/markdown\"`, with `content` a single markdown string. The item is **optional** \u2014 events from before it was disseminated, and events for which no note was produced, have none.\n", + "\n", + "In this notebook, we will cover:\n", + "\n", + "1. What you receive \u2014 coverage in the 2026Q3 archive, and six events to explore\n", + "2. How a preview is built (at a high level)\n", + "3. When it is built \u2014 and the caveat for the 2026Q3 archive\n", + "4. Anatomy \u2014 the note's skeleton across the six\n", + "5. One note in full\n", + "6. The other five\n", + "7. Ex ante vs. ex post \u2014 preview, facts, surprise, and the realized reaction\n", + "\n", + "With a live API key it reads the real 2026Q3 archive; without one it uses six **real** archive lines bundled in `data/sample/` (see `data/README.md`), so everything below runs either way." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9cf73098", + "metadata": {}, + "outputs": [], + "source": [ + "from pathlib import Path\n", + "\n", + "import pandas as pd\n", + "from IPython.display import Markdown, display\n", + "\n", + "from examples import Client, load_config\n", + "from examples.archive import download_archive, read_jsonl_gz\n", + "from examples.preview import (\n", + " PREVIEW_PLUGIN,\n", + " PREVIEW_PLUGIN_REPO,\n", + " PREVIEW_SKILL,\n", + " PREVIEW_SKILL_PATH,\n", + " earnings_preview_from_disclosure,\n", + " preview_display_markdown,\n", + " preview_sections,\n", + " preview_title,\n", + " previews_frame,\n", + ")\n", + "from examples.scoring import outcomes_frame\n", + "from examples.summary import facts_from_disclosure\n", + "\n", + "REPO = next(p for p in [Path.cwd(), *Path.cwd().parents] if (p / \"data\" / \"sample\").is_dir())\n", + "ARCHIVE_DIR = REPO / \"data\" / \"archive\" # gitignored download cache\n", + "SAMPLE_DIR = REPO / \"data\" / \"sample\"\n", + "\n", + "QUARTER = \"2026Q3\" # the first quarter with previews\n", + "TICKERS = [\"NVDA\", \"WMT\", \"HD\", \"CRM\", \"AVGO\", \"ORCL\"]\n", + "\n", + "config = load_config()\n", + "print(\"Mode:\", \"LIVE\" if config.is_live else \"SAMPLE (no EM_API_KEY \u2014 using bundled data)\")" + ] + }, + { + "cell_type": "markdown", + "id": "b575b353", + "metadata": {}, + "source": [ + "## 1. What you receive\n", + "\n", + "Previews start partway through 2026Q3 \u2014 the archive quarter in which dissemination began \u2014 so earlier events, and the first weeks of that quarter, carry none. Load the one file and count.\n", + "\n", + "In live mode this pulls just the 2026Q3 file. The quarter is still unsealed, so the file grows as events are scored; `download_archive` compares sizes against the manifest and re-downloads when the cache is stale." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "70976f11", + "metadata": {}, + "outputs": [], + "source": [ + "if config.is_live:\n", + " with Client.from_env() as client:\n", + " manifest = client.archive_manifest()\n", + " (path,) = download_archive(\n", + " manifest,\n", + " ARCHIVE_DIR,\n", + " client=client,\n", + " only=lambda f: f.event_type == \"EARNINGS_RELEASE\" and f.quarter == QUARTER,\n", + " )\n", + "else:\n", + " path = SAMPLE_DIR / f\"archive_EARNINGS_RELEASE_{QUARTER}.jsonl.gz\"\n", + " print(f\"Sample mode: reading {path.name}\")\n", + "\n", + "records = list(read_jsonl_gz(path))\n", + "coverage = previews_frame(records)\n", + "\n", + "n_with = int(coverage[\"has_preview\"].sum())\n", + "print(f\"{len(records):,} events in {QUARTER}; {n_with:,} carry a preview\")\n", + "if n_with:\n", + " first = coverage.loc[coverage[\"has_preview\"], \"event_datetime\"].min()\n", + " print(f\"Earliest event with a preview: {first:%Y-%m-%d}\")" + ] + }, + { + "cell_type": "markdown", + "id": "895d7bda", + "metadata": {}, + "source": [ + "Now the six events we will read. Each is the ticker's one 2026Q3 event (the latest, should a ticker ever have more than one in a quarter)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "00413900", + "metadata": {}, + "outputs": [], + "source": [ + "def latest_record(ticker: str) -> dict | None:\n", + " hits = [r for r in records if r[\"focal_assets\"][0][\"identifier_value\"] == ticker]\n", + " return max(hits, key=lambda r: r[\"event_datetime\"]) if hits else None\n", + "\n", + "\n", + "chosen = {t: latest_record(t) for t in TICKERS}\n", + "missing = [t for t, r in chosen.items() if r is None]\n", + "if missing:\n", + " print(f\"Not in this file: {missing}\")\n", + "chosen = {t: r for t, r in chosen.items() if r is not None}\n", + "\n", + "overview = previews_frame(chosen.values()).set_index(\"tickers\").loc[list(chosen)]\n", + "display(\n", + " overview[[\"event_id\", \"event_datetime\", \"knowledge_cutoff\", \"has_preview\", \"preview_chars\"]]\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "320280c9", + "metadata": {}, + "source": [ + "## 2. How a preview is built\n", + "\n", + "The competition does not write the preview itself. It runs Anthropic's open **`earnings-reviewer` plugin** \u2014 specifically its **`earnings-preview` skill** \u2014 programmatically in a sandbox, where Claude has full access to the standard Claude Code harness. The plugin is pinned to one commit of the public repository below so that every note in a period comes from the same skill text.\n", + "\n", + "The agent receives a one-line request naming the company, its ticker, the fiscal period and the scheduled event time, and researches the rest itself on the public web. Nothing else is injected: no transcript, no consensus feed, no market data, no competition data. Whatever the note knows, the agent went and found \u2014 and the skill instructs it to cite the source and date of each estimate, to say when something (an options-implied move, a whisper number) could not be found, and never to estimate in its place.\n", + "\n", + "The skill prescribes the note's shape, which is why the six read alike:\n", + "\n", + "- the company, quarter and event date up top\n", + "- a **consensus estimates** table (and the company's own guidance)\n", + "- the **key metrics to watch**, ranked\n", + "- a **bull / base / bear** scenario table\n", + "- a **catalyst checklist** for the call\n", + "- the **trading setup**: recent performance, historical earnings reactions, the options-implied move, positioning\n", + "\n", + "In production the skill runs on Claude Opus 5 at high effort. Scheduling and infrastructure are out of scope here \u2014 the point is that the note is a public-sources research product with a fixed brief, not a proprietary signal." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "07a29b5a", + "metadata": {}, + "outputs": [], + "source": [ + "print(f\"plugin : {PREVIEW_PLUGIN}\")\n", + "print(f\"skill : {PREVIEW_SKILL}\")\n", + "print(f\"source : {PREVIEW_PLUGIN_REPO}\")\n", + "print(f\" {PREVIEW_SKILL_PATH}\")" + ] + }, + { + "cell_type": "markdown", + "id": "bb61ba3c", + "metadata": {}, + "source": [ + "## 3. When it is built\n", + "\n", + "Every event has a `knowledge_cutoff` (notebook 00): the instant after which an agent must not use information, and the moment the one-day abnormal-return window (`car1`) opens \u2014 the prior close for a pre-market reporter, the same-day close for an after-close reporter.\n", + "\n", + "**Going forward, a disseminated preview is assembled before that cutoff**, so it is a clean ex ante object: nothing in it post-dates the start of the measurement window.\n", + "\n", + "**The 2026Q3 archive is the exception.** Those previews were produced live, ahead of each release, and attached to the archive afterwards \u2014 but the run was not yet keyed to the cutoff, so a note for a pre-market reporter may have been finished the evening before or the morning of, after the prior close. No timestamps travel with the item, so you cannot tell which from the archive alone \u2014 though a note sometimes says so itself (WMT's below opens with a timing caveat that the release is minutes away). Treat the 2026Q3 previews as illustrative of what you will receive, not as strictly cutoff-clean inputs for a backtest." + ] + }, + { + "cell_type": "markdown", + "id": "03c9d044", + "metadata": {}, + "source": [ + "## 4. Anatomy\n", + "\n", + "`preview_title` and `preview_sections` pull the `#` title and the `##` headings, which is enough to see the skill's skeleton \u2014 and how far each note departs from it." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1c8e1df5", + "metadata": {}, + "outputs": [], + "source": [ + "previews = {t: earnings_preview_from_disclosure(r) for t, r in chosen.items()}\n", + "previews = {t: p for t, p in previews.items() if p is not None}\n", + "\n", + "rows = [\n", + " {\"ticker\": t, \"title\": preview_title(p), \"n\": i, \"section\": s}\n", + " for t, p in previews.items()\n", + " for i, s in enumerate(preview_sections(p), start=1)\n", + "]\n", + "pd.set_option(\"display.max_colwidth\", 80)\n", + "display(pd.DataFrame(rows).set_index([\"ticker\", \"title\", \"n\"]))" + ] + }, + { + "cell_type": "markdown", + "id": "a1e35a8f", + "metadata": {}, + "source": [ + "The common spine \u2014 consensus, key metrics, scenarios, catalysts, trading setup \u2014 is the skill's brief. The extras are the agent's judgment about what *this* event turns on: a \"setup\" paragraph, a section of guidance arithmetic, a data-quality note.\n", + "\n", + "## 5. One note in full\n", + "\n", + "NVIDIA's Q2 FY2027 preview. (Dollar signs are escaped for display only \u2014 the disseminated string is plain markdown.)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d6a754d6", + "metadata": {}, + "outputs": [], + "source": [ + "def show(ticker: str) -> None:\n", + " record, preview = chosen[ticker], previews[ticker]\n", + " display(\n", + " Markdown(\n", + " f\"---\\n**{ticker}** \u00b7 `{record['event_id']}` \u00b7 event {record['event_datetime']} \u00b7 \"\n", + " f\"knowledge cutoff {record['knowledge_cutoff']} \u00b7 {len(preview):,} chars\\n\\n\"\n", + " + preview_display_markdown(preview)\n", + " )\n", + " )\n", + "\n", + "\n", + "show(\"NVDA\")" + ] + }, + { + "cell_type": "markdown", + "id": "3eca18dd", + "metadata": {}, + "source": [ + "## 6. The other five\n", + "\n", + "Read them for the range: a chip name where the note argues the next-quarter guide is the whole event (AVGO), two retailers whose prints are a read on the consumer (WMT, HD), and two software names where the note says the P&L is not the point \u2014 CRM's guidance arithmetic, ORCL's backlog and capex." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2ac15f09", + "metadata": {}, + "outputs": [], + "source": [ + "for ticker in previews:\n", + " if ticker != \"NVDA\":\n", + " show(ticker)" + ] + }, + { + "cell_type": "markdown", + "id": "ba08f4fb", + "metadata": {}, + "source": [ + "## 7. Ex ante vs. ex post\n", + "\n", + "Each archive line lets you put the three pieces side by side:\n", + "\n", + "- the **preview** \u2014 what a careful reader could know going in,\n", + "- the **facts** \u2014 what the company then disclosed on the call, and\n", + "- the **reaction** \u2014 the earnings surprise and the realized one-day abnormal return (`car1`), which is what the competition scores.\n", + "\n", + "`examples.scoring.outcomes_frame` pulls the surprise and `car1` per (event, asset); we deliberately do **not** call `add_percentiles` here \u2014 the competition ranks across a whole quarter, and percentiles over six events would be meaningless." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "833eb700", + "metadata": {}, + "outputs": [], + "source": [ + "outcomes = outcomes_frame(chosen.values()) # one row per (event, asset) that has a car1\n", + "outcomes = (\n", + " outcomes.set_index(\"identifier_value\")[[\"surprise\", \"car1\"]]\n", + " if not outcomes.empty\n", + " else pd.DataFrame(columns=[\"surprise\", \"car1\"])\n", + ")\n", + "\n", + "table = pd.DataFrame(\n", + " {\n", + " \"event_datetime\": {t: r[\"event_datetime\"] for t, r in chosen.items()},\n", + " \"knowledge_cutoff\": {t: r[\"knowledge_cutoff\"] for t, r in chosen.items()},\n", + " \"preview_chars\": {t: len(previews.get(t) or \"\") for t in chosen},\n", + " \"n_facts\": {t: len(facts_from_disclosure(r)) for t, r in chosen.items()},\n", + " }\n", + ").join(outcomes, how=\"left\") # a very recent live event may have no car1 yet\n", + "\n", + "display(table.style.format({\"surprise\": \"{:+.4f}\", \"car1\": \"{:+.2%}\"}, na_rep=\"\\u2014\"))" + ] + }, + { + "cell_type": "markdown", + "id": "cd6d289d", + "metadata": {}, + "source": [ + "Pick one and read the three together \u2014 for CRM, say, the preview's \"what would it take\" arithmetic, the facts that answered it, and a double-digit `car1` \u2014 and you have the competition's problem in miniature: given the expectation and the disclosure, where in the quarter's cross-section will the reaction land?\n", + "\n", + "## What to take away\n", + "\n", + "- The preview is **optional** and arrived in 2026Q3. Match on `id` / `kind`, never on position, and build for its absence.\n", + "- It is a **public-sources research note with a fixed brief** \u2014 Anthropic's open `earnings-preview` skill, run by Claude in a sandboxed Claude Code session \u2014 not a model prediction and not a proprietary feed. Its sources and dates are cited inline.\n", + "- Going forward it is assembled **before the CAR1 window opens**; the 2026Q3 archive previews are not guaranteed to be, so treat them as illustrative.\n", + "- The facts (notebook 02) are the ex post disclosure; the preview is the ex ante expectation; `car1` is the reaction. `examples.scoring` turns the last into the percentile target the competition scores." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "examples (3.11.14)", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.11.14" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/src/examples/__init__.py b/src/examples/__init__.py index 052a377..6802faa 100644 --- a/src/examples/__init__.py +++ b/src/examples/__init__.py @@ -13,6 +13,7 @@ archive — download, cache, and load the historical event archive scoring — the competition's exact percentile scoring transform summary — the exact 10-fact extraction behind each event's disclosure + preview — the agent-written earnings preview attached from 2026Q3 onward """ from __future__ import annotations diff --git a/src/examples/preview.py b/src/examples/preview.py new file mode 100644 index 0000000..e793b3a --- /dev/null +++ b/src/examples/preview.py @@ -0,0 +1,149 @@ +"""The earnings preview: the ex ante information set attached to an event. + +From 2026Q3 onward, an ``EARNINGS_RELEASE`` event's ``disclosure.items[]`` may carry a +second item next to the ten earnings-call facts: ``id="earnings-preview"`` +(``kind="text"``, ``source="claude_code_web_research"``, ``media_type="text/markdown"``). +Its ``content`` is ONE markdown string — an agent-written research note assembled from +public sources *before* the release, which is why it stands in for the market's ex +ante expectation about the announcement. It is optional: events from before it was +disseminated, and events for which no preview was produced, legitimately have none. + +How it is built, at a high level: the competition runs Anthropic's open +``earnings-reviewer`` plugin — specifically its ``earnings-preview`` skill — +programmatically in a sandbox where Claude has the standard Claude Code harness, +pinned to a fixed commit of :data:`PREVIEW_PLUGIN_REPO`. The agent gets a one-line +request naming the company, ticker, fiscal period and scheduled event time and +researches the rest on the public web. No transcript, no consensus feed and no +competition data are injected; whatever the note knows, it found and cited itself. +The skill prescribes the note's shape (consensus estimates, key metrics to watch, +bull/base/bear scenarios, a catalyst checklist and the trading setup), and the ``##`` +headings you see in :func:`preview_sections` are that skeleton with per-report +variation. + +Timing: a preview disseminated live is assembled before the event's CAR1 measurement +window opens — that is, before its ``knowledge_cutoff``. The 2026Q3 archive is the +exception: those previews were produced ahead of each release and attached to the +archive afterwards, but not always before the knowledge cutoff, so treat them as +illustrative rather than strictly cutoff-clean. No timestamps, model or job +identifiers are disseminated with the item. +""" + +from __future__ import annotations + +import re +from collections.abc import Iterable + +import pandas as pd + +from examples.summary import facts_from_disclosure + +EARNINGS_PREVIEW_ID = "earnings-preview" +EARNINGS_PREVIEW_SOURCE = "claude_code_web_research" +TEXT_KIND = "text" + +# Provenance of the note, for display. The plugin is public; the pipeline pins one +# commit of it so every preview in a period comes from the same skill text. +PREVIEW_PLUGIN = "earnings-reviewer" +PREVIEW_SKILL = "earnings-preview" +PREVIEW_PLUGIN_REPO = "https://github.com/anthropics/financial-services" +PREVIEW_SKILL_PATH = "plugins/agent-plugins/earnings-reviewer/skills/earnings-preview/SKILL.md" + +_H1 = re.compile(r"^#\s+(.+?)\s*$", re.MULTILINE) +_H2 = re.compile(r"^##\s+(.+?)\s*$", re.MULTILINE) +_LEADING_NUMBER = re.compile(r"^\d+[.)]?\s*") + +# Columns, in reading order, for the coverage table. +_PREVIEW_COLUMNS = [ + "event_datetime", + "knowledge_cutoff", + "quarter", + "tickers", + "event_id", + "n_facts", + "has_preview", + "preview_chars", + "n_sections", +] + + +def earnings_preview_from_disclosure(record: dict) -> str | None: + """Pull the agent-written earnings preview out of an event or archive record. + + Looks for the ``disclosure.items[]`` entry with ``id="earnings-preview"`` and + ``kind="text"`` and returns its markdown ``content``. Returns ``None`` when the + record carries no such item — the preview is optional, and events from before + it was disseminated legitimately have none. + + ``record`` is a raw archive line (as yielded by + :func:`examples.archive.read_jsonl_gz`), an equivalent event payload, or a row + of :func:`examples.archive.load_archive` turned back into a dict — where a + missing ``disclosure`` shows up as ``NaN`` rather than ``None``, hence the type + checks rather than ``or {}``. + """ + disclosure = record.get("disclosure") + items = disclosure.get("items") if isinstance(disclosure, dict) else None + for item in items if isinstance(items, list) else []: + if item.get("id") == EARNINGS_PREVIEW_ID and item.get("kind") == TEXT_KIND: + content = item.get("content") + return content if isinstance(content, str) and content else None + return None + + +def preview_title(preview_md: str) -> str | None: + """The note's ``#`` heading, e.g. ``"NVIDIA (NVDA) — Q2 FY2027 Earnings Preview"``.""" + match = _H1.search(preview_md) + return match.group(1) if match else None + + +def preview_sections(preview_md: str) -> list[str]: + """The note's ``##`` headings in order, with any ``1.`` / ``1)`` numbering stripped. + + Useful for seeing the skill's prescribed skeleton — and how much each report + departs from it — without reading every note in full. + """ + return [_LEADING_NUMBER.sub("", h) for h in _H2.findall(preview_md)] + + +def preview_display_markdown(preview_md: str) -> str: + """Escape ``$`` so a notebook renders dollar amounts as text, not as math. + + Previews are full of prices and estimates; JupyterLab's markdown renderer would + otherwise read ``$1.20 vs $1.25`` as an inline MathJax span. Use this only for + display — the disseminated content is the unescaped string. + """ + return preview_md.replace("$", r"\$") + + +def previews_frame(records: Iterable[dict]) -> pd.DataFrame: + """One row per record: does it carry a preview, and how big is it? + + Accepts raw archive lines or ``load_archive`` rows (``to_dict("records")``). + ``quarter`` is taken from the record when present (``load_archive`` adds it from + the filename) so coverage can be grouped by quarter; ``tickers`` is the + comma-joined focal assets, as in :func:`examples.frames.events_frame`. + """ + rows = [] + for record in records: + preview = earnings_preview_from_disclosure(record) + focal = record.get("focal_assets") + rows.append( + { + "event_datetime": record.get("event_datetime"), + "knowledge_cutoff": record.get("knowledge_cutoff"), + "quarter": record.get("quarter"), + "tickers": ", ".join( + a["identifier_value"] for a in (focal if isinstance(focal, list) else []) + ), + "event_id": record.get("event_id"), + "n_facts": len(facts_from_disclosure(record)), + "has_preview": preview is not None, + "preview_chars": len(preview) if preview else 0, + "n_sections": len(preview_sections(preview)) if preview else 0, + } + ) + df = pd.DataFrame(rows, columns=_PREVIEW_COLUMNS) + if not df.empty: + df["event_datetime"] = pd.to_datetime(df["event_datetime"], utc=True, format="ISO8601") + df["knowledge_cutoff"] = pd.to_datetime(df["knowledge_cutoff"], utc=True, format="ISO8601") + df = df.sort_values("event_datetime").reset_index(drop=True) + return df diff --git a/src/examples/schemas.py b/src/examples/schemas.py index 622f96e..95c8bab 100644 --- a/src/examples/schemas.py +++ b/src/examples/schemas.py @@ -47,9 +47,15 @@ class DisclosureItem(_Model): For earnings calls the item to look for is ``kind="facts"`` with ``source="earnings_call"``: its ``content`` is the ten-sentence summary whose - provenance is documented in :mod:`examples.summary`. Both fields are open - string sets, like ``event_type`` — new kinds and sources can appear without - breaking parsing, so match on them rather than assuming a single item. + provenance is documented in :mod:`examples.summary`. A second item, + ``id="earnings-preview"`` (``kind="text"``, ``source="claude_code_web_research"``, + ``media_type="text/markdown"``), carries an agent-written pre-release research + note as ONE markdown string; it may be absent, and :mod:`examples.preview` + documents where it comes from. ``content``'s JSON type follows + ``kind`` (``facts`` → list of strings, ``text`` → string). All of these are + open string sets, like ``event_type`` — new kinds and sources can appear + without breaking parsing, so match on them rather than assuming a single + item or a position. Larger or binary artifacts are referenced rather than inlined, via ``media_type`` / ``url`` / ``bytes`` / ``sha256``. @@ -58,7 +64,7 @@ class DisclosureItem(_Model): id: str | None = None kind: str source: str | None = None - content: list[str] = [] + content: list[str] | str | None = None media_type: str | None = None url: str | None = None bytes: int | None = None diff --git a/src/examples/summary.py b/src/examples/summary.py index c9b8552..eaaa7f5 100644 --- a/src/examples/summary.py +++ b/src/examples/summary.py @@ -377,10 +377,14 @@ def facts_from_disclosure(record: dict) -> list[str]: a call whose summarization failed, legitimately has none. ``record`` is a raw archive line (as yielded by - :func:`examples.archive.read_jsonl_gz`) or an equivalent event payload. + :func:`examples.archive.read_jsonl_gz`), an equivalent event payload, or a + ``load_archive`` row as a dict (where a missing ``disclosure`` is ``NaN``, not + ``None`` — hence the type checks). The second disclosure item, the earnings + preview, has its own reader in :mod:`examples.preview`. """ - items = (record.get("disclosure") or {}).get("items") or [] - for item in items: + disclosure = record.get("disclosure") + items = disclosure.get("items") if isinstance(disclosure, dict) else None + for item in items if isinstance(items, list) else []: if item.get("kind") == FACTS_KIND and item.get("source") == EARNINGS_CALL_SOURCE: return list(item.get("content") or []) return [] diff --git a/tests/conftest.py b/tests/conftest.py index 343db81..00cc696 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -13,6 +13,7 @@ SAMPLE_DIR = REPO_ROOT / "data" / "sample" SAMPLE_EVENTS = SAMPLE_DIR / "events_sample.json" SAMPLE_ARCHIVE = SAMPLE_DIR / "archive_EARNINGS_RELEASE_2025Q3.jsonl.gz" +SAMPLE_ARCHIVE_2026Q3 = SAMPLE_DIR / "archive_EARNINGS_RELEASE_2026Q3.jsonl.gz" SAMPLE_TRANSCRIPT = SAMPLE_DIR / "transcript_sample.json" SAMPLE_SUMMARY = SAMPLE_DIR / "summary_sample.json" diff --git a/tests/test_archive.py b/tests/test_archive.py index 9454ecf..6857a60 100644 --- a/tests/test_archive.py +++ b/tests/test_archive.py @@ -13,7 +13,7 @@ from examples.client import Client from examples.config import Config from examples.schemas import ArchiveManifest -from tests.conftest import SAMPLE_ARCHIVE, TEST_BASE_URL +from tests.conftest import SAMPLE_ARCHIVE, SAMPLE_DIR, TEST_BASE_URL def _gz_bytes(records: list[dict]) -> bytes: @@ -39,6 +39,17 @@ def test_load_archive_adds_provenance_from_filename() -> None: assert set(df["event_type"]) == {"EARNINGS_RELEASE"} +def test_load_archive_directory_concatenates_both_samples() -> None: + # The synthetic 2025Q3 file and the real 2026Q3 file differ in columns; the + # union is taken and the missing cells are NaN. + df = load_archive(SAMPLE_DIR) + assert len(df) == 11 + assert set(df["quarter"]) == {"2025Q3", "2026Q3"} + assert set(df["event_type"]) == {"EARNINGS_RELEASE"} + assert df.loc[df["quarter"] == "2025Q3", "event_returns"].isna().all() + assert df.loc[df["quarter"] == "2026Q3", "event_returns"].notna().all() + + def test_load_archive_empty_dir(tmp_path: Path) -> None: assert load_archive(tmp_path).empty diff --git a/tests/test_preview.py b/tests/test_preview.py new file mode 100644 index 0000000..9107261 --- /dev/null +++ b/tests/test_preview.py @@ -0,0 +1,152 @@ +"""The earnings-preview disclosure item and its helpers.""" + +from __future__ import annotations + +from examples.archive import load_archive, read_jsonl_gz +from examples.preview import ( + EARNINGS_PREVIEW_ID, + EARNINGS_PREVIEW_SOURCE, + earnings_preview_from_disclosure, + preview_display_markdown, + preview_sections, + preview_title, + previews_frame, +) +from examples.schemas import Disclosure +from examples.summary import facts_from_disclosure +from tests.conftest import SAMPLE_ARCHIVE_2026Q3 + +SAMPLE_TICKERS = ["NVDA", "WMT", "HD", "CRM", "AVGO", "ORCL"] + +FACTS_ITEM = { + "id": "earnings-call-facts", + "kind": "facts", + "source": "earnings_call", + "content": ["f"], +} +PREVIEW_ITEM = { + "id": EARNINGS_PREVIEW_ID, + "kind": "text", + "source": EARNINGS_PREVIEW_SOURCE, + "media_type": "text/markdown", + "content": "# Preview\n\nConsensus vs. guidance...", +} +PREVIEW_MD = """# Acme (ACME) — Q2 FY2026 Earnings Preview + +**Event:** after close. + +## 1. Consensus Estimates +| EPS | $1.20 | + +## 2) Key Metrics to Watch — Ranked +text + +## Scenario Analysis +text +""" + + +def test_earnings_preview_from_disclosure_reads_text_item_or_none() -> None: + record = {"disclosure": {"items": [FACTS_ITEM, PREVIEW_ITEM]}} + preview = earnings_preview_from_disclosure(record) + assert preview is not None and preview.startswith("# Preview") + assert facts_from_disclosure(record) == ["f"] + + assert earnings_preview_from_disclosure({"disclosure": {"items": [FACTS_ITEM]}}) is None + assert earnings_preview_from_disclosure({}) is None + # Empty content is treated as absent. + empty = {"disclosure": {"items": [{**PREVIEW_ITEM, "content": ""}]}} + assert earnings_preview_from_disclosure(empty) is None + # A load_archive row over mixed files carries NaN, not None, for a missing field. + assert earnings_preview_from_disclosure({"disclosure": float("nan")}) is None + + +def test_disclosure_schema_accepts_string_content_items() -> None: + parsed = Disclosure.model_validate( + { + "schema_version": "1.0", + "items": [{**FACTS_ITEM, "content": ["f1", "f2"]}, PREVIEW_ITEM], + } + ) + assert parsed.items[0].content == ["f1", "f2"] + assert parsed.items[1].content == PREVIEW_ITEM["content"] + + +def test_preview_title_and_sections() -> None: + assert preview_title(PREVIEW_MD) == "Acme (ACME) — Q2 FY2026 Earnings Preview" + assert preview_sections(PREVIEW_MD) == [ + "Consensus Estimates", + "Key Metrics to Watch — Ranked", + "Scenario Analysis", + ] + assert preview_title("no headings here") is None + assert preview_sections("no headings here") == [] + + +def test_preview_display_markdown_escapes_dollars() -> None: + assert preview_display_markdown("EPS $1.20 vs $1.25") == r"EPS \$1.20 vs \$1.25" + assert preview_display_markdown("no money") == "no money" + + +def test_previews_frame_marks_presence_and_tolerates_nan() -> None: + with_preview = { + "event_id": "e1", + "event_datetime": "2026-08-26T21:00:00+00:00", + "knowledge_cutoff": "2026-08-26T20:00:00+00:00", + "quarter": "2026Q3", + "focal_assets": [{"identifier_type": "TICKER", "identifier_value": "NVDA"}], + "disclosure": {"items": [FACTS_ITEM, {**PREVIEW_ITEM, "content": PREVIEW_MD}]}, + } + without = { + "event_id": "e0", + "event_datetime": "2026-07-01T12:00:00+00:00", + "knowledge_cutoff": float("nan"), + "quarter": "2026Q3", + "focal_assets": [{"identifier_type": "TICKER", "identifier_value": "STZ"}], + "disclosure": float("nan"), + } + df = previews_frame([with_preview, without]) + + assert list(df.columns) == [ + "event_datetime", + "knowledge_cutoff", + "quarter", + "tickers", + "event_id", + "n_facts", + "has_preview", + "preview_chars", + "n_sections", + ] + # Sorted by event time, so the July event comes first. + assert list(df["tickers"]) == ["STZ", "NVDA"] + assert list(df["has_preview"]) == [False, True] + assert list(df["preview_chars"]) == [0, len(PREVIEW_MD)] + assert list(df["n_sections"]) == [0, 3] + assert list(df["n_facts"]) == [0, 1] + assert df["event_datetime"].is_monotonic_increasing + assert df["knowledge_cutoff"].isna().iloc[0] + + +def test_previews_frame_empty() -> None: + df = previews_frame([]) + assert df.empty + assert list(df.columns)[:2] == ["event_datetime", "knowledge_cutoff"] + + +def test_bundled_2026q3_sample_carries_a_preview_per_event() -> None: + records = list(read_jsonl_gz(SAMPLE_ARCHIVE_2026Q3)) + assert [r["focal_assets"][0]["identifier_value"] for r in records] == SAMPLE_TICKERS + for record in records: + preview = earnings_preview_from_disclosure(record) + assert preview is not None and preview_title(preview) is not None + assert len(facts_from_disclosure(record)) == 10 + # Real archive lines: scored, with the realized reaction and the surprise. + assert record["status"] == "scored" + assert "car1" in record["event_returns"][record["focal_assets"][0]["identifier_value"]] + + df = load_archive(SAMPLE_ARCHIVE_2026Q3) + assert set(df["quarter"]) == {"2026Q3"} + frame = previews_frame(df.to_dict("records")) + assert frame["has_preview"].all() + assert (frame["n_sections"] >= 4).all() diff --git a/tests/test_summary.py b/tests/test_summary.py index 4956ad7..7135583 100644 --- a/tests/test_summary.py +++ b/tests/test_summary.py @@ -360,3 +360,10 @@ def test_sample_summary_matches_the_published_artifact_shape() -> None: recovered, note = recover_facts(json.dumps({"facts": facts})) assert recovered == facts assert note == "strict" + + +def test_facts_from_disclosure_tolerates_nan_disclosure() -> None: + # A load_archive row over mixed files carries float NaN, not None, where a + # record lacks the field. + assert facts_from_disclosure({"disclosure": float("nan")}) == [] + assert facts_from_disclosure({"disclosure": {"items": float("nan")}}) == []