Skip to content

docs(experiments): one ledger for all experiments + list/due/index CLI - #391

Merged
ohld merged 3 commits into
productionfrom
docs/experiments-ledger
Oct 31, 2026
Merged

ohld merged 3 commits into
productionfrom
docs/experiments-ledger

Conversation

@ohld

@ohld ohld commented Oct 31, 2026

Copy link
Copy Markdown
Member

What

Experiment records are in one place now: experiments/YYYY-MM-DD-slug.md, one file per experiment, flat folder. Status lives in front matter (id, status, hypothesis, primary_metric, guardrails, unit, variants, started, check_on, decided, result, learning, code, readout), so files never move.

Open points

  • result: unknown where no readout exists: early_channel_popup (shipped with no readout) and crossposting_forward_intent_ranker (never built). Every running/proposed experiment is also unknown.
  • phash_shadow_v1 has no check date or ship rule, so due flags it as NO DATE.
  • monthly_reminder_v1 is proposed because MONTHLY_REMINDER_ENABLED defaults to false. Set it to running when the flag goes on.
  • The friend challenge readout SQL may be deleted by the PR that removes the code. The record keeps the path for history.

Tests

pytest --noconftest tests/test_experiments_ledger.py: 2 passed. With a hand-edited README the test fails. ruff check passes on src/ tests/ scripts/.

ohld added 3 commits October 31, 2026 13:14
One file per experiment in experiments/ with flat front matter (status,
check_on, result, learning, readout). active/ and completed/ are merged,
HYPOTHESES.md H-sections and the recent unrecorded experiments get their
own files, and experiments/README.md is generated by
scripts/experiments.py index. A test checks every file parses and the
README is current.
@ohld
ohld merged commit 5c88175 into production Oct 31, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant