Repository navigation
docs(experiments): one ledger for all experiments + list/due/index CLI - #391
Merged
Merged
Conversation
One file per experiment in experiments/ with flat front matter (status, check_on, result, learning, readout). active/ and completed/ are merged, HYPOTHESES.md H-sections and the recent unrecorded experiments get their own files, and experiments/README.md is generated by scripts/experiments.py index. A test checks every file parses and the README is current.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Experiment records are in one place now:
experiments/YYYY-MM-DD-slug.md, one file per experiment, flat folder. Status lives in front matter (id, status, hypothesis, primary_metric, guardrails, unit, variants, started, check_on, decided, result, learning, code, readout), so files never move.active/+completed/(git mv, front matter added, bodies kept). New files for H2, H3, H3b, H3c, H4, H5, H7, H9, H10, H11, H13 fromdocs/growth/HYPOTHESES.md(section text moved verbatim; H1/H6/H8 appended to their existing files). Also new:channel_hits_v1,first_meme_nudge,meme_like_count(numbers from the 2026-10-30 closure),friend_challenge_h12(killed: 0 participations),monthly_reminder_v1(split out of the cadence holdout note),onboarding_default_language_v1(fix(onboarding): every new user gets a meme on /start #385),raw_impr_rank_reaction_rate(feat(stats): rank memes within source by reaction rate, not views #384),phash_shadow_v1(feat(dedup): perceptual hash (dHash) duplicate detection in shadow mode #381),ocr_dedup_watermark_fix(fix(dedup): strip watermark and require 0.6 similarity for OCR duplicates #380/fix(ocr): narrow watermark regex, unbias boolean prompt example #382 + restore).scripts/experiments.py(stdlib):list [--status],due,index(regeneratesexperiments/README.md: how to add/close + template, then Open with check dates and Decided with result + learning, newest first),readout <id>(prints thepsql "$ANALYST_DATABASE_URL" -f ...command; does not run it).tests/test_experiments_ledger.py: every file parses with required keys, a valid status and ISO dates; README equalsindexoutput.docs/growth/HYPOTHESES.mdandexperiments/HYPOTHESES.md: reduced to a pointer / removed. Growth bar moved into the README header; the healthy-snapshot lines moved todocs/analyst/README.md. Links updated in AGENTS.md, SPEC.md, TODOS.md, docs/, specs/ and src comments. Historical readouts still link HYPOTHESES.md, which now points to the ledger.Open points
result: unknownwhere no readout exists:early_channel_popup(shipped with no readout) andcrossposting_forward_intent_ranker(never built). Every running/proposed experiment is alsounknown.phash_shadow_v1has no check date or ship rule, sodueflags it as NO DATE.monthly_reminder_v1isproposedbecauseMONTHLY_REMINDER_ENABLEDdefaults to false. Set it torunningwhen the flag goes on.Tests
pytest --noconftest tests/test_experiments_ledger.py: 2 passed. With a hand-edited README the test fails.ruff checkpasses on src/ tests/ scripts/.