diff --git a/AGENTS.md b/AGENTS.md index c3903c13..107d4cd0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,6 +26,12 @@ - User reactions are persisted through `create_user_meme_reaction` / `update_user_meme_reaction` (see [`src/recommendations/service.py`](src/recommendations/service.py)). These records drive stats aggregation and recommendation eligibility. - Share attribution: deep links `s_{user_id}_{meme_id}` in `user_deep_link_log` — see `CONTEXT.md` **Share Attribution**. Growth thesis and measurement plan: [`docs/growth/virality-loop.md`](docs/growth/virality-loop.md). +## Experiments +- Experiments live in [`experiments/`](experiments/), one file each with flat front matter (id, status, check_on, result, learning, readout, ...). +- Before designing an experiment, read [`experiments/README.md`](experiments/README.md): open checks plus every decided result and learning. +- When starting or closing one, edit its file, then run `python scripts/experiments.py index` and commit the README (CI fails if it is stale). +- `python scripts/experiments.py due` lists checks that are due; `readout ` prints the read-only SQL command for `$ANALYST_DATABASE_URL`. + ## Operational notes - Manual upload review happens entirely inside the designated Telegram upload review chat. Keep communications and escalations there for traceability. - Weekly maintenance (Prefect flow health checks, data hygiene jobs, etc.) runs through Prefect deployment definitions. Use Prefect CLI to trigger flows during scheduled operations. diff --git a/SPEC.md b/SPEC.md index 7e0c42d8..854ea3df 100644 --- a/SPEC.md +++ b/SPEC.md @@ -56,7 +56,7 @@ Living entries (short list): | [specs/moderator-community-loop.md](specs/moderator-community-loop.md) | Source voting | | [docs/growth/virality-loop.md](docs/growth/virality-loop.md) | Growth thesis | -Finished experiment write-ups: [experiments/completed/](experiments/completed/). +Experiments (running and decided, one file each): [experiments/README.md](experiments/README.md). ## Invariants diff --git a/TODOS.md b/TODOS.md index 69f12a8f..27b6b120 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,7 +1,7 @@ # TODOS > Living backlog only. Completed items live in git history and dated experiment -> write-ups under `experiments/completed/`. +> write-ups in [`experiments/`](experiments/README.md). > Last cleaned: 2026-06 (Wave A hygiene). **Product compass:** [docs/growth/virality-loop.md](docs/growth/virality-loop.md) @@ -9,13 +9,13 @@ ## P1 — Growth & feed measurement ### ~~Virality score for ranking~~ — viral_shares_blender_v1 KILLED 2026-10-30 -**What:** Engine `viral_shares` + mature blend A/B (`viral_shares_blender_v1`). See `experiments/completed/2026-08-09-viral-shares-blender-v1.md`. +**What:** Engine `viral_shares` + mature blend A/B (`viral_shares_blender_v1`). See `experiments/2026-08-09-viral-shares-blender-v1.md`. **Closed:** clickers/1k 0.031 control vs 0.025 treatment. Enrollment stopped, all users on control. -**Rules + all open hypotheses:** [`docs/growth/HYPOTHESES.md`](docs/growth/HYPOTHESES.md). +**All experiments and learnings:** [`experiments/README.md`](experiments/README.md). ### Cold-start first-session quality floors — SHIPPING **What:** explore raw-LR ≥0.50 / n≥25; CS3 blend uses best_uploaded + text_light (no like_spread). -**Check:** H5 in `experiments/HYPOTHESES.md` — smoke 2026-08-12, primary 2026-08-16. +**Check:** `experiments/2026-08-09-cold-start-floors.md` (H5). ### Broadcast HQ meme pick — SHIPPING **What:** retention pushes pick affinity×LR meme (`broadcast_reengagement_hq`) vs queue fallback. diff --git a/docs/analyst/README.md b/docs/analyst/README.md index 0a6b3884..cf53ce55 100644 --- a/docs/analyst/README.md +++ b/docs/analyst/README.md @@ -17,11 +17,9 @@ These queries and reports support product-health monitoring and experiment analy - `docs/analyst/dwell-feed-vs-broadcast.sql` — `sec_to_react` percentiles: like vs skip × feed vs broadcast; dwell buckets; demote inventory risk. - `docs/analyst/source-affinity-demote-guardrails.sql` — post-deploy volume / LR checks for soft-demote (#340). - `docs/analyst/broadcast-reengagement.sql` — retention push HQ vs queue fallback vs feed. -- Living decision calendar (which experiments are live and which SQL each uses): [`docs/growth/HYPOTHESES.md`](../growth/HYPOTHESES.md). SQL for killed experiments is deleted; recover it from version control history if needed. +- Experiment ledger (open experiments with check dates, decided ones with result + learning): [`experiments/README.md`](../../experiments/README.md). `python scripts/experiments.py due` lists checks that are due; `readout ` prints the SQL command. SQL for killed experiments is deleted; recover it from version control history if needed. - `docs/analyst/crossposting.sql` — reusable read-only queries for Telegram channel views/forwards, 24h labels, and pre-posting share signals. - `specs/crossposting-share-optimization-2026-05-18.md` — current compact crossposting readout and next experiment gate. -- `experiments/active/` — Currently running experiments to monitor -- `experiments/completed/` — Historical experiments for context ## Database Schema Reference Full table definitions: `src/database.py` @@ -77,7 +75,7 @@ No hardcoded engine list or expected LRs here; they drift. Sources of truth: - **Prefect**: Pipeline health (parser → ETL → final_pipeline) ## North Star Metric -Defined once in [`SPEC.md`](../../SPEC.md) (session length + organic growth via shares). The experiment ship bar (funnel + k-factor, with session length and block rate as guardrails) is in [`docs/growth/HYPOTHESES.md`](../growth/HYPOTHESES.md). Do not restate either here. +Defined once in [`SPEC.md`](../../SPEC.md) (session length + organic growth via shares). The experiment ship bar (funnel + k-factor, with session length and block rate as guardrails) is in [`experiments/README.md`](../../experiments/README.md). Do not restate either here. Operational health signals: @@ -85,4 +83,6 @@ Operational health signals: **Chat Agent metrics**: agent_calls_24h, active_chats_24h, avg_response_ms, token_cost_usd, chat meme like rate. See "CHAT AGENT" section in metrics.sql. -See `experiments/README.md` for report format and `docs/analyst/metrics.sql` for the actual queries. +Healthy product snapshot (rough, not ship gates): new_memes_24h > 100, ok_pct ~90–96%, active reactors 24h not collapsing vs prior week, multi-engine mix still present. + +Queries: `docs/analyst/metrics.sql`. diff --git a/docs/analyst/broadcast-holdout-v1.sql b/docs/analyst/broadcast-holdout-v1.sql index fb0d2cf9..d2da9ef2 100644 --- a/docs/analyst/broadcast-holdout-v1.sql +++ b/docs/analyst/broadcast-holdout-v1.sql @@ -1,6 +1,6 @@ -- Readout: broadcast_cadence_holdout_v1_ (15m/24h/1w/2w/4w) and -- monthly_reminder_v1, treated vs holdout. Read-only, one SELECT. --- Experiment note: experiments/active/2026-10-30-broadcast-cadence-holdout.md +-- Experiment note: experiments/2026-10-30-broadcast-cadence-holdout.md -- -- Assignment = first time the user was selected by that step (intent-to-treat; -- holdout users were selected but not pushed). Steps are randomized diff --git a/docs/analyst/broadcast-reengagement.sql b/docs/analyst/broadcast-reengagement.sql index 17847923..d33c5b27 100644 --- a/docs/analyst/broadcast-reengagement.sql +++ b/docs/analyst/broadcast-reengagement.sql @@ -8,7 +8,7 @@ -- Compare to in-session feed engines (recommended_by NOT LIKE 'broadcast%'). -- Primary: reactivation (react within 1h), LR, sec_to_react p50. -- --- See experiments/HYPOTHESES.md H3 / H5. +-- See experiments/2026-08-09-broadcast-hq-pick.md (H3b) and experiments/README.md. \set ON_ERROR_STOP on diff --git a/docs/analyst/crossposting-content-tags-gates.py b/docs/analyst/crossposting-content-tags-gates.py index bd531184..cd31c65f 100644 --- a/docs/analyst/crossposting-content-tags-gates.py +++ b/docs/analyst/crossposting-content-tags-gates.py @@ -1,6 +1,6 @@ """Gates for the crosspost content-tags hypothesis (read-only). -Experiment: experiments/active/2026-10-31-crosspost-content-tags.md +Experiment: experiments/2026-10-31-crosspost-content-tags.md Research: docs/analyst/research/2026-10-30-forward-drivers/ Usage (analyst machine, needs pandas + statsmodels + scikit-learn): diff --git a/docs/analyst/crossposting-content-tags-shadow.sql b/docs/analyst/crossposting-content-tags-shadow.sql index 8e889e0c..7a45c001 100644 --- a/docs/analyst/crossposting-content-tags-shadow.sql +++ b/docs/analyst/crossposting-content-tags-shadow.sql @@ -1,4 +1,4 @@ --- Crosspost content-tags shadow readout (experiments/active/2026-10-31-crosspost-content-tags.md). +-- Crosspost content-tags shadow readout (experiments/2026-10-31-crosspost-content-tags.md). -- One SELECT, one row per channel. Production pick is unchanged; v5c is log-only. -- -- score_v5c = score_v5 * content_mult_v1, content_mult_v1 = 0.85 ^ (relatable + tweet_screenshot) diff --git a/docs/analyst/crossposting-src-fwd-shadow.sql b/docs/analyst/crossposting-src-fwd-shadow.sql index 2eb28bb1..07561182 100644 --- a/docs/analyst/crossposting-src-fwd-shadow.sql +++ b/docs/analyst/crossposting-src-fwd-shadow.sql @@ -1,5 +1,5 @@ -- Crosspost source-forward-rate shadow readout --- (experiments/active/2026-11-01-crosspost-src-forward-rate.md). +-- (experiments/2026-11-01-crosspost-src-forward-rate.md). -- One SELECT, one row per channel. Production pick is unchanged; v5f is log-only. -- -- score_v5f = score_v5 * (0.5 + src_fwd_pct), 1.0 multiplier when src_fwd_pct is NULL diff --git a/docs/analyst/crossposting-v5-shadow.sql b/docs/analyst/crossposting-v5-shadow.sql index adfd0818..752b8fca 100644 --- a/docs/analyst/crossposting-v5-shadow.sql +++ b/docs/analyst/crossposting-v5-shadow.sql @@ -1,4 +1,4 @@ --- Crosspost v5 shadow readout (experiments/active/2026-10-30-crosspost-v5-shadow.md). +-- Crosspost v5 shadow readout (experiments/2026-10-30-crosspost-v5-shadow.md). -- One SELECT, one row per channel. Production pick is unchanged; v5 is log-only. -- -- zf = ln(1+forwards) - channel x MSK-week mean over posted image memes >= 48h old diff --git a/docs/analyst/explore-budget-v1.sql b/docs/analyst/explore-budget-v1.sql index 876cf04f..eec3f234 100644 --- a/docs/analyst/explore-budget-v1.sql +++ b/docs/analyst/explore-budget-v1.sql @@ -1,5 +1,5 @@ -- explore_budget_v1 readout (one SELECT; read-only) --- Experiment doc: experiments/active/2026-10-30-explore-budget.md +-- Experiment doc: experiments/2026-10-30-explore-budget.md -- Rows: variant x segment (mod = moderator/admin, user = everyone else). -- Per-user means, feed sends only (recommended_by NOT LIKE 'broadcast%'), -- counted from each user's assigned_at. diff --git a/docs/analyst/onboarding-funnel.sql b/docs/analyst/onboarding-funnel.sql index 982ac7d7..bae43b42 100644 --- a/docs/analyst/onboarding-funnel.sql +++ b/docs/analyst/onboarding-funnel.sql @@ -1,7 +1,7 @@ -- Onboarding funnel + first-meme quality -- -- Companion: docs/analyst/readouts/2026-09-07-onboarding-and-dormant-blast.md --- Hypotheses: docs/growth/HYPOTHESES.md H9 / H10 / H11 +-- Experiments: experiments/2026-09-07-onboarding-first-tap.md (H9), -cold-start-fresh-viral.md (H10), -onboarding-skip-countdown.md (H11) -- -- Growth bar for any onboarding change: -- 1) less leaky: first-tap %, first-like %, reached-5 diff --git a/docs/analyst/research/2026-10-30-forward-drivers/README.md b/docs/analyst/research/2026-10-30-forward-drivers/README.md index 4c3c021a..91064ff6 100644 --- a/docs/analyst/research/2026-10-30-forward-drivers/README.md +++ b/docs/analyst/research/2026-10-30-forward-drivers/README.md @@ -7,7 +7,7 @@ Result: two traits predict **fewer** forwards: a relatable everyday joke and a t Threads screenshot. Each one costs about −0.1 zf. Political or news memes get **more** forwards (+0.19 zf), but only 57 posts have that label, so we log it and do not use it. -Experiment: [`experiments/active/2026-10-31-crosspost-content-tags.md`](../../../../experiments/active/2026-10-31-crosspost-content-tags.md). +Experiment: [`experiments/2026-10-31-crosspost-content-tags.md`](../../../../experiments/2026-10-31-crosspost-content-tags.md). ## Data and method diff --git a/docs/broadcasts.md b/docs/broadcasts.md index 5d3942e0..f7638886 100644 --- a/docs/broadcasts.md +++ b/docs/broadcasts.md @@ -51,7 +51,7 @@ Cadence steps require a real reaction near the step window (not just month 13:00 MSK) uses `send_meme_broadcast()` with `broadcast_id = monthly-reminder-YYYY-MM`, so a retry or manual re-run in the same month is deduped. Both carry holdouts logged in `experiment_assignment`; -see `experiments/active/2026-10-30-broadcast-cadence-holdout.md`. +see `experiments/2026-10-30-broadcast-cadence-holdout.md`. ### 3. Monitor diff --git a/docs/growth/HYPOTHESES.md b/docs/growth/HYPOTHESES.md index 30c3e158..8457ca83 100644 --- a/docs/growth/HYPOTHESES.md +++ b/docs/growth/HYPOTHESES.md @@ -1,686 +1,13 @@ -# Living hypotheses & check-in schedule +# Hypotheses (moved) -**Purpose:** When an agent (or human) resumes after days/weeks, this file answers: -what is live, what we expected, when to re-measure, and what “good / kill” means. +Every experiment (design, log, result, learning) now lives in one file under +[`experiments/`](../../experiments/). Start at +[`experiments/README.md`](../../experiments/README.md): open experiments with check +dates, then everything decided with its result and learning. -**Last updated:** 2026-10-30 (UTC). Experiment closure: H1/H7/H8/H10/channel_hits killed, orphan splits closed -**Prod DB clock at last interim:** `2026-10-30 ~16:35 UTC` +- Running and due checks: `python scripts/experiments.py due` +- Readout command for one experiment: `python scripts/experiments.py readout ` +- North star: [`SPEC.md`](../../SPEC.md). Growth thesis: [`virality-loop.md`](virality-loop.md). -This file is the **canonical registry**. `experiments/HYPOTHESES.md` is a pointer here. - -**Growth bar (all new Hs):** ship only if the funnel gets less leaky (first-tap / first-like / reached-5) **and/or** k-factor moves (unique non-self share clicks, new-user invites). Session length and block rate are guardrails. See [`docs/growth/virality-loop.md`](virality-loop.md). - -How to use on resume: - -1. Read this file top → bottom. -2. For each row with `Next check ≤ today`, run the linked SQL against - `ANALYST_DATABASE_URL` (read-only). -3. Write numbers into the linked readout path; update **Status** / **Decision**. -4. If status becomes `ship` or `kill`, open a code PR (planner weights / freeze - enrollment) and move the experiment note `active/` → `completed/`. - ---- - -## Decision log: 2026-10-30 - -Readout: [`docs/analyst/readouts/2026-10-30-experiment-closure.md`](../analyst/readouts/2026-10-30-experiment-closure.md). - -| Hypothesis | Decision | Key numbers | -|------------|----------|-------------| -| H1 `viral_shares_blender_v1` | **KILL**. Enrollment stopped in code, all users on control | clickers/1k C 0.031 vs T 0.025, mean user LR −2.2 pp, viral engine 0.76% of T sends (n 344/350) | -| H2 `source_affinity_demote_v1` | **KEEP at 100%** | no send-volume cliff, weekly LR 32.6–36.1% vs 30.7–33.5% pre | -| H3 broadcast label | **DONE** | broadcast p50 sec-to-react 3.8–5.5k s vs feed 5.7 s | -| H3b HQ broadcast pick | **KEEP** | 14d LR 57.2%, react<1h 36.5%, queue fallback 0.4% | -| H3c channel-viral pick | **UNCHANGED**. Confounded, so it waits for a user-randomized readout | pooled 1h −9.6 pp vs HQ, but within-user (67 users) +6.9 pp, LR −5.4 pp | -| H5 cold-start floors | **KEEP** (no rollback trigger) | first-tap 61.5 → 62.0%, n=100 post, underpowered | -| H6 crosspost v4 like-volume | **KEEP at 100%** | hit rate 32.1 → 41.0% (+8.9 pp, z≈2.4), f1k 26.2 → 28.9 | -| H7 taste-cohort shadow | **KILL as signal** (no canary, decision_log fields left for the crossposting PR) | top vs bottom n_taste f1k 28.52 vs 28.72 (n=287) | -| H8 shadow hybrid `v4_x_src_v1` | **KILL as signal** (no canary, fields left for the crossposting PR) | corr(shadow, f1k) −0.09, agreed vs disagreed f1k 29.01 vs 29.10 | -| H10 `cold_start_fresh_viral_v1` | **KILL**. Flag default off, so everyone gets control | first tap 60.0% C vs 48.5% T, blocked<1h 19 vs 27 (n 95/99) | -| H11 skip countdown | **KEEP (neutral)** | p50 to first meme 11.6 → 2.8 s, first-tap flat (H10 control 60.0% vs ~61.5% pre) | -| `channel_hits_v1` | **KILL** (inconclusive per protocol, window ended 2026-09-19) | retained invitees T 1 vs C 0, host activity −23% T vs −5% C (n 42/43) | -| `ocr_length_preference_v1` | **INVALID** (quality_failed) | late OCR 23% vs 5% limit | -| `first_meme_nudge` (orphan, since 2026-04-28) | **KILL**. Control (no nudge) for everyone, code removed | first tap 15.8% C vs 16.0% T, user LR 39.2 vs 36.1%, d1–7 return 5.0 vs 5.0% (n 1526/1456) | -| `meme_like_count` (orphan, since 2026-05-12) | **KILL**. Control (plain heart) for everyone, code removed | mean user LR 45.6 vs 46.2%, react rate 83.4 vs 83.3%, active days/14d 0.85 vs 0.86 (n 6306/6208) | -| `text_light_lr_smoothed` sends (~20/wk) | **Not a leftover.** It is the cold-start CS3 blend weight and CS fallback shipped in #271. The disabled overlay is a separate thing | 28d: 75 sends, all to users under 100 prior sends | - ---- - -## Check calendar (hard dates) - -| Date (UTC) | What to run | Hypotheses | -|------------|-------------|------------| -| **2026-09-08** | Dormant blast T+24h | **H3c** | -| **2026-09-14** | Onboarding funnel + fresh CS split | **H9**, **H10**, **H11** | -| **2026-08-12** | Smoke + mature v4 if n≥8 | H1, H2, H3b, H5, **H6** | -| **2026-08-16** | Primary feed/reco day-7 | H1, H2, H3b, H5 | -| **2026-08-17** | **Crosspost v4 mature keep/kill** | **H6** | -| **2026-08-23** | Feed exp finals if needed | H1 primarily | -| **2026-08-24** | Taste shadow readout | **H7** | -| Ad-hoc | Bot→channel ML re-run / walk-forward | **H8** | -| **2026-11-20** | `explore_budget_v1` keep/kill — [`experiments/active/2026-10-30-explore-budget.md`](../../experiments/active/2026-10-30-explore-budget.md) | explore budget | -| **2026-11-20** | Crosspost content tags Gate A/B (`crossposting-content-tags-gates.py`) + `crossposting-content-tags-shadow.sql`. Both pass → `score_v5c` live on RU — [`experiments/active/2026-10-31-crosspost-content-tags.md`](../../experiments/active/2026-10-31-crosspost-content-tags.md) | content tags (relatable / tweet −0.1 zf) | -| **2026-11-28** | Crosspost source forward rate shadow (`crossposting-src-fwd-shadow.sql`): `score_v5f = score_v5 * (0.5 + src_fwd_pct)`. RU Spearman gain over v5 with 95% CI excluding 0 → switch the RU pick to v5f as an alternate-day A/B — [`experiments/active/2026-11-01-crosspost-src-forward-rate.md`](../../experiments/active/2026-11-01-crosspost-src-forward-rate.md) | src forward rate (offline RU ρ +0.52) | -| **2026-11-14** | Signups with ≥1 meme sent / first reaction (H13 query below) | **H13** | -| **2026-11-21** | Within-source **reaction rate** beats views for cold-meme ranking (`raw_impr_rank` now = quartile of `reactions / views` within source × meme type; offline Spearman vs our like rate +0.22 vs +0.02 for views). Metric: like rate of memes with `raw_impr_rank = 0`, and of engines that use it (`candidates.py` rank-0 pool, crosspost impr penalty), 3 weeks before vs after deploy. Kill if rank-0 LR does not rise | reaction-rate rank | -| Ad-hoc | After any ranking/deploy incident | all active | - -Agent prompt on resume (copy-paste): - -```text -Read docs/growth/HYPOTHESES.md. For every hypothesis with Next check ≤ today, -run the readout SQL on ANALYST_DATABASE_URL, update the Status/Decision rows -and the linked readout markdown, then recommend ship/hold/kill with numbers. -``` - ---- - -## H1 — `viral_shares` in mature blend (A/B) - -| Field | Value | -|-------|--------| -| **ID** | `viral_shares_blender_v1` | -| **Status** | **KILLED 2026-10-30**: clickers/1k C 0.031 vs T 0.025. Enrollment stopped, all users on control | -| **Shipped** | 2026-08-09 ~14:20 UTC (first assignment) | -| **Code** | `candidates.viral_shares`, `blender_experiments.get_viral_shares_blender_v1_weights` | -| **Note** | `experiments/completed/2026-08-09-viral-shares-blender-v1.md` | -| **SQL** | removed 2026-10-31 (killed; in git history) | -| **Readout log** | removed 2026-10-31 (killed; in git history); final numbers in the 2026-10-30 closure readout | - -### Hypothesis (precise) - -Giving mature users (`nmemes_sent ≥ 100`) weight **0.2** on engine -`viral_shares` (memes ranked by `invited_count / ln(nmemes_sent+e)`), taken from -`lr_smoothed`, increases **unique non-self share clickers per 1k sends** and -**new-user invites per 1k sends** vs control, without harming session depth. - -### Interim baseline (2026-08-09 15:44 UTC, ~1.4h) - -| Variant | n_users | memes_sent | LR % | viral_sends | clickers/1k | invites/1k | p50 session | -|---------|---------|------------|------|-------------|-------------|------------|-------------| -| control | 10 | 297 | 39.6 | 0 | 0 | 0 | 16.5 | -| treatment | 12 | 367 | 54.2 | 13 (3.5%) | 0 | 0 | 19.0 | - -**Do not decide from this.** n≪gate; 0 growth events expected in first hours. -Engine is live (13 sends, high LR noise). - -### Decision rules (day-7 / day-14) - -**Sample (realistic for WAU~380):** -Do **not** wait for 1000/arm (unrealistic). Prefer: - -- day-7 if **≥80 users/variant** and **≥2k sends/variant**, else hold to day-14; -- day-14 decide even if underpowered (ship only if direction clear + guardrails OK). - -**Ship treatment → mature default** if all: - -1. `unique_clickers_per_1k_sends` treatment ≥ control **and** absolute lift makes - sense at our volume (or invites/1k clearly higher); -2. Guardrail: p50 session length treatment ≥ control − **10%** relative; -3. Guardrail: like rate treatment ≥ control − **3 pp**; -4. Engine slice: `viral_shares` continuation@30m not < lr_smoothed − **5 pp** - (on ≥100 viral sends). - -**Kill (remove weight, freeze enrollment to control)** if: - -- session p50 drops >10% relative, or -- LR drops >5 pp with ≥2k sends/arm, or -- day-14 still ~0 viral_engine exposure on treatment (wiring bug). - -**Hold** if underpowered or metrics mixed. - -### Next check - -- **2026-08-12** smoke: cohort sizes, viral % of treatment sends, LR -- **2026-08-16** primary -- **2026-08-23** final if needed - ---- - -## H2 — Soft demote majority-dislike sources - -| Field | Value | -|-------|--------| -| **ID** | `source_affinity_demote_v1` (config flags, not A/B) | -| **Status** | **shipped default ON** (PR #340). **KEEP 2026-10-30**: no volume cliff, LR +1–3 pp | -| **Shipped** | 2026-08-09 deploy `687fd46a` | -| **Code** | `RECOMMENDATION_DEMOTE_*`, `disliked_source_demote_sql` | -| **SQL** | `docs/analyst/source-affinity-demote-guardrails.sql` + inventory section of `dwell-feed-vs-broadcast.sql` | -| **Baseline** | `docs/analyst/readouts/2026-08-09-dwell-and-source-demote-baseline.md` | - -### Hypothesis - -Deprioritizing sources with `ndislikes > nlikes` (n≥5) by score ×0.15 increases -feed quality (higher session continuation / slightly higher LR on demotable -traffic) **without** increasing empty-queue symptoms, unlike hard-block (~57% -inventory cut offline). - -### Decision rules (day-7) - -**Keep ON** if: - -1. Sends/day and active users not cliffing vs week before deploy; -2. Global like rate not down >3 pp vs pre-deploy 7d window; -3. No spike in empty-queue alerts (Sentry/logs: `has empty meme queue`). - -**Tune** multiplier (0.15 → 0.3) if LR up but users complain of “same sources”. -**Disable** (`RECOMMENDATION_DEMOTE_DISLIKED_SOURCES=false`) if empty-queue or -volume collapse. - -### Next check - -- **2026-08-12** smoke volume/LR -- **2026-08-16** full guardrails vs baseline - ---- - -## H3 — Broadcast delivery label for dwell analytics - -| Field | Value | -|-------|--------| -| **ID** | `broadcast_reengagement_label` | -| **Status** | **DONE 2026-10-30**: labels live, broadcast p50 sec-to-react 3.8–5.5k s vs feed 5.7 s | -| **Shipped** | 2026-08-09 with #340 | -| **Code** | `flows/broadcasts/meme.py` → `recommended_by=broadcast_*` | -| **SQL** | `docs/analyst/broadcast-reengagement.sql`, dwell SQL | - -### Hypothesis - -Labeling retention pushes separately from feed engines lets us measure true -in-session `sec_to_react` and optimize broadcast timing for **fast** reaction. - -### Success criteria - -1. ✅ Labeled rows exist (`broadcast_reengagement` / `_hq`) -2. Broadcast p50 `sec_to_react` **higher** than feed (delayed opens expected) -3. Share of broadcast reactions with `sec_to_react > 1h` documented - -### Next check - -- **2026-08-12 / 16** — full feed vs broadcast dwell table - ---- - -## H3b — High-quality meme pick for retention broadcasts - -| Field | Value | -|-------|--------| -| **ID** | `broadcast_reengagement_hq_pick` | -| **Status** | **KEEP 2026-10-30**: 14d LR 57.2%, react<1h 36.5%, queue fallback 0.4% | -| **Code** | `src/recommendations/broadcast_pick.py`, kill switch `BROADCAST_HIGH_QUALITY_PICK_ENABLED` | -| **Labels** | `broadcast_reengagement_hq` (SQL pick) vs `broadcast_reengagement` (queue fallback) | -| **SQL** | `docs/analyst/broadcast-reengagement.sql` | - -### Hypothesis - -Choosing the reengagement meme by **user×source affinity × raw like rate** -(avoiding majority-dislike sources) beats blind feed-queue pop on: - -1. **react_within_1h %** (primary — “came back because of this push”) -2. like rate among reacted -3. p50 `sec_to_react` among in-window reactions (secondary) - -### Pre-HQ baseline (2026-08-09, ~39 `broadcast_reengagement` sends) - -- like rate ~54% (small n) -- only ~4 reacts under 2 minutes — delayed open is the norm - -### Decision rules (day-7 from HQ deploy) - -**Keep HQ ON** if: - -- `broadcast_reengagement_hq` react_within_1h ≥ queue-fallback **or** ≥ pre-HQ baseline + directionally better LR; -- no send failure spike (empty HQ → fallback should keep volume). - -**Disable** (`BROADCAST_HIGH_QUALITY_PICK_ENABLED=false`) if HQ pool empty for -most users (fallback rate >80%) or LR collapses >5 pp vs fallback. - -### Next check - -- **2026-08-12** smoke: any `_hq` rows? fallback share? -- **2026-08-16** primary HQ vs fallback vs feed - ---- - -## H3c — Channel-viral memes for retention broadcasts - -| Field | Value | -|-------|--------| -| **ID** | `broadcast_channel_viral_pick` | -| **Status** | **disabled 2026-10-30** (default `false`; code path kept). Frozen rule trips (1h −9.6 pp vs HQ); arms reach different audiences; within users viral +6.9 pp at 1h but −5.4 pp LR, so HQ is the default (PR #366) | -| **Code** | `src/recommendations/broadcast_pick.py`, kill switch `BROADCAST_CHANNEL_VIRAL_PICK_ENABLED` | -| **Labels** | `broadcast_channel_viral` vs `broadcast_reengagement_hq` vs queue | -| **SQL** | `docs/analyst/broadcast-reengagement.sql` | - -### Hypothesis - -A meme that the channel audience already forwarded is a better single push than -bot-internal HQ (affinity × like rate), because forwards are a community check -that the joke is worth sending onward. - -### Rules - -- Last 7 days of `tgchannelru` / `tgchannelen` posts, at least 24h old, ranked - by latest snapshot forwards. Unseen, matching language, image, `published`. -- Skip known channel members (they already got the channel push). -- Skip `channel_hits_v1` assignees until exposure ends (feed test stays clean). -- Empty pool → existing HQ → queue. Never skip a user because viral is empty. - -**Keep ON** if `broadcast_channel_viral` react_within_1h ≥ HQ (same 14d window) -without a like-rate drop >3 pp on reacted rows. - -**Disable** (`BROADCAST_CHANNEL_VIRAL_PICK_ENABLED=false`) if viral share of -broadcasts is <10% (always empty) or 1h reactivation is worse than HQ by >5 pp -with ≥200 viral sends. - -### Decision 2026-10-30: disable, HQ is default - -Sep 10 → Oct 28 stratified readout: viral like rate 5–20 pp below HQ in every -stratum, react-within-1h/24h equal or worse, session length and 7d activity -equal (within-user too). Disable rule met → `BROADCAST_CHANNEL_VIRAL_PICK_ENABLED` -defaults to `false`. Follow-up: `experiments/active/2026-10-30-broadcast-cadence-holdout.md`. - -### One-shot idle blast (2026-09-07) - -`scripts/broadcast_dormant.py` (removed 2026-10-30; in git history) id `dormant-channel-viral-2026-09-07`. -**Let it finish.** Next extra wave: 7–28d + holdout, not 90d+ never-reacted. -Readout: `docs/analyst/readouts/2026-09-07-onboarding-and-dormant-blast.md`. - -### Cadence doctrine (founder 2026-09-07) - -Keep **15m / 24h / 48h / 1w / 2w / 4w** while the user is still fresh. -After that, **one monthly** reminder for people who already reacted. -Lever inside the cadence is meme quality (H3c), not fewer early nudges. - -2026-10-30: cadence targeting now requires a real reaction near the step -window; per-step no-push holdouts (`broadcast_cadence_holdout_v1_*`) and the -monthly reminder (`monthly_reminder_v1`, flag-gated) are live — see -`experiments/active/2026-10-30-broadcast-cadence-holdout.md`. - ---- - -## H4 — Skip ≠ hate (measurement contract, not a ship) - -| Field | Value | -|-------|--------| -| **ID** | `skip_is_next_not_ban` | -| **Status** | **accepted product doctrine** | -| **Evidence** | skip p50 4.7s vs like 7.1s; instant skip 19% vs like 9% (7d feed) | -| **SQL** | `docs/analyst/dwell-feed-vs-broadcast.sql` | - -### Hypothesis (ongoing) - -Dislike button is primarily “next”; ranking must prefer **likes + dwell**, not -raw skip counts as bans. Future work: down-weight `sec_to_react < 2s` skips and -ignore `>1h` reactions in affinity. - -### Next check - -- **2026-08-16** re-run dwell percentiles; compare to baseline readout -- No ship decision until a dedicated dwell-weight PR with offline counterfactual - ---- - -## H5 — Cold-start first-session quality floors - -| Field | Value | -|-------|--------| -| **ID** | `cold_start_first_impression_v2` | -| **Status** | **KEEP 2026-10-30**: no rollback trigger. First-tap 61.5 → 62.0% (n=100 post), so the lift is unproven | -| **Code** | `cold_start_explore` raw-LR floor; CS3 blend → `best_uploaded` + text-light | -| **SQL** | engine slice in `docs/analyst/metrics.sql` / cold_start quality scripts | - -### Hypothesis - -**A1.** Raising `cold_start_explore` floors (raw like rate ≥0.50, ≥25 reactions, -order by `lr_smoothed`) lifts first-10 LR and reached-10 vs pre-change -(guarded explore was ~**18%** LR on 7d — too weak for first impression). - -**A2.** CS3 (memes 16–29) replacing `like_spread` with `best_uploaded_memes` -improves LR/continuation in that band without starving personalization -(`cold_start_adapt` stays fixed_pos 0). - -### Decision rules (day-7) - -**Keep** if first-10 LR for `cold_start_explore*` ≥ pre-change + directionally -higher reached-5/10; no empty-queue spike for new users. - -**Rollback explore floors** if explore pool empty → fallback share spikes and -session depth drops. - -**Rollback CS3** if `best_uploaded` share causes lower continuation vs prior -week CS3 band. - -### Next check - -- **2026-08-12** smoke: explore volume + LR -- **2026-08-16** first-10 / CS3 band metrics - ---- - -## H6 — Crosspost RU: meme-level bot volume (not LR) × source prior - -| Field | Value | -|-------|--------| -| **ID** | `crosspost_meme_level_v1` / **score_version=4** | -| **Status** | **KEEP at 100% 2026-10-30**: hit rate 32.1 → 41.0% (+8.9 pp, z≈2.4), f1k 26.2 → 28.9 | -| **Kill switch** | `CROSSPOST_RU_MEME_LIKE_VOLUME_ENABLED=false` → v2 | -| **Readout offline** | `docs/analyst/readouts/2026-08-09-crosspost-meme-level-offline.md` | -| **Readout online** | `docs/analyst/crossposting-v4-like-volume.sql` | -| **n offline** | 611 RU image posts / 120d | - -### Hypothesis - -Among channel candidates, **like volume** (`ln(nlikes+1)` via `meme_stats`) improves -24h channel forwards beyond source prior. Bot **like rate** does not. - -### Offline result (2026-08-09) - -- pre_lr vs channel: ~0 (reject LR feature) -- pre_likes residual Spearman ~**0.16–0.20** -- top-20% **src × log1p(pre_likes)** lift **1.14×** forwards (time-split test too) -- HIT: f1k ≥ **~31** OR forwards ≥ **12** - -### Production formula (score_version=4) - -```text -v2_score * LN(COALESCE(nlikes,0) + 1) -``` - -Logged in decision candidates: `like_volume_factor`, `like_volume_enabled`. - -### Online success (7–14d after deploy) - -Compare v4 mature posts vs last 30d v2 baseline: - -- hit_rate_pct not down; preferably **+3pp** -- avg forwards_24h ≥ v2 baseline -- avg views_24h not < v2 − 15% -- Kill if hit_rate collapses or empty slots spike - -### Online early (2026-08-10 ~17 UTC) - -- 4 v4 posts live; **0 mature** yet — WATCH -- age-matched ~6h: v4 f1k **30.0** vs v2 **32.6** (n=4, not RED) -- Frequency: **keep 5/day** (do not post more/less until mature readout) - -### Next check - -- **2026-08-12**: mature n≥8 if possible -- **2026-08-17** (~7d): `crossposting-v4-like-volume.sql` keep/kill -- Full plan: `experiments/completed/2026-08-10-crosspost-v4-and-taste-cohort.md` - ---- - -## H7 — Taste cohort soft signal (SHADOW) - -| Field | Value | -|-------|--------| -| **ID** | `crosspost_taste_cohort_v1` | -| **Status** | **KILLED as signal 2026-10-30**: top vs bottom n_taste f1k 28.52 vs 28.72 (n=287). No canary | -| **Cohort** | `src/crossposting/data/ru_taste_cohort_v1.json` (top 50) | -| **Code** | `taste_cohort.py`, `_enrich_candidates_with_taste_shadow` | -| **SQL** | removed 2026-10-31 (killed; in git history) | -| **Experiment note** | `experiments/completed/2026-08-10-crosspost-v4-and-taste-cohort.md` | - -### Hypothesis - -A fixed set of ~50 users whose historical likes co-occur with high channel -fwd/1k provide a **weak but real** additional signal (soft boost), not a -replacement for like volume. - -### Offline (2026-08-10) - -- top50 likes top-20% lift **1.19** vs all-likes **1.10**; beats random-50 p95 **1.14** -- coverage ≥1 taste like ~**18%** of posts → never sole filter - -### Shadow fields on decision candidates - -`n_taste_likes`, `taste_boost_shadow` (= `1+0.15*min(n,5)`), `taste_cohort_version` - -### Predictions - -| Date | Pass if | -|------|---------| -| **2026-08-24** | Among mature picks with n_taste≥1, higher n_taste half has higher mean f1k (n≥20) | -| Canary | Only if H6 not RED and shadow pass → optional soft boost flag | - -### Not doing - -- Hardcode 50 users as only ranker -- Post less often until H6 mature fails - -### Refresh - -`python scripts/crosspost_taste_cohort.py` - ---- - - ---- - -## H8 — Bot→channel ML lab (OFFLINE) - -| Field | Value | -|-------|--------| -| **ID** | `bot2channel_ml_lab_v1` | -| **Status** | **KILLED as signal 2026-10-30**: offline HOLD vs v4, and the live shadow corr is −0.09 (see below) | -| **Code** | `experiments/ml-crosspost-bot2channel/` — removed 2026-10-30; recover from commit `177cad0e` | -| **Note** | `experiments/completed/2026-08-11-bot2channel-ml-lab.md` | -| **Report** | `experiments/ml-crosspost-bot2channel/reports/2026-08-11-models.md` | - -### Hypothesis - -Simple linear/logistic models on pre-post bot features + source prior beat -`ln(pre_likes+1)` on time-holdout top-20% f1k lift without tree overfit. - -### First result (2026-08-11) - -- n=624 labeled; single 70/30: logreg lift 1.187 vs v4 1.114 (looked good once) -- HGB depth3 **overfits** — discard for ship - -### Walk-forward (authoritative, 2026-08-11) - -- 3 expanding folds (~93 test each); bar: lift ≥ v4+0.05 on ≥2/3 + spearman gap <0.25 -- **logreg PASS** (2/3, mean lift 1.190 vs v4 1.080) -- **ridge PASS** (2/3, mean lift 1.171) -- HGB FAIL (overfit gap ~0.5) -- Report: `experiments/ml-crosspost-bot2channel/reports/2026-08-11-walkforward.md` -- RU summary: `.../reports/2026-08-11-otchet-ru.md` -- **No production pick change** — next is shadow score only - - -### Deep lifetime re-run (2026-08-11 evening) - -- Telethon full crawl: 11928 msgs, 4897 sc_ joinable → dataset **n=3889** (2023-11..2026-08) -- Label = lifetime f1k (TG now); HIT = f1k≥train p75 -- Walk-forward: **HOLD** — logreg/ridge/HGB do **not** beat v4 on ≥2/3 folds -- Source prior often ≥ volume on long history; bot volume ~flat on lifetime -- HTML: `experiments/ml-crosspost-bot2channel/reports/2026-08-11-results.html` -- Hypothesis battery: hybrids volume×source lead mean lift; **no bar PASS** - (`reports/2026-08-11-hypothesis-battery.html`) - -### Shadow hybrid in decision_log (shipping) - -| Field | Value | -|-------|--------| -| **Status** | **KILLED 2026-10-30**: corr(shadow, f1k) −0.092 on 363 posted picks, agreed vs disagreed f1k 29.01 vs 29.10. Fields stay until the crossposting PR replaces shadow scoring | -| **Formula** | `shadow_score = ln(nlikes+1) * src_quality_mult` | -| **Version** | `v4_x_src_v1` (+ `shadow_score_maturity` band) | -| **Code** | `src/crossposting/shadow_score.py`, `_build_decision_log` | -| **SQL** | removed 2026-10-31 (killed; in git history) | - -Also logs `shadow_rank`, `shadow_pick_meme_id`, `shadow_vs_prod_disagree`. - -### Next check - -- After deploy: decision_log has `shadow_score` on candidates -- **2026-08-25**: shadow-hybrid SQL (now removed) — disagree rate + corr(shadow, f1k_24h) -- Canary soft boost only if live corr/lift clearly > v4 alone -- After H7 merge: re-export + optional taste feature -- Do not schedule as feed experiment - ---- - -## H9 — First-tap is the onboarding leak - -| Field | Value | -|-------|--------| -| **ID** | `onboarding_first_tap_v1` | -| **Status** | **scoreboard** for H10/H11. **2026-10-30 flag:** users who never got a meme rose 3.1% → 12.6% (25/26 have no language row). Investigate the language picker | -| **Readout** | `docs/analyst/readouts/2026-09-07-onboarding-and-dormant-blast.md` | -| **SQL** | `docs/analyst/onboarding-funnel.sql` | - -### Hypothesis - -New-user growth is gated by **first meme → first tap**, not language picker -and not share skipping welcome. 90d (2026-09-07): 402 got a meme, 252 reacted, -**150** silent. Empty start 61% vs share 71%. - -### Growth bar - -First-tap % of got-meme; first-like %; reached-5. Secondary: non-self share -clicks and invites from the same cohort. Guardrail: block-in-1h. - -### Next check - -**2026-09-14** re-run `onboarding-funnel.sql`. - ---- - -## H10 — Fresh channel-quality memes as CS1 (A/B) - -| Field | Value | -|-------|--------| -| **ID** | `cold_start_fresh_viral_v1` | -| **Status** | **KILLED 2026-10-30**: first tap 60.0% C vs 48.5% T, blocked<1h 19 vs 27 (n 95/99). Flag default off | -| **Kill switch** | `COLD_START_FRESH_VIRAL_EXPERIMENT_ENABLED` | -| **Code** | `src/recommendations/cold_start_experiments.py`, `cold_start_explore(max_age_days=)` | -| **Labels** | `cold_start_explore_fresh` / `_guarded` vs classic `cold_start_explore` | - -### Hypothesis - -Same CS floors, but only memes **created in the last 30 days**, newest first, -lifts first-tap / first-like vs all-time `lr_smoothed` hits. Empty fresh pool -fills from classic CS so we never starve the first session. - -50/50 `sha256(experiment_id:user_id)%2` on true-new CS1 (`nmemes_sent < 6`, -`nsessions <= 1`). Observational 90d did **not** show an age→like gap (~22% -both); this A/B is the actual test. - -### Decision rules (day-7, ≥80 new users/arm if we get them) - -**Keep treatment / ship default** if first-tap ≥ control +5 pp (or first-like -+3 pp), reached-5 not down, invites/1k not down. -**Disable flag** if first-tap down, block-in-1h up, or empty-queue for new users. - -### Next check - -**2026-09-14** — assignment counts + first-tap by variant. - ---- - -## H11 — Drop empty-start 3-2-1 countdown - -| Field | Value | -|-------|--------| -| **ID** | `onboarding_skip_countdown_v1` | -| **Status** | **KEEP (neutral) 2026-10-30**: p50 to first meme 11.6 → 2.8 s, first-tap flat | -| **Code** | `src/tgbot/handlers/onboarding.py`, share tap in `reaction.py` | - -### Hypothesis - -Removing the ~9s 3-2-1 (p50 11.6s to first meme) and sending the first meme -right after welcome raises first-tap. Share path already did this at 0.2s and -converted better; after a share tap we now continue the feed (no second welcome). - -Not an A/B — UX debt. Score with H9. Rollback = restore the countdown block. - -### Next check - -**2026-09-14** empty-start p50 seconds to first meme should drop toward ~1s; -first-tap of empty start vs 61% baseline. - ---- - -## H13 — Default language instead of blocking picker - -| Field | Value | -|-------|--------| -| **ID** | `onboarding_default_language_v1` | -| **Status** | **live** (not an A/B; bug fix) | -| **Code** | `src/tgbot/handlers/start.py` (language seeded above the deep-link ladder), `language.py` picker callback | - -### Hypothesis - -Bare-`en` / unknown Telegram users got only the language picker on /start, no -`user_language` row, and so no meme (feed joins on `user_language`). 6 weeks to -2026-10-31: 21/150 signups (14%) had no language row, 0 of them got a meme, 0 -clicked the picker. Now they default to `en`, get welcome + first meme -immediately, and the picker follows the meme (`/lang` still works). - -### Growth bar - -Share of signups with ≥1 meme sent → ~100% (minus blocked); share with a first -reaction should rise. Baseline 6w: got meme 128/150 (85%), reacted 79/150 -(53%); last 2 weeks 42/56 (75%) and 27/56 (48%). Guardrail: no-language -signups = 0. - -```sql -SELECT date_trunc('week', u.created_at)::date wk, count(*) signups, - count(*) FILTER (WHERE NOT EXISTS (SELECT 1 FROM user_language l WHERE l.user_id=u.id)) no_lang, - count(*) FILTER (WHERE EXISTS (SELECT 1 FROM user_meme_reaction r WHERE r.user_id=u.id)) got_meme, - count(*) FILTER (WHERE EXISTS (SELECT 1 FROM user_meme_reaction r - WHERE r.user_id=u.id AND r.reacted_at IS NOT NULL)) reacted -FROM "user" u WHERE u.created_at >= now() - interval '42 days' GROUP BY 1 ORDER BY 1; -``` - -### Next check - -**2026-11-14**. - ---- - -## Explicitly not open experiments - -| Item | State | -|------|--------| -| `recently_liked_blender_v2` | **Shipped as mature default** (completed) | -| Hard-block majority-dislike | **Rejected** as default; opt-in flag only | -| Full Feed Turn rewrite | Not doing | -| New blender A/B | Do not start until H1 day-7 done | -| Crosspost rank by bot LR only | **Rejected** | -| Crosspost taste-only ranker | **Rejected** (coverage); soft boost only after H7 | -| Crosspost rank by bot LR | **Rejected** offline (H6) | -| Kill 15m–4w reengagement cadence | **Rejected** 2026-09-07 (habit while fresh; monthly after) | -| Rewrite new-user language as multi-select | **Already one-tap**; `/lang` is settings | -| “Fix” share skipping welcome/picker | **Rejected** — 90d share converts better than empty start | -| `channel_hits_v1` | **Killed** 2026-10-30. Inconclusive per protocol (retained invitees T 1 vs C 0) | -| `ocr_length_preference_v1` | **Invalid** 2026-10-05. Late OCR 23% vs the 5% limit | -| `first_meme_nudge` | **Killed** 2026-10-30. Orphan split, flat first tap (15.8 vs 16.0%). Code removed | -| `meme_like_count` | **Killed** 2026-10-30. Orphan split, flat LR (45.6 vs 46.2%). Code removed | -| H12 `friend_challenge_h12` | **KILLED** 2026-10-31. 106 assignments, 0 participations ever. Code removed, tables kept | - ---- - -## Agent checklist (weekly resume) - -```bash -# 1) Hypotheses file -# 2) Run: -psql "$ANALYST_DATABASE_URL" -f docs/analyst/source-affinity-demote-guardrails.sql -psql "$ANALYST_DATABASE_URL" -f docs/analyst/dwell-feed-vs-broadcast.sql -psql "$ANALYST_DATABASE_URL" -f docs/analyst/broadcast-reengagement.sql -psql "$ANALYST_DATABASE_URL" -f docs/analyst/onboarding-funnel.sql # H9–H11 -psql "$ANALYST_DATABASE_URL" -f docs/analyst/crossposting-v4-like-volume.sql # H6 -# 3) Update this file Status/Decision + write -# docs/analyst/readouts/YYYY-MM-DD-weekly-hypotheses.md -# 4) Crosspost plan tables: -# experiments/completed/2026-08-10-crosspost-v4-and-taste-cohort.md -``` - -Healthy product snapshot (rough, not ship gates): - -- new_memes_24h > 100, ok_pct ~90–96% -- active reactors 24h not collapsing vs prior week -- multi-engine mix still present (not 100% one engine) +The old H1–H13 sections and the decision calendar were moved into the experiment +files on 2026-10-31 (H-numbers are kept in each file's body). Old text is in git history. diff --git a/docs/growth/reco-experiment-platform.md b/docs/growth/reco-experiment-platform.md index 72c0ba11..c3905ca5 100644 --- a/docs/growth/reco-experiment-platform.md +++ b/docs/growth/reco-experiment-platform.md @@ -146,7 +146,7 @@ Rules: Before merge, required files: -1. `experiments/active/YYYY-MM-DD-name.md` — hypothesis, gates, kill rules +1. `experiments/YYYY-MM-DD-name.md` (front matter, see `experiments/README.md`) — hypothesis, gates, kill rules 2. `docs/analyst/.sql` — primary + guardrails 3. `recommended_by` or assignment variant for slice diff --git a/docs/growth/virality-loop.md b/docs/growth/virality-loop.md index d3e04ad9..79060197 100644 --- a/docs/growth/virality-loop.md +++ b/docs/growth/virality-loop.md @@ -52,7 +52,7 @@ A shippable ranking/growth experiment should have: 3. **Primary metric** + **guardrails**: - Primary growth: unique share clickers, new-user invites, invites per 1k memes sent - Guardrails: session continuation, LR, block rate -4. **Readout SQL** committed under `docs/analyst/` or `experiments/active/…` +4. **Readout SQL** committed under `docs/analyst/` (linked from the `experiments/` file) 5. **Control weights** imported from `src/feed_turn/planner.py` (`MATURE_BLEND_WEIGHTS` / `GROWING_BLEND_WEIGHTS`), never copy-pasted. ## Source affinity policy (v2, 2026-08-09) @@ -90,7 +90,7 @@ Full readout: [docs/analyst/recently-liked-blender-v2-closeout.md](../analyst/re | Mature wiring | `get_mature_blend_weights_with_experiments` | | Delivery prep | `src/tgbot/senders/delivery.py` | | Readout | killed 2026-10-30; SQL removed (in git history) | -| Experiment note | `experiments/completed/2026-08-09-viral-shares-blender-v1.md` | +| Experiment note | `experiments/2026-08-09-viral-shares-blender-v1.md` | ## Architecture still wanted @@ -106,7 +106,7 @@ Full readout: [docs/analyst/recently-liked-blender-v2-closeout.md](../analyst/re - [`specs/channel-growth-optimization.md`](../../specs/channel-growth-optimization.md) — channel side - [`specs/crossposting-share-optimization-2026-05-18.md`](../../specs/crossposting-share-optimization-2026-05-18.md) — channel share experiments - [`specs/data-hypotheses.md`](../../specs/data-hypotheses.md) — data findings -- [`experiments/README.md`](../../experiments/README.md) — how to run A/B write-ups +- [`experiments/README.md`](../../experiments/README.md) — experiment ledger: how to add/close one, everything tried and learned ## Anti-patterns diff --git a/experiments/completed/2026-03-20-cold-start-3phase.md b/experiments/2026-03-20-cold-start-3phase.md similarity index 85% rename from experiments/completed/2026-03-20-cold-start-3phase.md rename to experiments/2026-03-20-cold-start-3phase.md index ba6e0c76..dd155ff1 100644 --- a/experiments/completed/2026-03-20-cold-start-3phase.md +++ b/experiments/2026-03-20-cold-start-3phase.md @@ -1,3 +1,20 @@ +--- +id: cold_start_3phase +status: killed +hypothesis: Diversity-first cold start (best meme per top source, then adapt to reactions) beats single-engine lr_smoothed for new users +primary_metric: new-user D1 retention, 10-meme reach, first-meme LR +guardrails: median session >=17, WAU >=500, no new Sentry errors +unit: user +variants: none (pre/post) +started: 2026-03-20 +check_on: 2026-04-02 +decided: 2026-03-26 +result: first-meme LR 0% (28 new users), 10-meme reach 24.2%, cold_start_explore LR 18.8%; session 19 and WAU flat +learning: New users with no taste signal need the globally best memes first, not the most varied ones +code: src/recommendations/candidates.py cold_start_explore/cold_start_adapt, meme_queue.py +readout: +--- + # Experiment: Cold Start 3-Phase Adaptive System **Status:** active — EXTENDED diff --git a/experiments/completed/2026-03-20-goat-engine-investigation.md b/experiments/2026-03-20-goat-engine-investigation.md similarity index 78% rename from experiments/completed/2026-03-20-goat-engine-investigation.md rename to experiments/2026-03-20-goat-engine-investigation.md index bb6f63a1..46e81f96 100644 --- a/experiments/completed/2026-03-20-goat-engine-investigation.md +++ b/experiments/2026-03-20-goat-engine-investigation.md @@ -1,3 +1,20 @@ +--- +id: goat_engine_investigation +status: shipped +hypothesis: goat LR decline (44% to 16% in 6 days) is per-user pool exhaustion, not content quality +primary_metric: goat engine LR and session continuation +guardrails: +unit: user +variants: none (observational) +started: 2026-03-20 +check_on: +decided: 2026-03-20 +result: goat continuation 98.0% (best engine) while LR fell 44.6% to 16%, halving about every 3 days +learning: A fixed top-N engine exhausts its pool per user; continuation can stay high while LR decays, so add a per-user recency filter +code: src/recommendations/candidates.py goat +readout: +--- + # Experiment: Goat Engine Investigation **Status:** completed diff --git a/experiments/completed/2026-03-20-unstarve-like-spread.md b/experiments/2026-03-20-unstarve-like-spread.md similarity index 86% rename from experiments/completed/2026-03-20-unstarve-like-spread.md rename to experiments/2026-03-20-unstarve-like-spread.md index 350e5588..bf7ae0cc 100644 --- a/experiments/completed/2026-03-20-unstarve-like-spread.md +++ b/experiments/2026-03-20-unstarve-like-spread.md @@ -1,3 +1,20 @@ +--- +id: unstarve_like_spread +status: shipped +hypothesis: Dropping age_days<30 from like_spread_and_recent_memes multiplies its volume without hurting quality +primary_metric: engine sends/7d, median session length +guardrails: engine LR >=35% (revised from 40%), median session >=16 +unit: user +variants: none (pre/post) +started: 2026-03-20 +check_on: 2026-03-27 +decided: 2026-03-26 +result: engine sends 3,112 to 18,233 per 7d (5.9x), engine LR 51.1% to 42.5%, median session 18 to 19, continuation 97.7% +learning: Meme age is not a quality signal; widening a high-LR engine trades some LR for volume and longer sessions +code: src/recommendations/candidates.py like_spread_and_recent_memes +readout: +--- + # Experiment: Unstarve like_spread_and_recent engine **Status:** completed — SUCCESS diff --git a/experiments/completed/2026-03-26-cold-start-v2.md b/experiments/2026-03-26-cold-start-v2.md similarity index 89% rename from experiments/completed/2026-03-26-cold-start-v2.md rename to experiments/2026-03-26-cold-start-v2.md index c0111e37..c6b9227b 100644 --- a/experiments/completed/2026-03-26-cold-start-v2.md +++ b/experiments/2026-03-26-cold-start-v2.md @@ -1,3 +1,20 @@ +--- +id: cold_start_v2 +status: shipped +hypothesis: Quality-first cold start phase 1 (>=20 reactions, raw LR >=0.40) beats diversity-first for first impressions +primary_metric: first-meme LR, 10-meme reach (new users) +guardrails: median session >=18, WAU >=500 +unit: user +variants: none (pre/post) +started: 2026-03-26 +check_on: 2026-04-09 +decided: 2026-04-09 +result: first-meme LR 0% to 33.3%, organic 10-meme reach 24.2% to 52.4% (n=84; 44.5% incl. Apr 1 spike), WAU 897 +learning: Cold start should rank proven memes by raw like rate; lr_smoothed's user-bias correction empties the pool for new users +code: src/recommendations/candidates.py cold_start_explore +readout: +--- + # Experiment: Cold Start v2 — Quality-First Phase 1 **Status:** success diff --git a/experiments/completed/2026-03-29-upload-promotion-day1.md b/experiments/2026-03-29-upload-promotion-day1.md similarity index 89% rename from experiments/completed/2026-03-29-upload-promotion-day1.md rename to experiments/2026-03-29-upload-promotion-day1.md index 94f3c85b..8ed65017 100644 --- a/experiments/completed/2026-03-29-upload-promotion-day1.md +++ b/experiments/2026-03-29-upload-promotion-day1.md @@ -1,3 +1,20 @@ +--- +id: upload_promo_day1 +status: killed +hypothesis: A day-1 message explaining meme upload raises upload adoption and D7 retention +primary_metric: upload rate by day 7 +guardrails: D7 retention not lower, session length, spam uploads +unit: user +variants: control:50,treatment:50 +started: 2026-03-29 +check_on: 2026-04-12 +decided: 2026-04-08 +result: uploads 3.2% treatment vs 8.8% control; D1 retention 28.1% vs 28.6%; D7 never measured +learning: A passive 'did you know' message does not create an upload habit and may reduce uploads; needs a CTA after a positive moment +code: upload_promo_day1 assignment +readout: +--- + # Experiment: Upload Promotion Day 1 **Status:** archived diff --git a/experiments/completed/2026-04-20-early-channel-popup.md b/experiments/2026-04-20-early-channel-popup.md similarity index 93% rename from experiments/completed/2026-04-20-early-channel-popup.md rename to experiments/2026-04-20-early-channel-popup.md index ab47c751..d35b0133 100644 --- a/experiments/completed/2026-04-20-early-channel-popup.md +++ b/experiments/2026-04-20-early-channel-popup.md @@ -1,3 +1,20 @@ +--- +id: early_channel_popup +status: shipped +hypothesis: Moving the channel popup from meme #50 to #5 raises channel subscriptions and D7 retention +primary_metric: popup subscribe rate (shown to subscribed) +guardrails: median session >=10 (all) and >=5 (new users), LR >=35% +unit: user +variants: none (pre/post, never read) +started: +check_on: +decided: 2026-10-31 +result: unknown (shipped as default without a readout) +learning: none: unmeasured, do not cite as evidence +code: STATIC_POPUP_RULES in src/tgbot/senders/popups.py +readout: +--- + > **Closed 2026-10-31:** the meme #5 trigger shipped as the default (`STATIC_POPUP_RULES` in `src/tgbot/senders/popups.py`); no A/B and no readout was ever run. Not a live experiment. # Experiment: Early Channel Popup (Meme #5) diff --git a/experiments/2026-04-28-first-meme-nudge.md b/experiments/2026-04-28-first-meme-nudge.md new file mode 100644 index 00000000..8a6ae9af --- /dev/null +++ b/experiments/2026-04-28-first-meme-nudge.md @@ -0,0 +1,38 @@ +--- +id: first_meme_nudge +status: killed +hypothesis: A nudge with the first delivered meme raises first tap for passive new viewers +primary_metric: first meme reacted within 1h +guardrails: mean per-user LR, return days 1-7 +unit: user +variants: control:50,treatment:50 +started: 2026-04-28 +check_on: +decided: 2026-10-30 +result: first tap 15.8% control vs 16.0% treatment (n 1526/1456); reached-5 +2.1 pp (z~1.8); user LR -3.1 pp; d1-7 return 5.0% vs 5.0% +learning: The nudge did not move first tap; it ran 6 months as an orphan split with no registry entry, so register every split +code: removed 2026-10-30 (git history) +readout: docs/analyst/readouts/2026-10-30-experiment-closure.md +--- + +# Experiment: First meme nudge (orphan split) + +Assigned users on their first delivered meme (feed or broadcast) from 2026-04-28. +It had no registry entry and was found during the 2026-10-30 closure. Window: 48h +from assignment, users assigned more than 2 days ago; return measured on users +assigned more than 8 days ago. + +| Metric | Control | Treatment | +|--------|---------|-----------| +| n | 1526 | 1456 | +| First meme reacted within 1h (primary) | 15.8% | 16.0% | +| Any reaction in 48h | 17.5% | 17.5% | +| At least 5 reactions in 48h | 10.7% | 12.8% | +| Mean reactions in 48h | 5.4 | 6.2 | +| Mean per-user LR | 39.2% | 36.1% | +| Return on days 1-7 | 5.0% | 5.0% | + +The primary is flat. Reached-5 is up +2.1 pp (z~1.8) but LR is -3.1 pp, so the simpler +variant (no nudge) shipped and the code was removed. + +Readout: [2026-10-30 closure](../docs/analyst/readouts/2026-10-30-experiment-closure.md). diff --git a/experiments/completed/2026-05-09-recently-liked-blender-v2.md b/experiments/2026-05-09-recently-liked-blender-v2.md similarity index 80% rename from experiments/completed/2026-05-09-recently-liked-blender-v2.md rename to experiments/2026-05-09-recently-liked-blender-v2.md index 24ab79e2..0b996394 100644 --- a/experiments/completed/2026-05-09-recently-liked-blender-v2.md +++ b/experiments/2026-05-09-recently-liked-blender-v2.md @@ -1,3 +1,20 @@ +--- +id: recently_liked_blender_v2 +status: shipped +hypothesis: With LR-stratified assignment, more recently_liked weight for mature users lifts session depth without an LR loss +primary_metric: median session depth (14d) +guardrails: LR >= control - 2 pp, high-volume skipper balance <=1.3x +unit: user +variants: control:50,treatment:50 (stratified by 7d user LR) +started: 2026-05-09 +check_on: +decided: 2026-08-09 +result: median session 18 vs 19 (+5.6%), global LR 63.33% vs 63.67%, median user LR 62.65% vs 71.08% (n ~371/409; 1000/arm gate never reached) +learning: v1's -14 pp LR was an assignment failure (skipper imbalance 2.3x); stratify by user LR; recently_liked deserves more weight +code: src/recommendations/blender_experiments.py, MATURE_BLEND_WEIGHTS in src/feed_turn/planner.py +readout: docs/analyst/recently-liked-blender-v2-closeout.md +--- + # Experiment: recently_liked Blender V2 Created: 2026-05-09 Status: frozen - passive watch only @@ -58,7 +75,7 @@ The v1 failure is therefore treated as an experiment-design failure, not proof t ## Metrics After -See [docs/analyst/recently-liked-blender-v2-closeout.md](../../docs/analyst/recently-liked-blender-v2-closeout.md). +See [docs/analyst/recently-liked-blender-v2-closeout.md](../docs/analyst/recently-liked-blender-v2-closeout.md). | Metric | Control | Treatment | |--------|---------|-----------| diff --git a/experiments/2026-05-12-meme-like-count.md b/experiments/2026-05-12-meme-like-count.md new file mode 100644 index 00000000..780a18b5 --- /dev/null +++ b/experiments/2026-05-12-meme-like-count.md @@ -0,0 +1,38 @@ +--- +id: meme_like_count +status: killed +hypothesis: Showing a meme's like count on the like button (social proof) raises engagement vs a plain heart +primary_metric: mean per-user LR (14d) +guardrails: reaction rate, active days +unit: user +variants: control:50,treatment:50 +started: 2026-05-12 +check_on: +decided: 2026-10-30 +result: user LR 45.59% vs 46.20%, reaction rate 83.40% vs 83.26%, active days/14d 0.85 vs 0.86 (n 6306/6208) +learning: Like counts on the button do not change engagement; a plain heart is simpler and avoids popularity bias +code: removed 2026-10-30 (git history) +readout: docs/analyst/readouts/2026-10-30-experiment-closure.md +--- + +# Experiment: Meme like count (orphan split) + +Every user was assigned on their first delivery after 2026-05-12. It had no registry +entry and was found during the 2026-10-30 closure. Window: 14d from assignment, +non-broadcast sends, users assigned more than 15 days ago. + +| Metric | Control | Treatment | +|--------|---------|-----------| +| n | 6306 | 6208 | +| Users with at least 10 reactions | 507 | 496 | +| Mean per-user LR (primary) | 45.59% | 46.20% | +| Mean per-user LR (>=10 reactions) | 46.03% | 46.13% | +| Mean reaction rate (>=10 sends) | 83.40% | 83.26% | +| Mean reactions in 14d | 30.2 | 33.9 | +| Median reactions (active users) | 27 | 29 | +| Mean active days in 14d | 0.85 | 0.86 | + +Flat on LR, reaction rate and activity; the mean-reactions gap is driven by heavy +users. Control (plain heart) shipped and the code was removed. + +Readout: [2026-10-30 closure](../docs/analyst/readouts/2026-10-30-experiment-closure.md). diff --git a/experiments/completed/2026-05-19-crossposting-forward-intent-ranker.md b/experiments/2026-05-19-crossposting-forward-intent-ranker.md similarity index 87% rename from experiments/completed/2026-05-19-crossposting-forward-intent-ranker.md rename to experiments/2026-05-19-crossposting-forward-intent-ranker.md index 68c55f35..3a716523 100644 --- a/experiments/completed/2026-05-19-crossposting-forward-intent-ranker.md +++ b/experiments/2026-05-19-crossposting-forward-intent-ranker.md @@ -1,4 +1,21 @@ -> **Closed 2026-10-31, never built:** superseded by crosspost v4/v5/v5f scoring (see `docs/growth/HYPOTHESES.md`). Not a live experiment. +--- +id: crossposting_forward_intent_ranker +status: killed +hypothesis: A forward-intent score from exact-meme share clicks and good share users predicts channel forwards better than source prior +primary_metric: forwards per 1k views at 24h +guardrails: no empty posting slots, views not -20% +unit: channel_post +variants: none (never built) +started: +check_on: +decided: 2026-10-31 +result: unknown (never built); 2 manual source-prior posts: RU 27.5 fwd/1k, EN 0 +learning: Source prior alone found acceptable RU memes but no breakouts; superseded by v4/v5/v5f scoring +code: score_version=3 decision-log fields +readout: +--- + +> **Closed 2026-10-31, never built:** superseded by crosspost v4/v5/v5f scoring (see [crosspost_src_fwd_v5f](2026-11-01-crosspost-src-forward-rate.md) and the README). Not a live experiment. # Experiment: Crossposting Forward-Intent Ranker Created: 2026-05-19 diff --git a/experiments/completed/2026-07-14-cold-start-candidate-filter.md b/experiments/2026-07-14-cold-start-candidate-filter.md similarity index 83% rename from experiments/completed/2026-07-14-cold-start-candidate-filter.md rename to experiments/2026-07-14-cold-start-candidate-filter.md index 9b515c04..eb019f13 100644 --- a/experiments/completed/2026-07-14-cold-start-candidate-filter.md +++ b/experiments/2026-07-14-cold-start-candidate-filter.md @@ -1,3 +1,20 @@ +--- +id: cold_start_candidate_guardrails +status: shipped +hypothesis: Excluding 4 weak sources from cold-start positions 2-10 improves depth for true-new users +primary_metric: reached-10 sends (true-new users) +guardrails: first-meme continuation >=84%, first-10 LR not -3 pp, no fallback spike, no routing leakage +unit: user +variants: control,treatment (guarded cold start) +started: 2026-07-16 +check_on: +decided: 2026-07-29 +result: reached-10 32.4% treatment (n=37) vs 11.1% concurrent control vs 26.8% baseline; reached-5 70.3% vs 44.4%; all guardrails pass +learning: Dropping a few weak sources from early positions lifts new-user depth (directional, n=37); mostly validates guarded phase 1 +code: COLD_START_CANDIDATE_GUARDRAILS_ENABLED, src/recommendations/candidates.py +readout: docs/analyst/readouts/2026-07-29-cold-start-candidate-filter-readout.md +--- + # Experiment: Cold Start Candidate Filter Created: 2026-07-14 @@ -107,9 +124,9 @@ routes, not leakage from `cold_start_explore_guarded` or `cold_start_adapt_guarded`. The early six-user spot check was intentionally non-decisive; see -[`2026-07-18-cold-start-candidate-filter-spotcheck.md`](../../docs/analyst/readouts/2026-07-18-cold-start-candidate-filter-spotcheck.md). +[`2026-07-18-cold-start-candidate-filter-spotcheck.md`](../docs/analyst/readouts/2026-07-18-cold-start-candidate-filter-spotcheck.md). The final evidence is in -[`2026-07-29-cold-start-candidate-filter-readout.md`](../../docs/analyst/readouts/2026-07-29-cold-start-candidate-filter-readout.md). +[`2026-07-29-cold-start-candidate-filter-readout.md`](../docs/analyst/readouts/2026-07-29-cold-start-candidate-filter-readout.md). ## Conclusion diff --git a/experiments/2026-08-09-broadcast-hq-pick.md b/experiments/2026-08-09-broadcast-hq-pick.md new file mode 100644 index 00000000..017bd665 --- /dev/null +++ b/experiments/2026-08-09-broadcast-hq-pick.md @@ -0,0 +1,55 @@ +--- +id: broadcast_reengagement_hq_pick +status: shipped +hypothesis: Picking the push meme by user x source affinity x raw like rate beats a blind queue pop on react-within-1h +primary_metric: react_within_1h % +guardrails: LR not -5 pp vs fallback, fallback share <80% +unit: user +variants: none (HQ pick vs queue fallback labels, not randomized) +started: 2026-08-09 +check_on: 2026-08-16 +decided: 2026-10-30 +result: 14d: 5,533 sends, LR 57.2%, react<1h 36.5%, queue fallback 0.4% (pre-HQ LR ~54%, n~39) +learning: Affinity x like-rate is a solid default push meme +code: BROADCAST_HIGH_QUALITY_PICK_ENABLED, src/recommendations/broadcast_pick.py +readout: docs/analyst/broadcast-reengagement.sql +--- + +# H3b — High-quality meme pick for retention broadcasts + +| Field | Value | +|-------|--------| +| **ID** | `broadcast_reengagement_hq_pick` | +| **Status** | **KEEP 2026-10-30**: 14d LR 57.2%, react<1h 36.5%, queue fallback 0.4% | +| **Code** | `src/recommendations/broadcast_pick.py`, kill switch `BROADCAST_HIGH_QUALITY_PICK_ENABLED` | +| **Labels** | `broadcast_reengagement_hq` (SQL pick) vs `broadcast_reengagement` (queue fallback) | +| **SQL** | `docs/analyst/broadcast-reengagement.sql` | + +## Hypothesis + +Choosing the reengagement meme by **user×source affinity × raw like rate** +(avoiding majority-dislike sources) beats blind feed-queue pop on: + +1. **react_within_1h %** (primary — “came back because of this push”) +2. like rate among reacted +3. p50 `sec_to_react` among in-window reactions (secondary) + +## Pre-HQ baseline (2026-08-09, ~39 `broadcast_reengagement` sends) + +- like rate ~54% (small n) +- only ~4 reacts under 2 minutes — delayed open is the norm + +## Decision rules (day-7 from HQ deploy) + +**Keep HQ ON** if: + +- `broadcast_reengagement_hq` react_within_1h ≥ queue-fallback **or** ≥ pre-HQ baseline + directionally better LR; +- no send failure spike (empty HQ → fallback should keep volume). + +**Disable** (`BROADCAST_HIGH_QUALITY_PICK_ENABLED=false`) if HQ pool empty for +most users (fallback rate >80%) or LR collapses >5 pp vs fallback. + +## Next check + +- **2026-08-12** smoke: any `_hq` rows? fallback share? +- **2026-08-16** primary HQ vs fallback vs feed diff --git a/experiments/2026-08-09-broadcast-reengagement-label.md b/experiments/2026-08-09-broadcast-reengagement-label.md new file mode 100644 index 00000000..3c0f6b7d --- /dev/null +++ b/experiments/2026-08-09-broadcast-reengagement-label.md @@ -0,0 +1,41 @@ +--- +id: broadcast_reengagement_label +status: shipped +hypothesis: Labeling retention pushes separately from feed engines lets us measure true in-session sec_to_react +primary_metric: broadcast vs feed p50 sec_to_react +guardrails: +unit: user +variants: none (labeling) +started: 2026-08-09 +check_on: 2026-08-16 +decided: 2026-10-30 +result: broadcast p50 sec-to-react 3.8-5.5k s vs feed 5.7 s +learning: Push reactions are delayed opens; keep broadcast_* rows out of feed dwell and LR analysis +code: flows/broadcasts/meme.py recommended_by=broadcast_* +readout: docs/analyst/broadcast-reengagement.sql +--- + +# H3 — Broadcast delivery label for dwell analytics + +| Field | Value | +|-------|--------| +| **ID** | `broadcast_reengagement_label` | +| **Status** | **DONE 2026-10-30**: labels live, broadcast p50 sec-to-react 3.8–5.5k s vs feed 5.7 s | +| **Shipped** | 2026-08-09 with #340 | +| **Code** | `flows/broadcasts/meme.py` → `recommended_by=broadcast_*` | +| **SQL** | `docs/analyst/broadcast-reengagement.sql`, dwell SQL | + +## Hypothesis + +Labeling retention pushes separately from feed engines lets us measure true +in-session `sec_to_react` and optimize broadcast timing for **fast** reaction. + +## Success criteria + +1. ✅ Labeled rows exist (`broadcast_reengagement` / `_hq`) +2. Broadcast p50 `sec_to_react` **higher** than feed (delayed opens expected) +3. Share of broadcast reactions with `sec_to_react > 1h` documented + +## Next check + +- **2026-08-12 / 16** — full feed vs broadcast dwell table diff --git a/experiments/2026-08-09-cold-start-floors.md b/experiments/2026-08-09-cold-start-floors.md new file mode 100644 index 00000000..386ec544 --- /dev/null +++ b/experiments/2026-08-09-cold-start-floors.md @@ -0,0 +1,51 @@ +--- +id: cold_start_first_impression_v2 +status: shipped +hypothesis: Raising cold_start_explore floors (raw LR >=0.50, >=25 reactions) and best_uploaded in CS3 lifts first-10 LR and reached-10 +primary_metric: first tap, first-10 LR +guardrails: no empty-queue spike for new users, CS3 continuation +unit: user +variants: none (pre/post) +started: 2026-08-09 +check_on: 2026-08-16 +decided: 2026-10-30 +result: first tap 61.5% to 62.0% (n=100 post), underpowered; no rollback trigger +learning: Stricter quality floors did not measurably move first tap; kept as harmless +code: cold_start_explore raw-LR floor, CS3 blend (best_uploaded + text_light) +readout: docs/analyst/metrics.sql +--- + +# H5 — Cold-start first-session quality floors + +| Field | Value | +|-------|--------| +| **ID** | `cold_start_first_impression_v2` | +| **Status** | **KEEP 2026-10-30**: no rollback trigger. First-tap 61.5 → 62.0% (n=100 post), so the lift is unproven | +| **Code** | `cold_start_explore` raw-LR floor; CS3 blend → `best_uploaded` + text-light | +| **SQL** | engine slice in `docs/analyst/metrics.sql` / cold_start quality scripts | + +## Hypothesis + +**A1.** Raising `cold_start_explore` floors (raw like rate ≥0.50, ≥25 reactions, +order by `lr_smoothed`) lifts first-10 LR and reached-10 vs pre-change +(guarded explore was ~**18%** LR on 7d — too weak for first impression). + +**A2.** CS3 (memes 16–29) replacing `like_spread` with `best_uploaded_memes` +improves LR/continuation in that band without starving personalization +(`cold_start_adapt` stays fixed_pos 0). + +## Decision rules (day-7) + +**Keep** if first-10 LR for `cold_start_explore*` ≥ pre-change + directionally +higher reached-5/10; no empty-queue spike for new users. + +**Rollback explore floors** if explore pool empty → fallback share spikes and +session depth drops. + +**Rollback CS3** if `best_uploaded` share causes lower continuation vs prior +week CS3 band. + +## Next check + +- **2026-08-12** smoke: explore volume + LR +- **2026-08-16** first-10 / CS3 band metrics diff --git a/experiments/2026-08-09-skip-is-next.md b/experiments/2026-08-09-skip-is-next.md new file mode 100644 index 00000000..1123cf6c --- /dev/null +++ b/experiments/2026-08-09-skip-is-next.md @@ -0,0 +1,36 @@ +--- +id: skip_is_next_not_ban +status: shipped +hypothesis: The dislike button is mostly 'next'; ranking should prefer likes + dwell, not treat skips as bans +primary_metric: sec_to_react: skip vs like +guardrails: +unit: user +variants: none (measurement contract) +started: 2026-08-09 +check_on: +decided: 2026-08-09 +result: skip p50 4.7 s vs like 7.1 s; instant skip 19% vs like 9% (7d feed) +learning: Treat skip as next, not dislike; no dwell-weight ship without an offline counterfactual +code: +readout: docs/analyst/dwell-feed-vs-broadcast.sql +--- + +# H4 — Skip ≠ hate (measurement contract, not a ship) + +| Field | Value | +|-------|--------| +| **ID** | `skip_is_next_not_ban` | +| **Status** | **accepted product doctrine** | +| **Evidence** | skip p50 4.7s vs like 7.1s; instant skip 19% vs like 9% (7d feed) | +| **SQL** | `docs/analyst/dwell-feed-vs-broadcast.sql` | + +## Hypothesis (ongoing) + +Dislike button is primarily “next”; ranking must prefer **likes + dwell**, not +raw skip counts as bans. Future work: down-weight `sec_to_react < 2s` skips and +ignore `>1h` reactions in affinity. + +## Next check + +- **2026-08-16** re-run dwell percentiles; compare to baseline readout +- No ship decision until a dedicated dwell-weight PR with offline counterfactual diff --git a/experiments/2026-08-09-source-affinity-demote.md b/experiments/2026-08-09-source-affinity-demote.md new file mode 100644 index 00000000..a77ebd92 --- /dev/null +++ b/experiments/2026-08-09-source-affinity-demote.md @@ -0,0 +1,53 @@ +--- +id: source_affinity_demote_v1 +status: shipped +hypothesis: Soft-demoting sources with ndislikes > nlikes (n>=5) by x0.15 raises feed quality without empty-queue symptoms +primary_metric: weekly like rate +guardrails: sends/day and active users not cliffing, empty-queue alerts +unit: user +variants: none (pre/post, 100%) +started: 2026-08-09 +check_on: 2026-08-16 +decided: 2026-10-30 +result: weekly LR 32.6-36.1% post vs 30.7-33.5% pre, no send-volume cliff +learning: Soft demote of majority-dislike sources is safe (+1-3 pp LR); hard-block was rejected (cuts ~57% of inventory offline) +code: RECOMMENDATION_DEMOTE_*, disliked_source_demote_sql +readout: docs/analyst/source-affinity-demote-guardrails.sql, docs/analyst/dwell-feed-vs-broadcast.sql +--- + +# H2 — Soft demote majority-dislike sources + +| Field | Value | +|-------|--------| +| **ID** | `source_affinity_demote_v1` (config flags, not A/B) | +| **Status** | **shipped default ON** (PR #340). **KEEP 2026-10-30**: no volume cliff, LR +1–3 pp | +| **Shipped** | 2026-08-09 deploy `687fd46a` | +| **Code** | `RECOMMENDATION_DEMOTE_*`, `disliked_source_demote_sql` | +| **SQL** | `docs/analyst/source-affinity-demote-guardrails.sql` + inventory section of `dwell-feed-vs-broadcast.sql` | +| **Baseline** | `docs/analyst/readouts/2026-08-09-dwell-and-source-demote-baseline.md` | + +## Hypothesis + +Deprioritizing sources with `ndislikes > nlikes` (n≥5) by score ×0.15 increases +feed quality (higher session continuation / slightly higher LR on demotable +traffic) **without** increasing empty-queue symptoms, unlike hard-block (~57% +inventory cut offline). + +## Decision rules (day-7) + +**Keep ON** if: + +1. Sends/day and active users not cliffing vs week before deploy; +2. Global like rate not down >3 pp vs pre-deploy 7d window; +3. No spike in empty-queue alerts (Sentry/logs: `has empty meme queue`). + +**Tune** multiplier (0.15 → 0.3) if LR up but users complain of “same sources”. +**Disable** (`RECOMMENDATION_DEMOTE_DISLIKED_SOURCES=false`) if empty-queue or +volume collapse. + +## Next check + +- **2026-08-12** smoke volume/LR +- **2026-08-16** full guardrails vs baseline + +Rejected: hard-block majority-dislike sources as the default (opt-in flag only). diff --git a/experiments/2026-08-09-viral-shares-blender-v1.md b/experiments/2026-08-09-viral-shares-blender-v1.md new file mode 100644 index 00000000..3d0a938f --- /dev/null +++ b/experiments/2026-08-09-viral-shares-blender-v1.md @@ -0,0 +1,168 @@ +--- +id: viral_shares_blender_v1 +status: killed +hypothesis: 0.2 weight on a viral_shares engine in the mature blend raises non-self share clickers and invites per 1k sends +primary_metric: unique non-self share clickers per 1k sends +guardrails: p50 session >= control - 10%, LR >= control - 3 pp +unit: user +variants: control:50,treatment:50 +started: 2026-08-09 +check_on: 2026-08-16 +decided: 2026-10-30 +result: clickers/1k 0.031 control vs 0.025 treatment (n 344/350 users, 426k/395k sends); invites 5 vs 7; user LR -2.2 pp; engine was 0.76% of treatment sends +learning: Ranking by past share clicks did not raise sharing; the engine barely served, so the arms hardly differed +code: candidates.viral_shares, src/recommendations/blender_experiments.py (assignment removed) +readout: docs/analyst/readouts/2026-10-30-experiment-closure.md +--- + +# Experiment: Viral shares blender v1 + +Created: 2026-08-09 +Status: **completed — KILL (2026-10-30)** +Owner: engineer / analyst +Deployed: 2026-08-09 (~14:20 UTC first assignment) +**Measure after: 2026-08-16** (primary) +**Final if underpowered: 2026-08-23** +Smoke: 2026-08-12 + +Registry entry H1: see the end of this file. + +## Hypothesis + +Injecting a dedicated `viral_shares` engine (memes ranked by share-click / +invite conversion) into the mature blend at weight **0.2** (stolen from +`lr_smoothed`) increases unique **non-self** share clickers and new-user +invites per 1k memes sent without reducing session depth (p50 memes/session). + +## Changes Made + +- Engine: `src/recommendations/candidates.py::viral_shares` +- Experiment: `viral_shares_blender_v1` in `src/recommendations/blender_experiments.py` +- Wiring: mature path uses `get_mature_blend_weights_with_experiments` +- Delivery prep seam: `src/tgbot/senders/delivery.py` +- Crosspost fix: share-click CTEs accept both `m_` and `s_` deep links + +## Assignment + +- Eligible: mature users via mature blend path (`nmemes_sent ≥ 100`) +- Strategy: `sha256(experiment_id:user_id) % 2` +- Control: base mature weights (after recently_liked v2 default) +- Treatment: base + `viral_shares: 0.2`, `lr_smoothed` reduced by 0.2 +- Code constant `SAMPLE_GATE_PER_VARIANT = 1000` is **legacy**; decision sample + for this WAU is **≥80 users/variant and ≥2k sends/variant** at day-7, else + decide at day-14 (see H1 decision rules below) + +## Primary metrics + +1. Unique non-self share clickers (`user_deep_link_log` m_/s_) per 1k memes sent +2. New-user invites (`user.inviter_id`) per 1k memes sent + +## Guardrails + +- p50 session length (30m gap) — kill if treatment < control − 10% relative +- Like rate — kill if treatment < control − 5 pp (with enough volume) +- Engine continuation@30m for `viral_shares` vs `lr_smoothed` + +## Ship / kill rules + +See **H1 Decision rules** below. + +## Interim (2026-08-09 15:44 UTC) + +Full numbers: interim readout removed 2026-10-31 (in git history); final numbers in [`docs/analyst/readouts/2026-10-30-experiment-closure.md`](../docs/analyst/readouts/2026-10-30-experiment-closure.md) + +- 10 control / 12 treatment users; 0 share clickers; 0 invites +- 13 `viral_shares` sends on treatment (wiring OK) +- **Verdict: HOLD** + +## Metrics Before / After + +| Window | control n | treatment n | clickers/1k C/T | invites/1k C/T | p50 session C/T | Decision | +|--------|-----------|-------------|-----------------|----------------|-----------------|----------| +| 2026-08-09 interim | 10 | 12 | 0 / 0 | 0 / 0 | 16.5 / 19.0 | HOLD | +| 2026-10-30 day-82 (final) | 344 | 350 | 0.031 / 0.025 | 0.012 / 0.018 | 14 / 10 (pooled) | KILL | + +## Conclusion + +**KILL (2026-10-30).** Enrollment stopped in code; every mature user gets the +shipped default blend (`MATURE_BLEND_WEIGHTS`). Existing assignment rows stay +for readouts. The `viral_shares` engine stays in `candidates.py` but no blend uses it. + +- Primary failed: unique non-self share clickers per 1k sends 0.031 control vs + 0.025 treatment (13 vs 10 clickers, 426k / 395k sends). +- Invites 5 vs 7 (0.012 vs 0.018 per 1k). One treatment inviter accounts for 4, so this is noise. +- Mean per-user LR 49.7% vs 47.5% (−2.2 pp, inside the 3 pp guardrail). +- The engine reached only 0.76% of treatment sends, so the arms barely differed. +- Pooled p50 session dropped 14 → 10, which trips the frozen kill trigger. The + user-level median of per-user p50 is flat (10 vs 10), so the drop is + heavy-user composition. It is not why we killed the test. + +Readout: [`docs/analyst/readouts/2026-10-30-experiment-closure.md`](../docs/analyst/readouts/2026-10-30-experiment-closure.md). +The canned SQL §4 invite multiplier bug was fixed in the same PR. + +## Readout SQL + +`docs/analyst/viral-shares-blender-v1.sql` (removed 2026-10-31; in git history) + + +## Registry entry (moved from docs/growth/HYPOTHESES.md, 2026-10-31) + +## H1 — `viral_shares` in mature blend (A/B) + +| Field | Value | +|-------|--------| +| **ID** | `viral_shares_blender_v1` | +| **Status** | **KILLED 2026-10-30**: clickers/1k C 0.031 vs T 0.025. Enrollment stopped, all users on control | +| **Shipped** | 2026-08-09 ~14:20 UTC (first assignment) | +| **Code** | `candidates.viral_shares`, `blender_experiments.get_viral_shares_blender_v1_weights` | +| **Note** | `2026-08-09-viral-shares-blender-v1.md` | +| **SQL** | removed 2026-10-31 (killed; in git history) | +| **Readout log** | removed 2026-10-31 (killed; in git history); final numbers in the 2026-10-30 closure readout | + +### Hypothesis (precise) + +Giving mature users (`nmemes_sent ≥ 100`) weight **0.2** on engine +`viral_shares` (memes ranked by `invited_count / ln(nmemes_sent+e)`), taken from +`lr_smoothed`, increases **unique non-self share clickers per 1k sends** and +**new-user invites per 1k sends** vs control, without harming session depth. + +### Interim baseline (2026-08-09 15:44 UTC, ~1.4h) + +| Variant | n_users | memes_sent | LR % | viral_sends | clickers/1k | invites/1k | p50 session | +|---------|---------|------------|------|-------------|-------------|------------|-------------| +| control | 10 | 297 | 39.6 | 0 | 0 | 0 | 16.5 | +| treatment | 12 | 367 | 54.2 | 13 (3.5%) | 0 | 0 | 19.0 | + +**Do not decide from this.** n≪gate; 0 growth events expected in first hours. +Engine is live (13 sends, high LR noise). + +### Decision rules (day-7 / day-14) + +**Sample (realistic for WAU~380):** +Do **not** wait for 1000/arm (unrealistic). Prefer: + +- day-7 if **≥80 users/variant** and **≥2k sends/variant**, else hold to day-14; +- day-14 decide even if underpowered (ship only if direction clear + guardrails OK). + +**Ship treatment → mature default** if all: + +1. `unique_clickers_per_1k_sends` treatment ≥ control **and** absolute lift makes + sense at our volume (or invites/1k clearly higher); +2. Guardrail: p50 session length treatment ≥ control − **10%** relative; +3. Guardrail: like rate treatment ≥ control − **3 pp**; +4. Engine slice: `viral_shares` continuation@30m not < lr_smoothed − **5 pp** + (on ≥100 viral sends). + +**Kill (remove weight, freeze enrollment to control)** if: + +- session p50 drops >10% relative, or +- LR drops >5 pp with ≥2k sends/arm, or +- day-14 still ~0 viral_engine exposure on treatment (wiring bug). + +**Hold** if underpowered or metrics mixed. + +### Next check + +- **2026-08-12** smoke: cohort sizes, viral % of treatment sends, LR +- **2026-08-16** primary +- **2026-08-23** final if needed diff --git a/experiments/2026-08-10-crosspost-taste-cohort-shadow.md b/experiments/2026-08-10-crosspost-taste-cohort-shadow.md new file mode 100644 index 00000000..50db1931 --- /dev/null +++ b/experiments/2026-08-10-crosspost-taste-cohort-shadow.md @@ -0,0 +1,60 @@ +--- +id: crosspost_taste_cohort_v1 +status: killed +hypothesis: Likes from a fixed top-50 taste cohort add a weak signal for channel forwards on top of like volume +primary_metric: mean f1k: top vs bottom half by n_taste_likes (posted, n_taste >=1) +guardrails: shadow only, no pick change +unit: channel_post +variants: shadow (logged only) +started: 2026-08-10 +check_on: 2026-08-24 +decided: 2026-10-30 +result: f1k top vs bottom half 28.52 vs 28.72 (n=287); taste>=1 vs taste=0 28.62 vs 29.87 +learning: A hand-picked taste cohort that looked good offline (lift 1.19) carried no live signal +code: removed: src/crossposting/taste_cohort.py, data/ru_taste_cohort_v1.json (git history) +readout: docs/analyst/readouts/2026-10-30-experiment-closure.md +--- + +# H7 — Taste cohort soft signal (SHADOW) + +| Field | Value | +|-------|--------| +| **ID** | `crosspost_taste_cohort_v1` | +| **Status** | **KILLED as signal 2026-10-30**: top vs bottom n_taste f1k 28.52 vs 28.72 (n=287). No canary | +| **Cohort** | `src/crossposting/data/ru_taste_cohort_v1.json` (top 50) | +| **Code** | `taste_cohort.py`, `_enrich_candidates_with_taste_shadow` | +| **SQL** | removed 2026-10-31 (killed; in git history) | +| **Experiment note** | `2026-08-10-crosspost-v4-and-taste-cohort.md` | + +## Hypothesis + +A fixed set of ~50 users whose historical likes co-occur with high channel +fwd/1k provide a **weak but real** additional signal (soft boost), not a +replacement for like volume. + +## Offline (2026-08-10) + +- top50 likes top-20% lift **1.19** vs all-likes **1.10**; beats random-50 p95 **1.14** +- coverage ≥1 taste like ~**18%** of posts → never sole filter + +## Shadow fields on decision candidates + +`n_taste_likes`, `taste_boost_shadow` (= `1+0.15*min(n,5)`), `taste_cohort_version` + +## Predictions + +| Date | Pass if | +|------|---------| +| **2026-08-24** | Among mature picks with n_taste≥1, higher n_taste half has higher mean f1k (n≥20) | +| Canary | Only if H6 not RED and shadow pass → optional soft boost flag | + +## Not doing + +- Hardcode 50 users as only ranker +- Post less often until H6 mature fails + +## Refresh + +`python scripts/crosspost_taste_cohort.py` + +Design and frozen predictions: [2026-08-10-crosspost-v4-and-taste-cohort.md](2026-08-10-crosspost-v4-and-taste-cohort.md). diff --git a/experiments/completed/2026-08-10-crosspost-v4-and-taste-cohort.md b/experiments/2026-08-10-crosspost-v4-and-taste-cohort.md similarity index 64% rename from experiments/completed/2026-08-10-crosspost-v4-and-taste-cohort.md rename to experiments/2026-08-10-crosspost-v4-and-taste-cohort.md index bb2efe46..e0556d4e 100644 --- a/experiments/completed/2026-08-10-crosspost-v4-and-taste-cohort.md +++ b/experiments/2026-08-10-crosspost-v4-and-taste-cohort.md @@ -1,16 +1,33 @@ +--- +id: crosspost_meme_level_v1 +status: shipped +hypothesis: RU channel pick by bot like volume ln(nlikes+1) x source prior gets more forwards than v2; bot like rate does not +primary_metric: hit rate (f1k >=31 or forwards_24h >=12) +guardrails: views not < v2 - 15%, no empty slots +unit: channel_post +variants: none (v2 then v4 at 100%) +started: 2026-08-10 +check_on: 2026-08-17 +decided: 2026-10-30 +result: hit rate 32.1% to 41.0% (+8.9 pp, z~2.4, n 299/373), f1k 26.2 to 28.9; manual posts also +3.4 f1k (channel drift) +learning: Bot like volume, not like rate, predicts channel forwards; part of the lift is channel drift +code: CROSSPOST_RU_MEME_LIKE_VOLUME_ENABLED, src/crossposting/service.py (score_version=4) +readout: docs/analyst/crossposting-v4-like-volume.sql +--- + # Experiments: Crosspost RU v4 like-volume + taste-cohort shadow (H6 / H7) Created: 2026-08-10 Owner: engineer / analyst Status: **completed (2026-10-30)**: H6 kept at 100%, H7 killed as a signal Channel: `tgchannelru` (@fastfoodmemes) -**Predictions frozen:** 2026-08-10 (this file + `experiments/HYPOTHESES.md` H6/H7) +**Predictions frozen:** 2026-08-10 (this file; H6 registry entry at the end, H7 in 2026-08-10-crosspost-taste-cohort-shadow.md) > **Amendment 2026-10-30:** H7 taste cohort and the H8 `v4_x_src_v1` shadow are killed > (corr ≈ 0 / −0.09 with zf). Their code was removed. They are replaced by the v5 shadow: -> [`2026-10-30-crosspost-v5-shadow.md`](../active/2026-10-30-crosspost-v5-shadow.md). +> [`2026-10-30-crosspost-v5-shadow.md`](2026-10-30-crosspost-v5-shadow.md). -Authoritative calendar also in `experiments/HYPOTHESES.md` (H6, H7). +Check dates are in the front matter of this file and the H7 file. Do **not** edit pass/fail thresholds after freeze without a dated amendment; fill **Metrics after** only. @@ -104,7 +121,7 @@ python scripts/crosspost_taste_cohort.py --days 120 --top 50 --min-n 8 ## How future-me / agents check who was right -1. Open this file + `experiments/HYPOTHESES.md` H6/H7. +1. Open this file and 2026-08-10-crosspost-taste-cohort-shadow.md. 2. Run: - `docs/analyst/crossposting-v4-like-volume.sql` - taste shadow SQL in `docs/analyst/crossposting-taste-shadow.sql` (removed 2026-10-31 after the H7 kill; in git history) @@ -148,4 +165,62 @@ Full interim writeup: `docs/analyst/crossposting-v4-interim-readout.md` pre-registered Δ>0 test fails. The decision_log fields stay until the crossposting PR that replaces shadow scoring removes them. -Readout: [`docs/analyst/readouts/2026-10-30-experiment-closure.md`](../../docs/analyst/readouts/2026-10-30-experiment-closure.md). +Readout: [`docs/analyst/readouts/2026-10-30-experiment-closure.md`](../docs/analyst/readouts/2026-10-30-experiment-closure.md). + + +## Registry entry (moved from docs/growth/HYPOTHESES.md, 2026-10-31) + +## H6 — Crosspost RU: meme-level bot volume (not LR) × source prior + +| Field | Value | +|-------|--------| +| **ID** | `crosspost_meme_level_v1` / **score_version=4** | +| **Status** | **KEEP at 100% 2026-10-30**: hit rate 32.1 → 41.0% (+8.9 pp, z≈2.4), f1k 26.2 → 28.9 | +| **Kill switch** | `CROSSPOST_RU_MEME_LIKE_VOLUME_ENABLED=false` → v2 | +| **Readout offline** | `docs/analyst/readouts/2026-08-09-crosspost-meme-level-offline.md` | +| **Readout online** | `docs/analyst/crossposting-v4-like-volume.sql` | +| **n offline** | 611 RU image posts / 120d | + +### Hypothesis + +Among channel candidates, **like volume** (`ln(nlikes+1)` via `meme_stats`) improves +24h channel forwards beyond source prior. Bot **like rate** does not. + +### Offline result (2026-08-09) + +- pre_lr vs channel: ~0 (reject LR feature) +- pre_likes residual Spearman ~**0.16–0.20** +- top-20% **src × log1p(pre_likes)** lift **1.14×** forwards (time-split test too) +- HIT: f1k ≥ **~31** OR forwards ≥ **12** + +### Production formula (score_version=4) + +```text +v2_score * LN(COALESCE(nlikes,0) + 1) +``` + +Logged in decision candidates: `like_volume_factor`, `like_volume_enabled`. + +### Online success (7–14d after deploy) + +Compare v4 mature posts vs last 30d v2 baseline: + +- hit_rate_pct not down; preferably **+3pp** +- avg forwards_24h ≥ v2 baseline +- avg views_24h not < v2 − 15% +- Kill if hit_rate collapses or empty slots spike + +### Online early (2026-08-10 ~17 UTC) + +- 4 v4 posts live; **0 mature** yet — WATCH +- age-matched ~6h: v4 f1k **30.0** vs v2 **32.6** (n=4, not RED) +- Frequency: **keep 5/day** (do not post more/less until mature readout) + +### Next check + +- **2026-08-12**: mature n≥8 if possible +- **2026-08-17** (~7d): `crossposting-v4-like-volume.sql` keep/kill +- Full plan: `2026-08-10-crosspost-v4-and-taste-cohort.md` + +Rejected offline: ranking crossposts by bot like rate (no correlation with forwards). +H7 (taste cohort) has its own record: [2026-08-10-crosspost-taste-cohort-shadow.md](2026-08-10-crosspost-taste-cohort-shadow.md). diff --git a/experiments/completed/2026-08-11-bot2channel-ml-lab.md b/experiments/2026-08-11-bot2channel-ml-lab.md similarity index 52% rename from experiments/completed/2026-08-11-bot2channel-ml-lab.md rename to experiments/2026-08-11-bot2channel-ml-lab.md index 659dde6a..532ca987 100644 --- a/experiments/completed/2026-08-11-bot2channel-ml-lab.md +++ b/experiments/2026-08-11-bot2channel-ml-lab.md @@ -1,3 +1,20 @@ +--- +id: bot2channel_ml_lab_v1 +status: killed +hypothesis: Simple models on pre-post bot features + source prior beat ln(pre_likes+1) for RU channel picks (then shadow v4_x_src_v1) +primary_metric: walk-forward top-20% f1k lift; live corr(shadow_score, f1k) +guardrails: no production pick change +unit: channel_post +variants: shadow (logged only) +started: 2026-08-11 +check_on: 2026-08-25 +decided: 2026-10-30 +result: lifetime walk-forward HOLD (n=3889); live shadow corr -0.092 (n=363 posted), agreed vs disagreed f1k 29.01 vs 29.10 +learning: A one-split offline win (lift 1.19) did not survive lifetime walk-forward or live shadow; require both before trusting a ranker +code: removed: experiments/ml-crosspost-bot2channel/ (commit 177cad0e), v4_x_src_v1 shadow fields +readout: docs/analyst/readouts/2026-10-30-experiment-closure.md +--- + # Experiment: Bot→channel ML lab (offline) Created: 2026-08-11 @@ -112,3 +129,67 @@ are range-restricted and near zero. Agreed vs disagreed picks: f1k 29.01 vs 29.1 The "clearly better than v4" bar was not met. Offline lab code stays under `experiments/ml-crosspost-bot2channel/` for reference. The decision_log shadow fields stay until the crossposting PR that replaces shadow scoring removes them. + + +## Registry entry (moved from docs/growth/HYPOTHESES.md, 2026-10-31) + +## H8 — Bot→channel ML lab (OFFLINE) + +| Field | Value | +|-------|--------| +| **ID** | `bot2channel_ml_lab_v1` | +| **Status** | **KILLED as signal 2026-10-30**: offline HOLD vs v4, and the live shadow corr is −0.09 (see below) | +| **Code** | `experiments/ml-crosspost-bot2channel/` — removed 2026-10-30; recover from commit `177cad0e` | +| **Note** | `2026-08-11-bot2channel-ml-lab.md` | +| **Report** | `experiments/ml-crosspost-bot2channel/reports/2026-08-11-models.md` | + +### Hypothesis + +Simple linear/logistic models on pre-post bot features + source prior beat +`ln(pre_likes+1)` on time-holdout top-20% f1k lift without tree overfit. + +### First result (2026-08-11) + +- n=624 labeled; single 70/30: logreg lift 1.187 vs v4 1.114 (looked good once) +- HGB depth3 **overfits** — discard for ship + +### Walk-forward (authoritative, 2026-08-11) + +- 3 expanding folds (~93 test each); bar: lift ≥ v4+0.05 on ≥2/3 + spearman gap <0.25 +- **logreg PASS** (2/3, mean lift 1.190 vs v4 1.080) +- **ridge PASS** (2/3, mean lift 1.171) +- HGB FAIL (overfit gap ~0.5) +- Report: `experiments/ml-crosspost-bot2channel/reports/2026-08-11-walkforward.md` +- RU summary: `.../reports/2026-08-11-otchet-ru.md` +- **No production pick change** — next is shadow score only + + +### Deep lifetime re-run (2026-08-11 evening) + +- Telethon full crawl: 11928 msgs, 4897 sc_ joinable → dataset **n=3889** (2023-11..2026-08) +- Label = lifetime f1k (TG now); HIT = f1k≥train p75 +- Walk-forward: **HOLD** — logreg/ridge/HGB do **not** beat v4 on ≥2/3 folds +- Source prior often ≥ volume on long history; bot volume ~flat on lifetime +- HTML: `experiments/ml-crosspost-bot2channel/reports/2026-08-11-results.html` +- Hypothesis battery: hybrids volume×source lead mean lift; **no bar PASS** + (`reports/2026-08-11-hypothesis-battery.html`) + +### Shadow hybrid in decision_log (shipping) + +| Field | Value | +|-------|--------| +| **Status** | **KILLED 2026-10-30**: corr(shadow, f1k) −0.092 on 363 posted picks, agreed vs disagreed f1k 29.01 vs 29.10. Fields stay until the crossposting PR replaces shadow scoring | +| **Formula** | `shadow_score = ln(nlikes+1) * src_quality_mult` | +| **Version** | `v4_x_src_v1` (+ `shadow_score_maturity` band) | +| **Code** | `src/crossposting/shadow_score.py`, `_build_decision_log` | +| **SQL** | removed 2026-10-31 (killed; in git history) | + +Also logs `shadow_rank`, `shadow_pick_meme_id`, `shadow_vs_prod_disagree`. + +### Next check + +- After deploy: decision_log has `shadow_score` on candidates +- **2026-08-25**: shadow-hybrid SQL (now removed) — disagree rate + corr(shadow, f1k_24h) +- Canary soft boost only if live corr/lift clearly > v4 alone +- After H7 merge: re-export + optional taste feature +- Do not schedule as feed experiment diff --git a/experiments/2026-09-05-channel-hits.md b/experiments/2026-09-05-channel-hits.md new file mode 100644 index 00000000..66599b97 --- /dev/null +++ b/experiments/2026-09-05-channel-hits.md @@ -0,0 +1,35 @@ +--- +id: channel_hits_v1 +status: killed +hypothesis: Showing proven channel hits (by 24h forward rate) to confirmed non-subscribers in their feed brings retained invited guests +primary_metric: retained new invitees per host (needed >=6 in treatment) +guardrails: host activity days 7-13, host reactions vs 28d baseline +unit: user +variants: control:50,treatment:50 (fixed host cohort) +started: 2026-09-05 +check_on: 2026-09-19 +decided: 2026-10-30 +result: n 43/42 hosts; retained invitees 0 vs 1, new invitees 1 vs 4; host reactions vs baseline 0.95 vs 0.77; hosts active d7-13 79.1% vs 73.8% +learning: Channel hits in the feed did not create retained invitees and may cut host activity; inconclusive per protocol +code: removed in PR #369 (git history) +readout: docs/analyst/readouts/2026-10-30-experiment-closure.md +--- + +# Experiment: Channel hits v1 + +Added in PR #355 (2026-09-05), return-session fix in PR #356. Published channel memes +were excluded from every private feed, including users who do not subscribe to the +channels. The test showed channel hits (comparable 24h forward rates, conservative +smoothing, source affinity) to confirmed non-subscribers, with membership cached +from channel events and a rate-limited backfill. Fixed-cohort enrollment, flags +default off. Exposure window ended 2026-09-19. + +## Result (2026-10-30 closure) + +ITT, n = 43 control / 42 treatment hosts. Exposure was 531 hit rows (2.06% of the +treatment feed). New invitees 1 vs 4, retained invitees 0 vs 1, unique non-self link +recipients 1 vs 5. Hosts active in days 7-13 79.1% vs 73.8%. Host reactions vs the +28d baseline 0.95 vs 0.77. The study needed at least 6 retained guests in treatment +and got 1. Killed, code deleted in PR #369. + +Readout: [2026-10-30 closure](../docs/analyst/readouts/2026-10-30-experiment-closure.md). diff --git a/experiments/completed/2026-09-06-ocr-length-preference-v1.md b/experiments/2026-09-06-ocr-length-preference-v1.md similarity index 57% rename from experiments/completed/2026-09-06-ocr-length-preference-v1.md rename to experiments/2026-09-06-ocr-length-preference-v1.md index ebf50d4c..76be5bc3 100644 --- a/experiments/completed/2026-09-06-ocr-length-preference-v1.md +++ b/experiments/2026-09-06-ocr-length-preference-v1.md @@ -1,3 +1,20 @@ +--- +id: ocr_length_preference_v1 +status: inconclusive +hypothesis: OCR text length predicts like preference (prospective observational confirmation; protocol not committed) +primary_metric: positive-group effect (pp) +guardrails: late OCR <=5% (validity gate) +unit: user +variants: none (observational) +started: 2026-09-06 +check_on: 2026-10-05 +decided: 2026-10-05 +result: +5.78 pp (95% CI +3.80 to +7.88), support 25/36, but late OCR 23.1% > 5% gate: invalid +learning: OCR coverage lag can invalidate text-feature readouts; check OCR freshness before reading them +code: +readout: +--- + # OCR length preference v1 Concluded October 5, 2026: **quality_failed**. Recorded positive-group effect +5.78 pp (95% CI +3.80 to +7.88), support 25/36, but late OCR 23.14% positive / 22.84% overall exceeds the 5% validity gate. No confirmation, extension or ranking rollout. One-shot automation paused. The protocol and final-readout JSON (`docs/analyst/research/2026-09-06-ocr-length/`) were never committed to this repo; this note is the surviving record. diff --git a/experiments/2026-09-07-broadcast-channel-viral-pick.md b/experiments/2026-09-07-broadcast-channel-viral-pick.md new file mode 100644 index 00000000..733318ce --- /dev/null +++ b/experiments/2026-09-07-broadcast-channel-viral-pick.md @@ -0,0 +1,74 @@ +--- +id: broadcast_channel_viral_pick +status: killed +hypothesis: A meme the channel audience already forwarded is a better single push than the bot-internal HQ pick +primary_metric: react_within_1h % vs HQ +guardrails: LR on reacted rows not -3 pp, viral share of broadcasts >=10% +unit: user +variants: none (viral vs HQ labels, not randomized) +started: 2026-09-07 +check_on: 2026-09-08 +decided: 2026-10-30 +result: pooled react<1h 26.9% vs HQ 36.5% (-9.6 pp); within-user (67) +6.9 pp at 1h but -5.4 pp LR; viral LR 5-20 pp below HQ in every stratum +learning: Channel forwards do not make a better personal push; HQ affinity x LR wins on like rate +code: BROADCAST_CHANNEL_VIRAL_PICK_ENABLED=false (code path kept), src/recommendations/broadcast_pick.py +readout: docs/analyst/broadcast-reengagement.sql +--- + +# H3c — Channel-viral memes for retention broadcasts + +| Field | Value | +|-------|--------| +| **ID** | `broadcast_channel_viral_pick` | +| **Status** | **disabled 2026-10-30** (default `false`; code path kept). Frozen rule trips (1h −9.6 pp vs HQ); arms reach different audiences; within users viral +6.9 pp at 1h but −5.4 pp LR, so HQ is the default (PR #366) | +| **Code** | `src/recommendations/broadcast_pick.py`, kill switch `BROADCAST_CHANNEL_VIRAL_PICK_ENABLED` | +| **Labels** | `broadcast_channel_viral` vs `broadcast_reengagement_hq` vs queue | +| **SQL** | `docs/analyst/broadcast-reengagement.sql` | + +## Hypothesis + +A meme that the channel audience already forwarded is a better single push than +bot-internal HQ (affinity × like rate), because forwards are a community check +that the joke is worth sending onward. + +## Rules + +- Last 7 days of `tgchannelru` / `tgchannelen` posts, at least 24h old, ranked + by latest snapshot forwards. Unseen, matching language, image, `published`. +- Skip known channel members (they already got the channel push). +- Skip `channel_hits_v1` assignees until exposure ends (feed test stays clean). +- Empty pool → existing HQ → queue. Never skip a user because viral is empty. + +**Keep ON** if `broadcast_channel_viral` react_within_1h ≥ HQ (same 14d window) +without a like-rate drop >3 pp on reacted rows. + +**Disable** (`BROADCAST_CHANNEL_VIRAL_PICK_ENABLED=false`) if viral share of +broadcasts is <10% (always empty) or 1h reactivation is worse than HQ by >5 pp +with ≥200 viral sends. + +## Decision 2026-10-30: disable, HQ is default + +Sep 10 → Oct 28 stratified readout: viral like rate 5–20 pp below HQ in every +stratum, react-within-1h/24h equal or worse, session length and 7d activity +equal (within-user too). Disable rule met → `BROADCAST_CHANNEL_VIRAL_PICK_ENABLED` +defaults to `false`. Follow-up: `2026-10-30-broadcast-cadence-holdout.md`. + +## One-shot idle blast (2026-09-07) + +`scripts/broadcast_dormant.py` (removed 2026-10-30; in git history) id `dormant-channel-viral-2026-09-07`. +**Let it finish.** Next extra wave: 7–28d + holdout, not 90d+ never-reacted. +Readout: `docs/analyst/readouts/2026-09-07-onboarding-and-dormant-blast.md`. + +## Cadence doctrine (founder 2026-09-07) + +Keep **15m / 24h / 48h / 1w / 2w / 4w** while the user is still fresh. +After that, **one monthly** reminder for people who already reacted. +Lever inside the cadence is meme quality (H3c), not fewer early nudges. + +2026-10-30: cadence targeting now requires a real reaction near the step +window; per-step no-push holdouts (`broadcast_cadence_holdout_v1_*`) and the +monthly reminder (`monthly_reminder_v1`, flag-gated) are live — see +`2026-10-30-broadcast-cadence-holdout.md`. + +Rejected 2026-09-07: killing the 15m-4w reengagement cadence (habit while fresh; monthly after). +Follow-ups: [broadcast_cadence_holdout_v1](2026-10-30-broadcast-cadence-holdout.md), [monthly_reminder_v1](2026-10-30-monthly-reminder.md). diff --git a/experiments/2026-09-07-cold-start-fresh-viral.md b/experiments/2026-09-07-cold-start-fresh-viral.md new file mode 100644 index 00000000..c84d954e --- /dev/null +++ b/experiments/2026-09-07-cold-start-fresh-viral.md @@ -0,0 +1,46 @@ +--- +id: cold_start_fresh_viral_v1 +status: killed +hypothesis: Same cold-start floors but only memes from the last 30 days, newest first, lift first tap / first like +primary_metric: first tap +guardrails: reached-5, block within 1h, empty queue for new users +unit: user +variants: control:50,treatment:50 +started: 2026-09-07 +check_on: 2026-09-14 +decided: 2026-10-30 +result: first tap 60.0% control vs 48.5% treatment, blocked <1h 19 vs 27 (n 95/99) +learning: For a first impression, all-time proven hits beat fresh memes +code: COLD_START_FRESH_VIRAL_EXPERIMENT_ENABLED=false, src/recommendations/cold_start_experiments.py +readout: docs/analyst/onboarding-funnel.sql +--- + +# H10 — Fresh channel-quality memes as CS1 (A/B) + +| Field | Value | +|-------|--------| +| **ID** | `cold_start_fresh_viral_v1` | +| **Status** | **KILLED 2026-10-30**: first tap 60.0% C vs 48.5% T, blocked<1h 19 vs 27 (n 95/99). Flag default off | +| **Kill switch** | `COLD_START_FRESH_VIRAL_EXPERIMENT_ENABLED` | +| **Code** | `src/recommendations/cold_start_experiments.py`, `cold_start_explore(max_age_days=)` | +| **Labels** | `cold_start_explore_fresh` / `_guarded` vs classic `cold_start_explore` | + +## Hypothesis + +Same CS floors, but only memes **created in the last 30 days**, newest first, +lifts first-tap / first-like vs all-time `lr_smoothed` hits. Empty fresh pool +fills from classic CS so we never starve the first session. + +50/50 `sha256(experiment_id:user_id)%2` on true-new CS1 (`nmemes_sent < 6`, +`nsessions <= 1`). Observational 90d did **not** show an age→like gap (~22% +both); this A/B is the actual test. + +## Decision rules (day-7, ≥80 new users/arm if we get them) + +**Keep treatment / ship default** if first-tap ≥ control +5 pp (or first-like ++3 pp), reached-5 not down, invites/1k not down. +**Disable flag** if first-tap down, block-in-1h up, or empty-queue for new users. + +## Next check + +**2026-09-14** — assignment counts + first-tap by variant. diff --git a/experiments/2026-09-07-onboarding-first-tap.md b/experiments/2026-09-07-onboarding-first-tap.md new file mode 100644 index 00000000..2fe57d03 --- /dev/null +++ b/experiments/2026-09-07-onboarding-first-tap.md @@ -0,0 +1,42 @@ +--- +id: onboarding_first_tap_v1 +status: shipped +hypothesis: New-user growth is gated by first meme to first tap, not by the language picker or share skipping welcome +primary_metric: first tap % of got-meme, first like %, reached-5 +guardrails: block within 1h +unit: user +variants: none (funnel scoreboard) +started: 2026-09-07 +check_on: 2026-09-14 +decided: 2026-09-07 +result: 90d to 2026-09-07: 402 got a meme, 252 reacted, 150 silent; empty start 61% vs share 71%; 2026-10-30: never-got-meme rose 3.1% to 12.6% +learning: First tap is the onboarding leak; the 2026-10-30 rise came from the language picker (fixed by onboarding_default_language_v1) +code: +readout: docs/analyst/onboarding-funnel.sql +--- + +# H9 — First-tap is the onboarding leak + +| Field | Value | +|-------|--------| +| **ID** | `onboarding_first_tap_v1` | +| **Status** | **scoreboard** for H10/H11. **2026-10-30 flag:** users who never got a meme rose 3.1% → 12.6% (25/26 have no language row). Investigate the language picker | +| **Readout** | `docs/analyst/readouts/2026-09-07-onboarding-and-dormant-blast.md` | +| **SQL** | `docs/analyst/onboarding-funnel.sql` | + +## Hypothesis + +New-user growth is gated by **first meme → first tap**, not language picker +and not share skipping welcome. 90d (2026-09-07): 402 got a meme, 252 reacted, +**150** silent. Empty start 61% vs share 71%. + +## Growth bar + +First-tap % of got-meme; first-like %; reached-5. Secondary: non-self share +clicks and invites from the same cohort. Guardrail: block-in-1h. + +## Next check + +**2026-09-14** re-run `onboarding-funnel.sql`. + +Rejected: 'fixing' share links that skip welcome/picker (90d share converts better than empty start). diff --git a/experiments/2026-09-07-onboarding-skip-countdown.md b/experiments/2026-09-07-onboarding-skip-countdown.md new file mode 100644 index 00000000..59206af3 --- /dev/null +++ b/experiments/2026-09-07-onboarding-skip-countdown.md @@ -0,0 +1,37 @@ +--- +id: onboarding_skip_countdown_v1 +status: shipped +hypothesis: Removing the ~9 s 3-2-1 countdown and sending the first meme right after welcome raises first tap +primary_metric: first tap (empty start), seconds to first meme +guardrails: +unit: user +variants: none (pre/post) +started: 2026-09-07 +check_on: 2026-09-14 +decided: 2026-10-30 +result: p50 to first meme 11.6 s to 2.8 s; first tap flat (H10 control 60.0% vs ~61.5% pre) +learning: A faster first meme alone does not raise first tap; kept as UX debt removal +code: src/tgbot/handlers/onboarding.py +readout: docs/analyst/onboarding-funnel.sql +--- + +# H11 — Drop empty-start 3-2-1 countdown + +| Field | Value | +|-------|--------| +| **ID** | `onboarding_skip_countdown_v1` | +| **Status** | **KEEP (neutral) 2026-10-30**: p50 to first meme 11.6 → 2.8 s, first-tap flat | +| **Code** | `src/tgbot/handlers/onboarding.py`, share tap in `reaction.py` | + +## Hypothesis + +Removing the ~9s 3-2-1 (p50 11.6s to first meme) and sending the first meme +right after welcome raises first-tap. Share path already did this at 0.2s and +converted better; after a share tap we now continue the feed (no second welcome). + +Not an A/B — UX debt. Score with H9. Rollback = restore the countdown block. + +## Next check + +**2026-09-14** empty-start p50 seconds to first meme should drop toward ~1s; +first-tap of empty start vs 61% baseline. diff --git a/experiments/active/2026-10-30-broadcast-cadence-holdout.md b/experiments/2026-10-30-broadcast-cadence-holdout.md similarity index 79% rename from experiments/active/2026-10-30-broadcast-cadence-holdout.md rename to experiments/2026-10-30-broadcast-cadence-holdout.md index 2916a6b2..813d781a 100644 --- a/experiments/active/2026-10-30-broadcast-cadence-holdout.md +++ b/experiments/2026-10-30-broadcast-cadence-holdout.md @@ -1,3 +1,20 @@ +--- +id: broadcast_cadence_holdout_v1 +status: running +hypothesis: The 15m push mostly re-times sessions heavy users would start anyway; later steps (24h, 1w, 2w, 4w) bring back lapsing users +primary_metric: reactions/user/week (15m, 24h); active d7-14 (1w, 2w, 4w) +guardrails: blocks <1%, weekly-reactor % not lower in holdout, >=150 users/arm +unit: user +variants: push:85,holdout:15 (15m, 24h); push:80,holdout:20 (1w, 2w, 4w) +started: 2026-10-30 +check_on: 2026-11-20 +decided: +result: unknown +learning: +code: BROADCAST_CADENCE_HOLDOUT_ENABLED, broadcast_cadence_holdout_v1_ +readout: docs/analyst/broadcast-holdout-v1.sql +--- + # Experiment: Broadcast cadence holdout + monthly reminder Created: 2026-10-30 @@ -6,8 +23,8 @@ Owner: engineer / analyst Deployed: pending (holdout starts on deploy; monthly reminder waits for `MONTHLY_REMINDER_ENABLED=true`) **Check 1: 2026-11-20** (15m / 24h) **Check 2: 2026-12-11** (1w / 2w / 4w, monthly reminder) -SQL: [`docs/analyst/broadcast-holdout-v1.sql`](../../docs/analyst/broadcast-holdout-v1.sql) -Registry: [`docs/growth/HYPOTHESES.md`](../../docs/growth/HYPOTHESES.md) H3c (cadence doctrine) +SQL: [`docs/analyst/broadcast-holdout-v1.sql`](../docs/analyst/broadcast-holdout-v1.sql) +Background: [broadcast_channel_viral_pick](2026-09-07-broadcast-channel-viral-pick.md) (H3c, cadence doctrine). Monthly reminder has its own record: [monthly_reminder_v1](2026-10-30-monthly-reminder.md) ## Why diff --git a/experiments/active/2026-10-30-crosspost-v5-shadow.md b/experiments/2026-10-30-crosspost-v5-shadow.md similarity index 81% rename from experiments/active/2026-10-30-crosspost-v5-shadow.md rename to experiments/2026-10-30-crosspost-v5-shadow.md index 4d3b6264..1390b2da 100644 --- a/experiments/active/2026-10-30-crosspost-v5-shadow.md +++ b/experiments/2026-10-30-crosspost-v5-shadow.md @@ -1,10 +1,27 @@ +--- +id: crosspost_v5_shadow +status: running +hypothesis: v5 score (source Beta hit rate x user taste, x like volume on RU) tracks channel virality better than v4 +primary_metric: corr(score_v5, zf) - corr(v4, zf) >= 0.05 on posted picks, per channel +guardrails: n_posted >= 60 per channel +unit: channel_post +variants: shadow (logged only, v5_src_beta_taste_v1) +started: 2026-10-30 +check_on: 2026-11-14 +decided: +result: unknown +learning: +code: src/crossposting/shadow_score.py +readout: docs/analyst/crossposting-v5-shadow.sql +--- + # Experiment: Crosspost v5 shadow score (RU + EN) Created: 2026-10-30 Status: **active (shadow only, production pick unchanged)** Channels: `tgchannelru`, `tgchannelen` Check date: **2026-11-14** -Readout: [`docs/analyst/crossposting-v5-shadow.sql`](../../docs/analyst/crossposting-v5-shadow.sql) +Readout: [`docs/analyst/crossposting-v5-shadow.sql`](../docs/analyst/crossposting-v5-shadow.sql) ## Evidence (DS study 2026-10-30) diff --git a/experiments/active/2026-10-30-explore-budget.md b/experiments/2026-10-30-explore-budget.md similarity index 90% rename from experiments/active/2026-10-30-explore-budget.md rename to experiments/2026-10-30-explore-budget.md index 0f1a6b00..89c9fe76 100644 --- a/experiments/active/2026-10-30-explore-budget.md +++ b/experiments/2026-10-30-explore-budget.md @@ -1,3 +1,20 @@ +--- +id: explore_budget_v1 +status: running +hypothesis: Spend exploration only on labeling fresh memes, show fresh winners to everyone, and fill the idle-return slot from like_spread +primary_metric: likes per active user per week, memes per session +guardrails: fresh_ru_labeled_share >=0.90, empty-queue errors, pos2_abandon <= control +unit: user +variants: control:50,treatment:50 +started: 2026-10-30 +check_on: 2026-11-20 +decided: +result: unknown +learning: +code: EXPLORE_BUDGET_V1_ENABLED, src/recommendations/pipeline.py, candidates.py, blender_experiments.py, src/feed_turn/planner.py +readout: docs/analyst/explore-budget-v1.sql +--- + # Experiment: Explore budget v1 (labeling budget + fresh-viral + idle-return slot) Created: 2026-10-30 diff --git a/experiments/2026-10-30-friend-challenge.md b/experiments/2026-10-30-friend-challenge.md new file mode 100644 index 00000000..9c591c72 --- /dev/null +++ b/experiments/2026-10-30-friend-challenge.md @@ -0,0 +1,35 @@ +--- +id: friend_challenge_h12 +status: killed +hypothesis: A 'five memes for a friend' game turns regular users' friends into retained feed readers +primary_metric: retained new invitees per 100 assigned senders (>=3 feed memes reacted on days 7-13) +guardrails: recipient block rate <30% +unit: user +variants: treatment:90,holdout:10 +started: 2026-10-30 +check_on: +decided: 2026-10-31 +result: 0 participations ever +learning: Nobody played, so the friend-invite idea is untested, not disproven +code: FRIEND_CHALLENGE_ENABLED (default false), removed in a follow-up PR +readout: deleted in PR #390 (see git history) +--- + +# Experiment: Friend challenge "Пять мемов для друга" (H12) + +Merged 2026-10-30 in PR #364 behind `FRIEND_CHALLENGE_ENABLED` (default false). +Design: [docs/growth/2026-09-05-friend-meme-challenge.md](../docs/growth/2026-09-05-friend-meme-challenge.md) +(the 2026-10-30 header block overrides the September protocol). + +- After an ordinary feed like, an eligible user (active on >=8 of the last 28 days, + >=20 likes in that window, not blocked; about 234 users) was assigned once in + `experiment_assignment` (`friend_challenge_h12`): 10% holdout, 90% treatment. +- Treatment users got one offer (also `/friend`): pick five memes, share a + `start=fc_` link. The friend rates 5 memes, sees "N из 5", continues into + their own feed; a new friend gets `user.inviter_id` = sender. +- Ambitious target: double weekly activated new users (baseline ~10/week) within 6 + weeks. Kill: <5 retained invitees per 100 senders at d28 or recipient block rate >30%. + +## Log +- 2026-10-30: merged behind the flag. +- 2026-10-31: killed (0 participations ever); code removal is in a separate PR. diff --git a/experiments/2026-10-30-monthly-reminder.md b/experiments/2026-10-30-monthly-reminder.md new file mode 100644 index 00000000..cdc02bbc --- /dev/null +++ b/experiments/2026-10-30-monthly-reminder.md @@ -0,0 +1,43 @@ +--- +id: monthly_reminder_v1 +status: running +hypothesis: One monthly HQ meme raises 30d activity of idle users who reacted before by >=3 pp at <2% blocks +primary_metric: react_30d_pct treated - holdout +guardrails: blocked_30d_pct treated <2% +unit: user +variants: treatment:50,holdout:50 +started: 2026-10-31 +check_on: 2026-12-11 +decided: +result: unknown +learning: +code: MONTHLY_REMINDER_ENABLED (true in prod, verified 2026-10-31), MONTHLY_REMINDER_HOLDOUT_PCT, broadcast_monthly_reminder flow +readout: docs/analyst/broadcast-holdout-v1.sql +--- + +# Experiment: Monthly reminder v1 + +Code shipped 2026-10-30 (PR #366) with +[broadcast_cadence_holdout_v1](2026-10-30-broadcast-cadence-holdout.md); the full +design and decision rules are in that file. The flow runs on the 1st of the month +at 13:00 MSK only once `MONTHLY_REMINDER_ENABLED=true` (default false, the owner flips +it after deploy). When the flag goes on: set `status: running` and `started:`. + +- Audience: `user_stats.last_reaction_at` 29-365 days ago, not blocked (1,859 users on + 2026-10-30, so about 930 pushed). One HQ meme via `send_meme_broadcast`, + `broadcast_id = monthly-reminder-YYYY-MM`. +- Holdout 50% (`MONTHLY_REMINDER_HOLDOUT_PCT`), kept on until the readout so the 30d + window stays clean. +- Prior evidence: the 2026-09-07 dormant blast brought about +110 returns in 24h, + about 95 still active a month later, about 20 blocks. + +## Decision rule (frozen 2026-10-30) + +Keep if `react_30d_pct` treated - holdout >= +3 pp and `blocked_30d_pct` treated < 2%, +then set `MONTHLY_REMINDER_HOLDOUT_PCT=0`. Otherwise set `MONTHLY_REMINDER_ENABLED=false`. +Fewer than 150 users per arm: extend 2 weeks. + +## Log +- 2026-10-30: code merged, flag off. + +- 2026-10-31: verified MONTHLY_REMINDER_ENABLED=true in the prod Prefect runner; status set to running (started = date verified, first send follows the flow schedule). diff --git a/experiments/active/2026-10-31-crosspost-content-tags.md b/experiments/2026-10-31-crosspost-content-tags.md similarity index 78% rename from experiments/active/2026-10-31-crosspost-content-tags.md rename to experiments/2026-10-31-crosspost-content-tags.md index 1b697bfb..a0922afa 100644 --- a/experiments/active/2026-10-31-crosspost-content-tags.md +++ b/experiments/2026-10-31-crosspost-content-tags.md @@ -1,12 +1,29 @@ +--- +id: crosspost_content_tags_v5c +status: running +hypothesis: Relatable jokes and tweet screenshots get ~8-10% fewer RU forwards at equal source and likes; down-weighting them raises forwards per post +primary_metric: Gate A kappa >=0.6; Gate B refit coefficient <= -0.05 with CI excluding 0 +guardrails: tagged_pct, corr_v5c - corr_v5 +unit: channel_post +variants: shadow (logged only, score_v5c) +started: 2026-10-31 +check_on: 2026-11-20 +decided: +result: unknown +learning: +code: Describe Memes content tags, src/crossposting/shadow_score.py +readout: docs/analyst/crossposting-content-tags-gates.py, docs/analyst/crossposting-content-tags-shadow.sql +--- + # Experiment: Crosspost content tags (relatable / tweet screenshot) - **Created:** 2026-10-31 - **Status:** active. Tags are collected and a shadow score is logged; the production pick is unchanged. - **Channels:** `tgchannelru` decides. `tgchannelen` is logged for reference only. - **Check date:** **2026-11-20** -- **Research:** [`docs/analyst/research/2026-10-30-forward-drivers/`](../../docs/analyst/research/2026-10-30-forward-drivers/README.md) -- **Gates:** [`docs/analyst/crossposting-content-tags-gates.py`](../../docs/analyst/crossposting-content-tags-gates.py) -- **Shadow readout:** [`docs/analyst/crossposting-content-tags-shadow.sql`](../../docs/analyst/crossposting-content-tags-shadow.sql) +- **Research:** [`docs/analyst/research/2026-10-30-forward-drivers/`](../docs/analyst/research/2026-10-30-forward-drivers/README.md) +- **Gates:** [`docs/analyst/crossposting-content-tags-gates.py`](../docs/analyst/crossposting-content-tags-gates.py) +- **Shadow readout:** [`docs/analyst/crossposting-content-tags-shadow.sql`](../docs/analyst/crossposting-content-tags-shadow.sql) ## Evidence diff --git a/experiments/2026-10-31-ocr-dedup-watermark-fix.md b/experiments/2026-10-31-ocr-dedup-watermark-fix.md new file mode 100644 index 00000000..adecf69f --- /dev/null +++ b/experiments/2026-10-31-ocr-dedup-watermark-fix.md @@ -0,0 +1,40 @@ +--- +id: ocr_dedup_watermark_fix +status: shipped +hypothesis: OCR duplicate merges that share only the watermark are false; stripping it and requiring similarity >=0.6 removes them +primary_metric: false-merge rate on hand-labelled OCR merges +guardrails: recall of true merges, lookup latency +unit: meme +variants: none (fix) +started: 2026-10-31 +check_on: +decided: 2026-10-31 +result: false merges 43% to 2.5% at 93.5% recall (152 labelled merges, 60 days); 7,762 false-merged memes restored to ok +learning: Watermark text made unrelated short captions look identical, and hub originals absorbed hundreds of dupes; strip boilerplate before text similarity +code: find_duplicate_by_ocr_text, OCR_WATERMARK_PATTERN (PRs #380, #382) +readout: docs/analyst/data/2026-10-31-dedup-restored-meme-ids.txt +--- + +# Fix: OCR dedup watermark (not an experiment, kept for the learning) + +PR #380 (2026-10-31): OCR duplicate matching strips the `@ffmemesbot` watermark (and +misreads) and `imgflip.com`, requires `similarity >= 0.6` (the `%` operator stays only +as the index prefilter), and picks the most similar candidate first. + +Why: of 152 hand-labelled OCR merges from the last 60 days, 43% were false, mostly +unrelated short captions that shared only the watermark. Published-first ordering +funnelled them into hub originals: one meme whose whole text is the watermark +absorbed 655 dupes, another 760. In the last 24h before the fix, 259 of 565 merges +were below the new rule. + +| rule | false merges | recall of true merges | +|---|---|---| +| raw text, similarity 0.3 | 43% | 100% (by definition) | +| stripped, similarity >= 0.6 | 2.5% | 93.5% | + +PR #382: narrowed the watermark regex (the old `\S*` removed caption words glued to +the watermark) and reused it in the `has_text` check. + +Restore (2026-10-31 14:40 UTC): 7,762 false-merged memes went back to `ok` +(ids: [`docs/analyst/data/2026-10-31-dedup-restored-meme-ids.txt`](../docs/analyst/data/2026-10-31-dedup-restored-meme-ids.txt)). +This is a confound for [explore_budget_v1](2026-10-30-explore-budget.md). diff --git a/experiments/2026-10-31-onboarding-default-language.md b/experiments/2026-10-31-onboarding-default-language.md new file mode 100644 index 00000000..a35b2baf --- /dev/null +++ b/experiments/2026-10-31-onboarding-default-language.md @@ -0,0 +1,54 @@ +--- +id: onboarding_default_language_v1 +status: running +hypothesis: Defaulting unknown/bare-en users to en (instead of a blocking picker) gets every signup a first meme and more first reactions +primary_metric: share of signups with >=1 meme sent, share with a first reaction +guardrails: no-language signups = 0 +unit: user +variants: none (pre/post bug fix) +started: 2026-10-31 +check_on: 2026-11-14 +decided: +result: unknown +learning: +code: src/tgbot/handlers/start.py, language.py (PR #385) +readout: experiments/2026-10-31-onboarding-default-language.md (inline SQL) +--- + +# H13 — Default language instead of blocking picker + +| Field | Value | +|-------|--------| +| **ID** | `onboarding_default_language_v1` | +| **Status** | **live** (not an A/B; bug fix) | +| **Code** | `src/tgbot/handlers/start.py` (language seeded above the deep-link ladder), `language.py` picker callback | + +## Hypothesis + +Bare-`en` / unknown Telegram users got only the language picker on /start, no +`user_language` row, and so no meme (feed joins on `user_language`). 6 weeks to +2026-10-31: 21/150 signups (14%) had no language row, 0 of them got a meme, 0 +clicked the picker. Now they default to `en`, get welcome + first meme +immediately, and the picker follows the meme (`/lang` still works). + +## Growth bar + +Share of signups with ≥1 meme sent → ~100% (minus blocked); share with a first +reaction should rise. Baseline 6w: got meme 128/150 (85%), reacted 79/150 +(53%); last 2 weeks 42/56 (75%) and 27/56 (48%). Guardrail: no-language +signups = 0. + +```sql +SELECT date_trunc('week', u.created_at)::date wk, count(*) signups, + count(*) FILTER (WHERE NOT EXISTS (SELECT 1 FROM user_language l WHERE l.user_id=u.id)) no_lang, + count(*) FILTER (WHERE EXISTS (SELECT 1 FROM user_meme_reaction r WHERE r.user_id=u.id)) got_meme, + count(*) FILTER (WHERE EXISTS (SELECT 1 FROM user_meme_reaction r + WHERE r.user_id=u.id AND r.reacted_at IS NOT NULL)) reacted +FROM "user" u WHERE u.created_at >= now() - interval '42 days' GROUP BY 1 ORDER BY 1; +``` + +## Next check + +**2026-11-14**. + +Not doing: a multi-select language picker for new users (it is already one tap; `/lang` is settings). diff --git a/experiments/2026-10-31-phash-shadow-dedup.md b/experiments/2026-10-31-phash-shadow-dedup.md new file mode 100644 index 00000000..2071ab28 --- /dev/null +++ b/experiments/2026-10-31-phash-shadow-dedup.md @@ -0,0 +1,32 @@ +--- +id: phash_shadow_v1 +status: running +hypothesis: An exact 64-bit dHash match finds duplicates OCR cannot (videos, animations, text-free images) with high precision +primary_metric: precision on sampled exact-hash pairs, agreement with duplicate_of +guardrails: no merges in shadow; degenerate (blank-frame) hash groups +unit: meme +variants: shadow (logged only) +started: 2026-10-31 +check_on: 2026-11-14 +decided: +result: unknown +learning: +code: src/storage/deduplication/image_hash.py, meme.image_hash, scripts/admin/backfill_image_hash.py +readout: docs/analyst/phash-shadow.sql +--- + +# Experiment: Perceptual hash dedup (shadow) + +Shipped 2026-10-31 in PR #381. Every new upload stores a 64-bit dHash in +`meme.image_hash` (video/animation: Telegram thumbnail). After a non-duplicate upload, +a live meme with the same hash is logged as `phash shadow match`; nothing is merged. +About 40% of sends go to memes OCR cannot dedup, which is the gap this targets. + +Known limits: backfill skips videos/animations; upload and backfill hash different +bytes (exact match may miss a few pairs); hashes 0/-1 are not matched. + +No check date or ship rule was set. Next step: run `phash-shadow.sql` after the +backfill, label sample pairs for precision, then decide on merges (Hamming distance is +the upgrade path). + +- 2026-10-31: check_on set to 2026-11-14: run scripts/admin/backfill_image_hash.py for 30 days of images, then docs/analyst/phash-shadow.sql; turn merging on only if a 50-pair sample shows >=95% true duplicates. diff --git a/experiments/2026-10-31-raw-impr-rank-reaction-rate.md b/experiments/2026-10-31-raw-impr-rank-reaction-rate.md new file mode 100644 index 00000000..2cc88b0c --- /dev/null +++ b/experiments/2026-10-31-raw-impr-rank-reaction-rate.md @@ -0,0 +1,35 @@ +--- +id: raw_impr_rank_reaction_rate +status: running +hypothesis: Within-source reaction rate (reactions/views) beats views for ranking cold memes +primary_metric: like rate of memes with raw_impr_rank = 0, 3 weeks before vs after deploy +guardrails: LR of engines that use the rank (candidates.py rank-0 pool, crosspost impr penalty) +unit: meme +variants: none (pre/post) +started: 2026-10-31 +check_on: 2026-11-21 +decided: +result: unknown +learning: +code: meme_raw_telegram.reactions, calculate_meme_raw_impressions_stats (PR #384) +readout: +--- + +# Experiment: raw_impr_rank by reaction rate + +Shipped 2026-10-31 in PR #384. `raw_impr_rank` is now the quartile of +`reactions / views` within source x meme type (VK rows and rows without reactions +rank by views after rated rows). Also fixed an off-by-one: the top meme of a small +source never got rank 0. Existing rows keep `reactions = NULL` until re-parsed, so the +switch is gradual. + +## Evidence (2026-10-31, 3,622 memes / 97 sources) + +- Views quartile vs our like rate: Spearman +0.02. +- Within-source reaction-rate percentile: Spearman +0.22 [0.18, 0.27]; top vs bottom + quintile like rate 0.44 vs 0.35; holds on cold memes' first 30 reactions. + +## Decision rule + +Kill (revert to views) if the like rate of rank-0 memes does not rise over 3 weeks +after deploy vs 3 weeks before. No readout SQL committed yet. diff --git a/experiments/active/2026-11-01-crosspost-src-forward-rate.md b/experiments/2026-11-01-crosspost-src-forward-rate.md similarity index 81% rename from experiments/active/2026-11-01-crosspost-src-forward-rate.md rename to experiments/2026-11-01-crosspost-src-forward-rate.md index 95b05ae0..cb8d9b1a 100644 --- a/experiments/active/2026-11-01-crosspost-src-forward-rate.md +++ b/experiments/2026-11-01-crosspost-src-forward-rate.md @@ -1,10 +1,27 @@ +--- +id: crosspost_src_fwd_v5f +status: running +hypothesis: Memes whose source post was forwarded more than usual for that source get more forwards in our channels +primary_metric: RU Spearman gain rho_v5f - rho_v5 with 95% CI excluding 0 +guardrails: coverage_pct >=50% +unit: channel_post +variants: shadow (logged only, score_v5f) +started: 2026-10-31 +check_on: 2026-11-28 +decided: +result: unknown +learning: +code: meme_raw_telegram.forwards, Collect Channel Stats flow, src/crossposting/shadow_score.py +readout: docs/analyst/crossposting-src-fwd-shadow.sql +--- + # Experiment: Crosspost source forward rate (v5f shadow) - **Created:** 2026-11-01 - **Status:** active, shadow only. Source-post forwards are collected and a shadow score is logged; the production pick is unchanged. - **Channels:** `tgchannelru` decides. `tgchannelen` is logged for reference. - **Check date:** **2026-11-28** -- **Shadow readout:** [`docs/analyst/crossposting-src-fwd-shadow.sql`](../../docs/analyst/crossposting-src-fwd-shadow.sql) +- **Shadow readout:** [`docs/analyst/crossposting-src-fwd-shadow.sql`](../docs/analyst/crossposting-src-fwd-shadow.sql) ## Evidence (2026-10-31, offline) diff --git a/experiments/HYPOTHESES.md b/experiments/HYPOTHESES.md deleted file mode 100644 index 8d79863a..00000000 --- a/experiments/HYPOTHESES.md +++ /dev/null @@ -1,6 +0,0 @@ -# Hypotheses registry (moved) - -The canonical registry is **[`docs/growth/HYPOTHESES.md`](../docs/growth/HYPOTHESES.md)**. - -This file was a stale copy (last updated 2026-08-11, without H9–H11). It was replaced -with this pointer on 2026-10-30 so the two would stop drifting apart. The old content is in git history. diff --git a/experiments/README.md b/experiments/README.md index 8a2bd017..e9f7c798 100644 --- a/experiments/README.md +++ b/experiments/README.md @@ -1,117 +1,103 @@ # Experiments -Structured experiment tracking for product development. - -## Start here on resume - -**[`docs/growth/HYPOTHESES.md`](../docs/growth/HYPOTHESES.md)** — living list of open hypotheses, ship/kill -rules, and **hard recheck dates** (day-3 / day-7 / day-14). Read it first when -picking up the project after a break. - -## Directory Structure - -``` -experiments/ -├── HYPOTHESES.md # Pointer to docs/growth/HYPOTHESES.md (canonical) -├── active/ # Currently running experiments -│ └── YYYY-MM-DD-experiment-name.md -├── completed/ # Finished experiments (never deleted) -│ └── YYYY-MM-DD-experiment-name.md -├── reports/ # Local analyst dumps (gitignored — do not commit) -│ └── YYYY-MM-DD-daily.md -└── log.jsonl # Local audit trail (gitignored — do not commit) -``` - -## Experiment Lifecycle - -1. **Create**: The experiment owner creates `active/YYYY-MM-DD-name.md` -2. **Monitor**: The owner or analyst tracks metrics in daily reports -3. **Complete**: The owner moves the file from `active/` to `completed/` with results -4. **Never delete**: Completed experiments are historical artifacts - -## Experiment File Format +One file per experiment: `experiments/YYYY-MM-DD-slug.md` (date = when it was +designed or started). Status lives in the front matter, so files never move. +This README is generated: `python scripts/experiments.py index`. +**Read it before designing a new experiment:** the Decided table is everything +we tried and what we now believe. + +Growth bar for new experiments: ship only if the funnel gets less leaky +(first tap / first like / reached-5) and/or the k-factor moves (unique non-self +share clicks, new-user invites). Session length and block rate are guardrails. +North star: [`SPEC.md`](../SPEC.md). Growth thesis: +[`docs/growth/virality-loop.md`](../docs/growth/virality-loop.md). + +## Add, check, close + +1. **Add:** copy the template into `experiments/YYYY-MM-DD-slug.md`, freeze the + decision rule in the body, commit the readout SQL under `docs/analyst/`. + Set `status: running` and `started:` when it deploys. +2. **Check:** `python scripts/experiments.py due` lists running experiments + whose `check_on` has passed; `python scripts/experiments.py readout ` + prints the command to run against `$ANALYST_DATABASE_URL` (read-only). + Append dated numbers to the file's log. +3. **Close:** set `status` (shipped / killed / inconclusive), `decided`, + `result` (numbers + CI or n) and `learning` (one line: what we now believe). + Never delete the file. Readouts stay in `docs/analyst/readouts/`; link them. +4. Run `python scripts/experiments.py index` and commit the README with it. + +Front matter is flat `key: value`; lists are comma-separated; unknown is `unknown`. +Status: proposed | running | shipped | killed | inconclusive. ```markdown -# Experiment: [Name] -Created: YYYY-MM-DD -Status: active | completed | cancelled -Owner: ceo | engineer | analyst -Deployed: YYYY-MM-DD | pending -Measure after: YYYY-MM-DD - -## Hypothesis -What we expect to happen and why. - -## Changes Made -What was changed (commits, config, etc.) - -## Metrics Before -Key metrics at experiment start. - -## Metrics After -Key metrics at experiment end (filled by CEO on completion). - -## Conclusion -What we learned (filled on completion). -``` - -For active experiments, `Measure after:` must be a parseable `YYYY-MM-DD` once -the deployment date is known. Avoid relative-only dates such as "14 days -post-deploy" after code has shipped; the weekly outcome audit uses this field to -find stale active experiments. - -## Daily Report Format (reports/) - -```markdown -# Daily Report: YYYY-MM-DD - -## Health Check -[Output of HEALTH CHECK query from docs/analyst/metrics.sql] - -## North Star -Session length: median=X, avg=Y - -## Anomalies -[Any metrics that deviate >30% from 7-day average] - -## Active Experiments -[Status update for each experiment in active/] - -## Community Feedback -[Summary of @ffmemes channel activity] - -## Recommendations for CEO -[Prioritized list of suggested actions] -``` - -## JSONL Audit Log (log.jsonl) - -Each line is a JSON object: - -```json -{ - "timestamp": "2026-03-20T19:30:00Z", - "agent": "analyst", - "action": "daily_report", - "status": "success", - "summary": "Generated daily report. No anomalies detected.", - "metrics": { - "session_length_median": 19, - "wau": 530, - "dau": 180, - "reactions_24h": 5200, - "like_rate": 43.4 - }, - "active_experiments": ["goat-fix"], - "error": null -} +--- +id: my_experiment_v1 +status: proposed +hypothesis: one line, falsifiable +primary_metric: likes per active user per week +guardrails: LR not down >3 pp, block rate, empty queue +unit: user +variants: control:50,treatment:50 +started: +check_on: YYYY-MM-DD +decided: +result: unknown +learning: +code: FLAG_NAME, src/path.py +readout: docs/analyst/my-experiment-v1.sql +--- + +# Experiment: my_experiment_v1 + +Design, assignment, frozen decision rules, confounds. + +## Log +- YYYY-MM-DD: started (deploy sha / PR). ``` -### Action Types -- `daily_report` — Analyst produced a metrics report -- `anomaly_detected` — Analyst flagged a metric deviation -- `experiment_created` — CEO started a new experiment -- `experiment_completed` — CEO concluded an experiment -- `task_created` — CEO assigned work to another agent -- `investigation` — Analyst dug deeper into an anomaly -- `post_published` — Comms Manager posted to @ffmemes +## Open (9) + +| check_on | experiment | status | primary metric | hypothesis | +| --- | --- | --- | --- | --- | +| 2026-11-14 | [crosspost_v5_shadow](2026-10-30-crosspost-v5-shadow.md) | running | corr(score_v5, zf) - corr(v4, zf) >= 0.05 on posted picks, per channel | v5 score (source Beta hit rate x user taste, x like volume on RU) tracks channel virality better than v4 | +| 2026-11-14 | [onboarding_default_language_v1](2026-10-31-onboarding-default-language.md) | running | share of signups with >=1 meme sent, share with a first reaction | Defaulting unknown/bare-en users to en (instead of a blocking picker) gets every signup a first meme and more first reactions | +| 2026-11-14 | [phash_shadow_v1](2026-10-31-phash-shadow-dedup.md) | running | precision on sampled exact-hash pairs, agreement with duplicate_of | An exact 64-bit dHash match finds duplicates OCR cannot (videos, animations, text-free images) with high precision | +| 2026-11-20 | [broadcast_cadence_holdout_v1](2026-10-30-broadcast-cadence-holdout.md) | running | reactions/user/week (15m, 24h); active d7-14 (1w, 2w, 4w) | The 15m push mostly re-times sessions heavy users would start anyway; later steps (24h, 1w, 2w, 4w) bring back lapsing users | +| 2026-11-20 | [crosspost_content_tags_v5c](2026-10-31-crosspost-content-tags.md) | running | Gate A kappa >=0.6; Gate B refit coefficient <= -0.05 with CI excluding 0 | Relatable jokes and tweet screenshots get ~8-10% fewer RU forwards at equal source and likes; down-weighting them raises forwards per post | +| 2026-11-20 | [explore_budget_v1](2026-10-30-explore-budget.md) | running | likes per active user per week, memes per session | Spend exploration only on labeling fresh memes, show fresh winners to everyone, and fill the idle-return slot from like_spread | +| 2026-11-21 | [raw_impr_rank_reaction_rate](2026-10-31-raw-impr-rank-reaction-rate.md) | running | like rate of memes with raw_impr_rank = 0, 3 weeks before vs after deploy | Within-source reaction rate (reactions/views) beats views for ranking cold memes | +| 2026-11-28 | [crosspost_src_fwd_v5f](2026-11-01-crosspost-src-forward-rate.md) | running | RU Spearman gain rho_v5f - rho_v5 with 95% CI excluding 0 | Memes whose source post was forwarded more than usual for that source get more forwards in our channels | +| 2026-12-11 | [monthly_reminder_v1](2026-10-30-monthly-reminder.md) | running | react_30d_pct treated - holdout | One monthly HQ meme raises 30d activity of idle users who reacted before by >=3 pp at <2% blocks | + +## Decided (28, newest first) + +| decided | experiment | status | result | learning | +| --- | --- | --- | --- | --- | +| 2026-10-31 | [ocr_dedup_watermark_fix](2026-10-31-ocr-dedup-watermark-fix.md) | shipped | false merges 43% to 2.5% at 93.5% recall (152 labelled merges, 60 days); 7,762 false-merged memes restored to ok | Watermark text made unrelated short captions look identical, and hub originals absorbed hundreds of dupes; strip boilerplate before text similarity | +| 2026-10-31 | [friend_challenge_h12](2026-10-30-friend-challenge.md) | killed | 0 participations ever | Nobody played, so the friend-invite idea is untested, not disproven | +| 2026-10-31 | [early_channel_popup](2026-04-20-early-channel-popup.md) | shipped | unknown (shipped as default without a readout) | none: unmeasured, do not cite as evidence | +| 2026-10-31 | [crossposting_forward_intent_ranker](2026-05-19-crossposting-forward-intent-ranker.md) | killed | unknown (never built); 2 manual source-prior posts: RU 27.5 fwd/1k, EN 0 | Source prior alone found acceptable RU memes but no breakouts; superseded by v4/v5/v5f scoring | +| 2026-10-30 | [viral_shares_blender_v1](2026-08-09-viral-shares-blender-v1.md) | killed | clickers/1k 0.031 control vs 0.025 treatment (n 344/350 users, 426k/395k sends); invites 5 vs 7; user LR -2.2 pp; engine was 0.76% of treatment sends | Ranking by past share clicks did not raise sharing; the engine barely served, so the arms hardly differed | +| 2026-10-30 | [source_affinity_demote_v1](2026-08-09-source-affinity-demote.md) | shipped | weekly LR 32.6-36.1% post vs 30.7-33.5% pre, no send-volume cliff | Soft demote of majority-dislike sources is safe (+1-3 pp LR); hard-block was rejected (cuts ~57% of inventory offline) | +| 2026-10-30 | [onboarding_skip_countdown_v1](2026-09-07-onboarding-skip-countdown.md) | shipped | p50 to first meme 11.6 s to 2.8 s; first tap flat (H10 control 60.0% vs ~61.5% pre) | A faster first meme alone does not raise first tap; kept as UX debt removal | +| 2026-10-30 | [meme_like_count](2026-05-12-meme-like-count.md) | killed | user LR 45.59% vs 46.20%, reaction rate 83.40% vs 83.26%, active days/14d 0.85 vs 0.86 (n 6306/6208) | Like counts on the button do not change engagement; a plain heart is simpler and avoids popularity bias | +| 2026-10-30 | [first_meme_nudge](2026-04-28-first-meme-nudge.md) | killed | first tap 15.8% control vs 16.0% treatment (n 1526/1456); reached-5 +2.1 pp (z~1.8); user LR -3.1 pp; d1-7 return 5.0% vs 5.0% | The nudge did not move first tap; it ran 6 months as an orphan split with no registry entry, so register every split | +| 2026-10-30 | [crosspost_taste_cohort_v1](2026-08-10-crosspost-taste-cohort-shadow.md) | killed | f1k top vs bottom half 28.52 vs 28.72 (n=287); taste>=1 vs taste=0 28.62 vs 29.87 | A hand-picked taste cohort that looked good offline (lift 1.19) carried no live signal | +| 2026-10-30 | [crosspost_meme_level_v1](2026-08-10-crosspost-v4-and-taste-cohort.md) | shipped | hit rate 32.1% to 41.0% (+8.9 pp, z~2.4, n 299/373), f1k 26.2 to 28.9; manual posts also +3.4 f1k (channel drift) | Bot like volume, not like rate, predicts channel forwards; part of the lift is channel drift | +| 2026-10-30 | [cold_start_fresh_viral_v1](2026-09-07-cold-start-fresh-viral.md) | killed | first tap 60.0% control vs 48.5% treatment, blocked <1h 19 vs 27 (n 95/99) | For a first impression, all-time proven hits beat fresh memes | +| 2026-10-30 | [cold_start_first_impression_v2](2026-08-09-cold-start-floors.md) | shipped | first tap 61.5% to 62.0% (n=100 post), underpowered; no rollback trigger | Stricter quality floors did not measurably move first tap; kept as harmless | +| 2026-10-30 | [channel_hits_v1](2026-09-05-channel-hits.md) | killed | n 43/42 hosts; retained invitees 0 vs 1, new invitees 1 vs 4; host reactions vs baseline 0.95 vs 0.77; hosts active d7-13 79.1% vs 73.8% | Channel hits in the feed did not create retained invitees and may cut host activity; inconclusive per protocol | +| 2026-10-30 | [broadcast_reengagement_label](2026-08-09-broadcast-reengagement-label.md) | shipped | broadcast p50 sec-to-react 3.8-5.5k s vs feed 5.7 s | Push reactions are delayed opens; keep broadcast_* rows out of feed dwell and LR analysis | +| 2026-10-30 | [broadcast_reengagement_hq_pick](2026-08-09-broadcast-hq-pick.md) | shipped | 14d: 5,533 sends, LR 57.2%, react<1h 36.5%, queue fallback 0.4% (pre-HQ LR ~54%, n~39) | Affinity x like-rate is a solid default push meme | +| 2026-10-30 | [broadcast_channel_viral_pick](2026-09-07-broadcast-channel-viral-pick.md) | killed | pooled react<1h 26.9% vs HQ 36.5% (-9.6 pp); within-user (67) +6.9 pp at 1h but -5.4 pp LR; viral LR 5-20 pp below HQ in every stratum | Channel forwards do not make a better personal push; HQ affinity x LR wins on like rate | +| 2026-10-30 | [bot2channel_ml_lab_v1](2026-08-11-bot2channel-ml-lab.md) | killed | lifetime walk-forward HOLD (n=3889); live shadow corr -0.092 (n=363 posted), agreed vs disagreed f1k 29.01 vs 29.10 | A one-split offline win (lift 1.19) did not survive lifetime walk-forward or live shadow; require both before trusting a ranker | +| 2026-10-05 | [ocr_length_preference_v1](2026-09-06-ocr-length-preference-v1.md) | inconclusive | +5.78 pp (95% CI +3.80 to +7.88), support 25/36, but late OCR 23.1% > 5% gate: invalid | OCR coverage lag can invalidate text-feature readouts; check OCR freshness before reading them | +| 2026-09-07 | [onboarding_first_tap_v1](2026-09-07-onboarding-first-tap.md) | shipped | 90d to 2026-09-07: 402 got a meme, 252 reacted, 150 silent; empty start 61% vs share 71%; 2026-10-30: never-got-meme rose 3.1% to 12.6% | First tap is the onboarding leak; the 2026-10-30 rise came from the language picker (fixed by onboarding_default_language_v1) | +| 2026-08-09 | [skip_is_next_not_ban](2026-08-09-skip-is-next.md) | shipped | skip p50 4.7 s vs like 7.1 s; instant skip 19% vs like 9% (7d feed) | Treat skip as next, not dislike; no dwell-weight ship without an offline counterfactual | +| 2026-08-09 | [recently_liked_blender_v2](2026-05-09-recently-liked-blender-v2.md) | shipped | median session 18 vs 19 (+5.6%), global LR 63.33% vs 63.67%, median user LR 62.65% vs 71.08% (n ~371/409; 1000/arm gate never reached) | v1's -14 pp LR was an assignment failure (skipper imbalance 2.3x); stratify by user LR; recently_liked deserves more weight | +| 2026-07-29 | [cold_start_candidate_guardrails](2026-07-14-cold-start-candidate-filter.md) | shipped | reached-10 32.4% treatment (n=37) vs 11.1% concurrent control vs 26.8% baseline; reached-5 70.3% vs 44.4%; all guardrails pass | Dropping a few weak sources from early positions lifts new-user depth (directional, n=37); mostly validates guarded phase 1 | +| 2026-04-09 | [cold_start_v2](2026-03-26-cold-start-v2.md) | shipped | first-meme LR 0% to 33.3%, organic 10-meme reach 24.2% to 52.4% (n=84; 44.5% incl. Apr 1 spike), WAU 897 | Cold start should rank proven memes by raw like rate; lr_smoothed's user-bias correction empties the pool for new users | +| 2026-04-08 | [upload_promo_day1](2026-03-29-upload-promotion-day1.md) | killed | uploads 3.2% treatment vs 8.8% control; D1 retention 28.1% vs 28.6%; D7 never measured | A passive 'did you know' message does not create an upload habit and may reduce uploads; needs a CTA after a positive moment | +| 2026-03-26 | [unstarve_like_spread](2026-03-20-unstarve-like-spread.md) | shipped | engine sends 3,112 to 18,233 per 7d (5.9x), engine LR 51.1% to 42.5%, median session 18 to 19, continuation 97.7% | Meme age is not a quality signal; widening a high-LR engine trades some LR for volume and longer sessions | +| 2026-03-26 | [cold_start_3phase](2026-03-20-cold-start-3phase.md) | killed | first-meme LR 0% (28 new users), 10-meme reach 24.2%, cold_start_explore LR 18.8%; session 19 and WAU flat | New users with no taste signal need the globally best memes first, not the most varied ones | +| 2026-03-20 | [goat_engine_investigation](2026-03-20-goat-engine-investigation.md) | shipped | goat continuation 98.0% (best engine) while LR fell 44.6% to 16%, halving about every 3 days | A fixed top-N engine exhausts its pool per user; continuation can stay high while LR decays, so add a per-user recency filter | diff --git a/experiments/active/.gitkeep b/experiments/active/.gitkeep deleted file mode 100644 index e69de29b..00000000 diff --git a/experiments/completed/.gitkeep b/experiments/completed/.gitkeep deleted file mode 100644 index e69de29b..00000000 diff --git a/experiments/completed/2026-08-09-viral-shares-blender-v1.md b/experiments/completed/2026-08-09-viral-shares-blender-v1.md deleted file mode 100644 index 3296df70..00000000 --- a/experiments/completed/2026-08-09-viral-shares-blender-v1.md +++ /dev/null @@ -1,88 +0,0 @@ -# Experiment: Viral shares blender v1 - -Created: 2026-08-09 -Status: **completed — KILL (2026-10-30)** -Owner: engineer / analyst -Deployed: 2026-08-09 (~14:20 UTC first assignment) -**Measure after: 2026-08-16** (primary) -**Final if underpowered: 2026-08-23** -Smoke: 2026-08-12 - -Living registry: [`../HYPOTHESES.md`](../HYPOTHESES.md) **H1**. - -## Hypothesis - -Injecting a dedicated `viral_shares` engine (memes ranked by share-click / -invite conversion) into the mature blend at weight **0.2** (stolen from -`lr_smoothed`) increases unique **non-self** share clickers and new-user -invites per 1k memes sent without reducing session depth (p50 memes/session). - -## Changes Made - -- Engine: `src/recommendations/candidates.py::viral_shares` -- Experiment: `viral_shares_blender_v1` in `src/recommendations/blender_experiments.py` -- Wiring: mature path uses `get_mature_blend_weights_with_experiments` -- Delivery prep seam: `src/tgbot/senders/delivery.py` -- Crosspost fix: share-click CTEs accept both `m_` and `s_` deep links - -## Assignment - -- Eligible: mature users via mature blend path (`nmemes_sent ≥ 100`) -- Strategy: `sha256(experiment_id:user_id) % 2` -- Control: base mature weights (after recently_liked v2 default) -- Treatment: base + `viral_shares: 0.2`, `lr_smoothed` reduced by 0.2 -- Code constant `SAMPLE_GATE_PER_VARIANT = 1000` is **legacy**; decision sample - for this WAU is **≥80 users/variant and ≥2k sends/variant** at day-7, else - decide at day-14 (see HYPOTHESES.md) - -## Primary metrics - -1. Unique non-self share clickers (`user_deep_link_log` m_/s_) per 1k memes sent -2. New-user invites (`user.inviter_id`) per 1k memes sent - -## Guardrails - -- p50 session length (30m gap) — kill if treatment < control − 10% relative -- Like rate — kill if treatment < control − 5 pp (with enough volume) -- Engine continuation@30m for `viral_shares` vs `lr_smoothed` - -## Ship / kill rules - -See **H1 Decision rules** in `experiments/HYPOTHESES.md`. - -## Interim (2026-08-09 15:44 UTC) - -Full numbers: interim readout removed 2026-10-31 (in git history); final numbers in [`docs/analyst/readouts/2026-10-30-experiment-closure.md`](../../docs/analyst/readouts/2026-10-30-experiment-closure.md) - -- 10 control / 12 treatment users; 0 share clickers; 0 invites -- 13 `viral_shares` sends on treatment (wiring OK) -- **Verdict: HOLD** - -## Metrics Before / After - -| Window | control n | treatment n | clickers/1k C/T | invites/1k C/T | p50 session C/T | Decision | -|--------|-----------|-------------|-----------------|----------------|-----------------|----------| -| 2026-08-09 interim | 10 | 12 | 0 / 0 | 0 / 0 | 16.5 / 19.0 | HOLD | -| 2026-10-30 day-82 (final) | 344 | 350 | 0.031 / 0.025 | 0.012 / 0.018 | 14 / 10 (pooled) | KILL | - -## Conclusion - -**KILL (2026-10-30).** Enrollment stopped in code; every mature user gets the -shipped default blend (`MATURE_BLEND_WEIGHTS`). Existing assignment rows stay -for readouts. The `viral_shares` engine stays in `candidates.py` but no blend uses it. - -- Primary failed: unique non-self share clickers per 1k sends 0.031 control vs - 0.025 treatment (13 vs 10 clickers, 426k / 395k sends). -- Invites 5 vs 7 (0.012 vs 0.018 per 1k). One treatment inviter accounts for 4, so this is noise. -- Mean per-user LR 49.7% vs 47.5% (−2.2 pp, inside the 3 pp guardrail). -- The engine reached only 0.76% of treatment sends, so the arms barely differed. -- Pooled p50 session dropped 14 → 10, which trips the frozen kill trigger. The - user-level median of per-user p50 is flat (10 vs 10), so the drop is - heavy-user composition. It is not why we killed the test. - -Readout: [`docs/analyst/readouts/2026-10-30-experiment-closure.md`](../../docs/analyst/readouts/2026-10-30-experiment-closure.md). -The canned SQL §4 invite multiplier bug was fixed in the same PR. - -## Readout SQL - -`docs/analyst/viral-shares-blender-v1.sql` (removed 2026-10-31; in git history) diff --git a/scripts/experiments.py b/scripts/experiments.py new file mode 100644 index 00000000..059d2abe --- /dev/null +++ b/scripts/experiments.py @@ -0,0 +1,198 @@ +"""Experiment ledger CLI. One file per experiment in experiments/, front matter on top. + +python scripts/experiments.py list [--status running] +python scripts/experiments.py due +python scripts/experiments.py index # regenerates experiments/README.md +python scripts/experiments.py readout +""" + +import argparse +import datetime as dt +import sys +from pathlib import Path + +DIR = Path(__file__).resolve().parent.parent / "experiments" +README = DIR / "README.md" +KEYS = ( + "id status hypothesis primary_metric guardrails unit variants started " + "check_on decided result learning code readout" +).split() +STATUSES = ("proposed", "running", "shipped", "killed", "inconclusive") +OPEN = ("proposed", "running") + + +def parse(path: Path) -> dict: + """Flat `key: value` front matter between the first two `---` lines.""" + lines = path.read_text(encoding="utf-8").splitlines() + if not lines or lines[0].strip() != "---": + raise ValueError(f"{path.name}: no front matter") + meta = {} + for line in lines[1:]: + if line.strip() == "---": + meta["file"] = path.name + return meta + key, sep, value = line.partition(":") + if not sep: + raise ValueError(f"{path.name}: bad front matter line {line!r}") + meta[key.strip()] = value.strip() + raise ValueError(f"{path.name}: front matter not closed") + + +def load() -> list[dict]: + return [parse(p) for p in sorted(DIR.glob("*.md")) if p != README] + + +def _row(cells) -> str: + return "| " + " | ".join(str(c).replace("|", "\\|") for c in cells) + " |" + + +def _table(header, rows) -> list[str]: + return [_row(header), _row(["---"] * len(header)), *(_row(r) for r in rows)] + + +TEMPLATE = """\ +--- +id: my_experiment_v1 +status: proposed +hypothesis: one line, falsifiable +primary_metric: likes per active user per week +guardrails: LR not down >3 pp, block rate, empty queue +unit: user +variants: control:50,treatment:50 +started: +check_on: YYYY-MM-DD +decided: +result: unknown +learning: +code: FLAG_NAME, src/path.py +readout: docs/analyst/my-experiment-v1.sql +--- + +# Experiment: my_experiment_v1 + +Design, assignment, frozen decision rules, confounds. + +## Log +- YYYY-MM-DD: started (deploy sha / PR). +""" + +HEADER = f"""\ +# Experiments + +One file per experiment: `experiments/YYYY-MM-DD-slug.md` (date = when it was +designed or started). Status lives in the front matter, so files never move. +This README is generated: `python scripts/experiments.py index`. +**Read it before designing a new experiment:** the Decided table is everything +we tried and what we now believe. + +Growth bar for new experiments: ship only if the funnel gets less leaky +(first tap / first like / reached-5) and/or the k-factor moves (unique non-self +share clicks, new-user invites). Session length and block rate are guardrails. +North star: [`SPEC.md`](../SPEC.md). Growth thesis: +[`docs/growth/virality-loop.md`](../docs/growth/virality-loop.md). + +## Add, check, close + +1. **Add:** copy the template into `experiments/YYYY-MM-DD-slug.md`, freeze the + decision rule in the body, commit the readout SQL under `docs/analyst/`. + Set `status: running` and `started:` when it deploys. +2. **Check:** `python scripts/experiments.py due` lists running experiments + whose `check_on` has passed; `python scripts/experiments.py readout ` + prints the command to run against `$ANALYST_DATABASE_URL` (read-only). + Append dated numbers to the file's log. +3. **Close:** set `status` (shipped / killed / inconclusive), `decided`, + `result` (numbers + CI or n) and `learning` (one line: what we now believe). + Never delete the file. Readouts stay in `docs/analyst/readouts/`; link them. +4. Run `python scripts/experiments.py index` and commit the README with it. + +Front matter is flat `key: value`; lists are comma-separated; unknown is `unknown`. +Status: {" | ".join(STATUSES)}. + +```markdown +{TEMPLATE}``` +""" + + +def render(exps: list[dict]) -> str: + def link(e): + return f"[{e['id']}]({e['file']})" + + open_ = sorted( + (e for e in exps if e["status"] in OPEN), + key=lambda e: (e["check_on"] or "9999", e["id"]), + ) + done = sorted( + (e for e in exps if e["status"] not in OPEN), + key=lambda e: (e["decided"] or e["started"], e["id"]), + reverse=True, + ) + out = [HEADER, f"## Open ({len(open_)})", ""] + out += _table( + ["check_on", "experiment", "status", "primary metric", "hypothesis"], + ( + [e["check_on"] or "-", link(e), e["status"], e["primary_metric"], e["hypothesis"]] + for e in open_ + ), + ) + out += ["", f"## Decided ({len(done)}, newest first)", ""] + out += _table( + ["decided", "experiment", "status", "result", "learning"], + ([e["decided"] or "-", link(e), e["status"], e["result"], e["learning"]] for e in done), + ) + return "\n".join(out) + "\n" + + +def overdue(e: dict, today: str) -> bool: + return e["status"] == "running" and (not e["check_on"] or e["check_on"] <= today) + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + sub = ap.add_subparsers(dest="cmd", required=True) + sub.add_parser("list").add_argument("--status", choices=STATUSES) + sub.add_parser("due") + sub.add_parser("index") + sub.add_parser("readout").add_argument("id") + args = ap.parse_args(argv) + exps = load() + today = dt.date.today().isoformat() + + if args.cmd in ("list", "due"): + rows = exps + if args.cmd == "due": + rows = [e for e in exps if overdue(e, today)] + elif args.status: + rows = [e for e in exps if e["status"] == args.status] + for e in rows: + flag = "" + if overdue(e, today): + flag = "OVERDUE" if e["check_on"] else "NO DATE" + print( + f"{e['id']:<38} {e['status']:<12} {e['check_on'] or '-':<10} " + f"{flag:<7} {e['primary_metric'][:40]:<40} {e['result'][:70]}" + ) + elif args.cmd == "index": + README.write_text(render(exps), encoding="utf-8") + print(f"wrote {README.relative_to(DIR.parent)} ({len(exps)} experiments)") + elif args.cmd == "readout": + match = [e for e in exps if e["id"] == args.id] + if not match: + print(f"no experiment with id {args.id}", file=sys.stderr) + return 1 + e = match[0] + print(f"file: experiments/{e['file']}") + paths = [p.strip() for p in e["readout"].split(",") if p.strip()] + if not paths: + print("no readout recorded") + for p in paths: + if p.endswith(".sql"): + print(f'psql "$ANALYST_DATABASE_URL" -f {p}') + elif p.endswith(".py"): + print(f"python {p}") + else: + print(f"read: {p}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/specs/describe-memes.md b/specs/describe-memes.md index d8dac433..6fccdebb 100644 --- a/specs/describe-memes.md +++ b/specs/describe-memes.md @@ -8,7 +8,7 @@ the image has no text. Rows described before 2026-10-30 lack the key; read known languages, never from a text-free image. Since 2026-10-31, `ocr_result` also stores three content tags for the crosspost -experiment (`experiments/active/2026-10-31-crosspost-content-tags.md`): +experiment (`experiments/2026-10-31-crosspost-content-tags.md`): - `format`: one of single_caption, multi_panel, tweet_screenshot, chat_screenshot, photo_no_text, drawing_comic, other; - `relatable`: bool; diff --git a/src/config.py b/src/config.py index c0929c3e..0998cfa6 100644 --- a/src/config.py +++ b/src/config.py @@ -67,7 +67,7 @@ class Config(BaseSettings): # blind Redis queue pop. Kill switch rolls back to queue path. BROADCAST_HIGH_QUALITY_PICK_ENABLED: bool = True # broadcast_cadence_holdout_v1: no-push holdout per cadence step (15m/24h - # 15%, 1w/2w/4w 20%). See experiments/active/2026-10-30-*. + # 15%, 1w/2w/4w 20%). See experiments/2026-10-30-broadcast-cadence-holdout.md. BROADCAST_CADENCE_HOLDOUT_ENABLED: bool = True # Monthly reminder (users whose last reaction was 29-365 days ago). # Owner flips on after deploy. Set HOLDOUT_PCT=0 after the v1 readout. diff --git a/src/crossposting/shadow_score.py b/src/crossposting/shadow_score.py index 9dd316ab..8e9e4755 100644 --- a/src/crossposting/shadow_score.py +++ b/src/crossposting/shadow_score.py @@ -1,7 +1,7 @@ """Crosspost v5 shadow scorer — log only, never changes the production pick. From the 2026-10-30 walk-forward study (target zf = log1p(forwards) minus -channel×week mean; see experiments/active/2026-10-30-crosspost-v5-shadow.md): +channel×week mean; see experiments/2026-10-30-crosspost-v5-shadow.md): RU: score_v5 = ln(nlikes+1) * theta_src * exp(0.5 * user_taste_z) EN: score_v5 = theta_src * exp(user_taste_z) (like volume hurts on EN) @@ -35,7 +35,7 @@ def content_mult_v1(content_format: str | None, relatable: bool | None) -> float """RU blind-label study 2026-10-30: relatable and tweet screenshots ≈ −0.1 zf each. Tags come from Describe Memes (meme.ocr_result); missing tags → 1.0. - See experiments/active/2026-10-31-crosspost-content-tags.md. + See experiments/2026-10-31-crosspost-content-tags.md. """ return 0.85 ** (int(relatable is True) + int(content_format == "tweet_screenshot")) @@ -46,7 +46,7 @@ def src_fwd_mult(src_fwd_pct: float | None) -> float: src_fwd_pct: within-source percentile of the source post's forwards/views (Telethon read at ~20–48h). Study 2026-10-31: Spearman with our crosspost forwards +0.52 RU / +0.28 EN. See - experiments/active/2026-11-01-crosspost-src-forward-rate.md. + experiments/2026-11-01-crosspost-src-forward-rate.md. """ return 1.0 if src_fwd_pct is None else 0.5 + float(src_fwd_pct) diff --git a/src/flows/storage/describe_memes_repository.py b/src/flows/storage/describe_memes_repository.py index 9df7b481..4f9953e7 100644 --- a/src/flows/storage/describe_memes_repository.py +++ b/src/flows/storage/describe_memes_repository.py @@ -25,7 +25,7 @@ } # Content tags for the crosspost shadow multiplier -# (experiments/active/2026-10-31-crosspost-content-tags.md). +# (experiments/2026-10-31-crosspost-content-tags.md). MEME_FORMATS = { "single_caption", "multi_panel", diff --git a/src/recommendations/blender_experiments.py b/src/recommendations/blender_experiments.py index 61394139..6d73a7c3 100644 --- a/src/recommendations/blender_experiments.py +++ b/src/recommendations/blender_experiments.py @@ -7,7 +7,7 @@ # --- explore_budget_v1: labeling budget + fresh-viral + idle-return first slot --- -# See experiments/active/2026-10-30-explore-budget.md. +# See experiments/2026-10-30-explore-budget.md. EXPLORE_BUDGET_V1_EXPERIMENT_ID = "explore_budget_v1" EXPLORE_BUDGET_V1_CONTROL = "control" diff --git a/tests/test_experiments_ledger.py b/tests/test_experiments_ledger.py new file mode 100644 index 00000000..dfc39f6e --- /dev/null +++ b/tests/test_experiments_ledger.py @@ -0,0 +1,38 @@ +"""experiments/*.md parse and experiments/README.md matches `scripts/experiments.py index`.""" + +import importlib.util +import re +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +_spec = importlib.util.spec_from_file_location( + "experiments_cli", ROOT / "scripts" / "experiments.py" +) +ledger = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(ledger) + +DATE = re.compile(r"^\d{4}-\d{2}-\d{2}$") + + +def test_every_experiment_file_is_valid(): + exps = ledger.load() + assert exps + ids = [e["id"] for e in exps] + assert len(ids) == len(set(ids)), "duplicate experiment id" + for e in exps: + name = e["file"] + assert re.match(r"^\d{4}-\d{2}-\d{2}-[a-z0-9-]+\.md$", name), name + assert set(ledger.KEYS) <= set(e), f"{name}: missing {set(ledger.KEYS) - set(e)}" + assert e["status"] in ledger.STATUSES, f"{name}: status {e['status']!r}" + for key in ("id", "hypothesis", "primary_metric", "result"): + assert e[key], f"{name}: empty {key}" + for key in ("started", "check_on", "decided"): + assert not e[key] or DATE.match(e[key]), f"{name}: {key} {e[key]!r}" + if e["status"] not in ledger.OPEN: + assert e["decided"], f"{name}: decided experiment needs a decided date" + + +def test_readme_is_up_to_date(): + assert ledger.README.read_text(encoding="utf-8") == ledger.render(ledger.load()), ( + "experiments/README.md is stale: run `python scripts/experiments.py index`" + )