From 8f4701e96659f6ac45efad8aa6724c56f08d7f11 Mon Sep 17 00:00:00 2001 From: Abhyudaya Baiju Tharakan Date: Sun, 15 Nov 2026 09:03:37 +0530 Subject: [PATCH] docs(readme): open with the pitch, a screenshot and a quick start The README began with deployment and branch-status detail written for hackathon judges, so a new reader could not tell what CounterLab does or how to try it. The top now gives a one-line description, the start screen, a diagram of the claim-to-verifier flow, the two measured results from ACHIEVED_METRICS.json, and the shortest path to running it. The earlier introduction is kept under Status and scope. The screenshot moves from the repository root to docs/assets. --- README.md | 61 +++++++++++++++++- .../assets/counterlab-start.png | Bin 2 files changed, 58 insertions(+), 3 deletions(-) rename cloak-normal-counterlab.png => docs/assets/counterlab-start.png (100%) diff --git a/README.md b/README.md index 3dadab3a..e830bb06 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,66 @@ # CounterLab -**Ask like chat. Prove it like science.** CounterLab is a scientific debugger -for beliefs: it formalizes a claim, commits a prediction, runs a discriminating -test, refuses invalid evidence, verifies transfer, and only then unlocks repair. +**A belief debugger for machine-learning evaluation mistakes.** State a claim +about your model, lock in a prediction, and CounterLab runs a controlled +experiment on your Jupyter notebook's evidence. A fixed verifier, not a language +model, decides whether the result counts. > Chatbots explain. CounterLab lets reality answer. +**Try it now, no account needed:** + +

+ CounterLab start screen: state a claim or attach a notebook, then test the claim +

+ +## How it works + +```mermaid +flowchart LR + A["1. Claim
'My model generalises:
98% test accuracy'"] --> B["2. Prediction
locked before the test"] + B --> C["3. Controlled test
fixed kernel changes
one thing"] + C --> D{"4. Frozen verifier"} + D -->|evidence valid| E["5. Result, boundary map
and verified repair"] + D -->|evidence invalid| F["Refused with a
typed reason"] +``` + +A language model only proposes how to frame the claim. Fixed code computes every +number, and a frozen verifier releases a result only when the evidence binds to +it. Unsupported notebooks are refused, not guessed at. + +## What it catches + +Measured on the bundled fixtures, recorded in +[docs/ACHIEVED_METRICS.json](docs/ACHIEVED_METRICS.json): + +| Mistake | What the notebook reports | What the controlled test shows | +| --------------------------------- | ------------------------------------------------------------------------- | ---------------------------------------------------- | +| Entity leakage across the split | 98.5% accuracy on a random split | 59.4% accuracy once each customer stays in one split | +| Class imbalance and metric choice | 98.9% accuracy from always predicting the majority class (1.1% positives) | 0% recall: that model catches none of the rare cases | + +Seeded mutation benchmarks detected 14 of 14 leakage variants and 15 of 15 +imbalance variants. No learner-study outcome has been measured. + +## Quick start + +The fastest route is the hosted demo above: choose **Start verified sample**. + +To run it locally you need Node 22+, pnpm 11.13.1 and Python 3.12+: + +```bash +git clone https://github.com/ALikesToCode/CounterLab.git +cd CounterLab +cp .env.example .env.local +./scripts/clean-demo.sh +``` + +Then open . The script installs dependencies, sets up the +local database, starts the fixed kernel and the web app, and prints which live +capabilities are available. The sample and replay paths need no API key; +[Live mode](#live-mode) covers the keys for testing your own notebook. + +## Status and scope + CounterLab is a narrow Education-track product for machine-learning evaluation misconceptions in supported Jupyter notebooks. Its released Subject Packs cover entity leakage and class imbalance/metric choice; it is not a generic notebook diff --git a/cloak-normal-counterlab.png b/docs/assets/counterlab-start.png similarity index 100% rename from cloak-normal-counterlab.png rename to docs/assets/counterlab-start.png