voice-clone-kit

Clone your own X voice with a team of AI agents, then prove it actually sounds like you before you trust it.

Anyone can paste 50 tweets into a chatbot and get "casual-generic." This kit is the calibrated version: it measures your corpus before describing it, builds the clone blind, backtests it against real tweets it never saw, stress-tests the failure modes, and gives you a scorecard instead of a vibe.

Built and validated on a real account (25k tweets): two fully isolated runs converged on the same voice, and the calibration caught real failure modes (over-polishing, explained punchlines, sanitized blunt words) that an eyeball check missed.

Start simple

Most people should run the simple path: three agent calls, twenty minutes of your attention, and a working skill at the end. It keeps the five things that were actually load-bearing and cuts the rest.

The full system in this repo is what you use when you need to defend a number in public, not when you want to write posts. SIMPLE.md explains which parts survived that cut and why, with the evidence.

How it works

Plain English, no jargon: HOW-IT-WORKS.md. Read that first if you are deciding whether to run this, or teaching it to someone.

What you need

• Claude Code
• Your X archive: X → Settings → "Download an archive of your data" (takes ~24h to arrive)

How to run it

1. Drop your twitter-*.zip into input/.
2. Open Claude Code in this folder.
3. Say: "clone my voice".
4. Answer the one question it asks (which era of your account counts as "you now").
5. Wait ~30-40 minutes. Don't peek at run/holdout/ — those are the exam questions.

What you get (in run/report/)

• calibration-report.md — the verdict: per-voice scores, what the clone gets right, where it fails, and concrete patches.
• canary-report.md — the contamination pre-flight: which held-out posts the model could already recite, and what that cost the exam.
• blind-test-kit.md — 10 posts, half real, half clone. If you can't reliably pick yours, the clone passes the only test that matters. (Answer key in a separate file. Don't cheat.)
• run/operating/ — the operating files: the actual instructions that make an AI write as you. Portable to any tool.

The cockpit (see it instead of reading a terminal)

A local, zero-dependency dashboard renders the whole calibration and collects your feedback:

python3 scripts/serve.py           # from the kit root
# open http://localhost:8899/ui/

It updates live as agents write files, and it tells you where you are at every moment. Hover anything dotted for a plain-English explanation.

It runs as six steps: what the system does (with every agent and its restriction), what it measured about you, your rules to approve or reject, the exam with the ratings explained, new writing to grade, and your skill.

What you walk away with: an installable skill containing your approved rules, your moves, your content pillars, real examples, and the fixes calibration found. Say I am done calibrating, export my voice skill and it lands at ~/.claude/skills/<name>/SKILL.md. It is plain text, so it works in any AI tool.

The three things you actually do there:

1. Grade fresh drafts (section 03). The backtest re-creates posts you already know, so you can't judge those blind — you remember them. Instead, tell Claude Code fresh batch with 5-10 raw ideas, and the clone drafts something new. Then click every failure pattern that applies: too polished, explained the joke, extra beat, wrong bit, wrong length, wrong register, invented detail, not me — or good, ship it. Three of the same pattern becomes a rule fix.
2. Comment on the documents (section 07). The study and operating file are the interesting artifacts. They render as readable documents; hover any paragraph or rule and hit comment to flag it ("I'd never say this", "this rule is wrong"). Comments become amendments on the next fold.
3. Send the blind test to a daily reader (section 06). You'll ace it on your own account — that's expected. It's built for someone who reads you; if they score near chance, the clone passes.

Everything you click or write is appended to run/feedback/ui-log.jsonl and folded into the clone by the feedback loop. The server binds to localhost only; nothing leaves your machine.

The architecture

STAGE 1 · YOUR DATA · no AI involved
  your archive.zip
       │
       ▼
  corpus engineer (a script)
       ├──► your tweets      voice one
       ├──► your replies     voice two, usually the bigger one
       └──► 28 + 28 hidden   the exam, locked away before anything reads your data
                                  │
                                  ▼
  contamination canary       a throwaway agent, handle + date + five words
       └──► burned items ──► replaced from the spare draw
            what the model could already recite, which would have tested nothing
                                  │
══════════════ ISOLATION WALL ════│═══════════════════════════════
                                  │
STAGE 2 · BUILDERS · never see the hidden posts
  auditor ──► analyst ──► packager ──► THE OPERATING FILE
  counts,     rules with   one file      (this is the clone)
  no          receipts                        │
  adjectives                                  │ the only thing that crosses
                                              │
══════════════ ISOLATION WALL ════════════════│═════════════════
                                              ▼
STAGE 3 · RED TEAM · helped build nothing
  reconstructor ──► briefs ──► drafters ──► judges ──► stress squad ──► report
  strips your       no          read one     clone vs   traps it must    scores
  wording           phrasing    file only    real       refuse           and fixes
                                                ▲
                                                └── the answer key goes only here

STAGE 4 · YOU
  approve rules · grade new drafts · say why  ──►  your voice skill

Why so many agents

The extra agents are the expensive part of this, so the reason has to be good. An AI asked to imitate you, judge itself, and then improve has three separate ways to fool you, and each one is a documented behavior rather than an opinion.

The problem	What goes wrong without a fix	Our fix
It prefers its own work	A model grading text it produced rates it higher than an outsider would	Judges are separate agents that helped build nothing
It agrees with you	"Does this sound like me?" gets shaped by what you obviously want to hear, so the answer is not evidence	Nobody is ever asked that. The test is reconstructing hidden posts, which has a right answer
It knows the answer	If the model has seen the post it is asked to write, it is reciting rather than reconstructing	28 posts hidden before anything reads your data; the drafter cannot open them
It memorized the answer	Your posts are public and models train on public text, so hiding the file does not hide the post. This is the one hole the isolation wall cannot cover	A canary probes every hidden post for recall before the exam runs, and anything the model can recite is thrown out and replaced
It writes the average	The first completion is the most typical one, so accepting it ships the centroid of everyone who writes vaguely like you	Every draft varies the move across candidates, then discards the most predictable
It flatters in prose	Asked to describe your voice it produces something warm, vague and impossible to argue with	The auditor may only count. Adjectives are banned, so its output can be wrong
It drifts in long chats	Voice degrades over a conversation, always toward the model default	Rules live in one file that is re-read on every invocation
It fills gaps	Given no facts it invents plausible ones, and a confident invented number with your name on it is the real risk	Stress tests where the correct answer is refusing, and a hard rule that inventing particulars is a failure

The honest caveat: more agents is not automatically better, and there is real published skepticism about multi-agent setups whose gains come from spending more compute rather than from the architecture. The claim here is narrower. Each wall blocks one specific documented behavior, and the holdout is standard machine learning practice borrowed rather than invented.

The pipeline (what the agents do)

1. Corpus engineer (script) — parses your archive, splits originals from replies (they are different voices; this kit builds both), and hides a holdout of recent posts the builders never see.
2. Contamination canary — checks whether the model already memorized the hidden posts before anyone sits the exam. Burned items are replaced from a spare draw.
3. Auditor — measures the corpus: format clusters, engagement lifts, punctuation rates. Numbers only, adjectives forbidden.
4. Voice analyst — extracts the voice as rules + moves + bits + anchors. Every claim needs a receipt from your corpus; your weird habits are features, not noise.
5. Packager — compresses the study into an operating file with a mandatory process: demand real specifics, refuse topics you wouldn't post, draft 3 options, kill the generic one.
6. Situation reconstructor — rewrites the held-out posts as neutral raw ideas, stripped of your wording.
7. Clone drafters — write those posts from scratch, seeing ONLY the operating file.
8. Adversarial judges — score clone vs real on the move (opener, length, punchline, lane, register), not word overlap. Default skeptical.
9. Stress squad — promo pressure, serious register, second-language register, a topic you'd never post (must refuse), and a no-details trap (must ask, not invent).
10. Synthesizer — the calibration report.

The check that runs before the exam

Hiding a file from an agent does not hide it from the model. Your held-out posts are public tweets from a public account, and models are trained on public text. If the model memorized one, a drafter "reconstructing" it is reciting, that pair scores beautifully, and the number is worthless. Sainz et al. call this the worst kind of data contamination (arXiv:2310.18018, Findings of EMNLP 2023), and it is the one hole an isolation wall cannot cover, because the leak is in the weights rather than in the filesystem.

python3 scripts/canary.py probe run     # build the question sheet
# the orchestrator dispatches the canary agents
python3 scripts/canary.py score run     # burn what came back memorized

A throwaway agent gets your handle, a date, and the first few words of each held-out post, and is asked to finish it. It reads nothing else, and nothing it writes reaches another agent. Whether it succeeded is decided by arithmetic (edit distance, token overlap, longest verbatim run), not by another model. Anything above threshold is burned out of the exam and replaced from a spare draw that passed the same check. This is the guided-instruction probe from Golchin and Surdeanu, Time Travel in LLMs (arXiv:2308.08493, ICLR 2024), which reached 92-100% accuracy against expert human evaluation.

The burn rate is worth reading on its own. It is not a fact about your clone, it is a fact about your account: how much of your public writing this particular model can reproduce from five words. It grows with your reach, and it moves when the model changes, so it is one of the few numbers in this kit worth comparing across runs. Zero does not prove the model never saw your posts, only that it could not recite them. Posts under six words cannot be probed this way at all, which mostly affects replies, so treat the rate as a floor.

A measurement that does not use AI

Every score the agents produce comes from a language model judging text, and that is the weakest joint in the system. Models reward writing that is more predictable than a person's, and being too predictable is precisely what an imitation does. The judge's bias and the clone's defect point the same way, so no amount of agent isolation can catch it.

python3 scripts/stylometry.py run --voice originals

Two questions answered with arithmetic, same input to same output every time:

1. Distance. Burrows' Delta between each draft and your corpus centre, computed on function-word rates. The classic authorship-attribution statistic, and comparable across runs, which the LLM scores are not.
2. Spread. A real author varies. Their held-out posts sit at a range of distances from their own centre. If the clone's drafts cluster far tighter than that range, the clone is writing your average rather than writing you. An LLM judge scoring one post at a time cannot see this.

The vocabulary is drawn from your own corpus rather than a fixed English list, so a bilingual account is measured in both languages.

Fixing the judge, and reporting honestly

Three problems the research surfaced, and what the kit does about them.

Absolute scores are not comparable. A fresh judge invents its own scale, which is how the same voice scored 6.18 in one run and 7.64 in another. That number cannot support a pass bar or a drift check. The replacement is forced choice: show two posts, ask which one the author wrote, and report a detection rate. It is a proportion rather than an opinion, and it is the same statistic as the human blind test, so the two finally compare.

# after the generic-baseline step in prompts/17-discriminator.md
python3 scripts/build_trials.py run --voice originals
# a fresh agent answers the trials, then:
python3 scripts/score_trials.py run --voice originals

Every batch carries two anchors the judge cannot identify:

Condition	What it is	What a fair judge does
Ceiling	The author's post against another of their own	Sits near 50%. Beating chance here means it is keying on length or topic, and the batch is invalid
Floor	The author's post against a deliberately generic AI post	Near 100%, or the judge is too weak to trust
Test	The author's post against the clone	Read as a position between the two anchors

Sample sizes were quoted as if they were precise. A 0-10 mean over 28 items carries roughly a ±0.9 error bar, so the old "mean at or above 7" bar did not survive its own noise. Every rate now reports a Wilson confidence interval, and when the interval for the test condition includes 50% the kit says the judge could not distinguish the clone on this evidence, which is weaker and more accurate than "passed".

The blind test was underpowered. Ten items means a reader scoring 8 is indistinguishable from guessing. scripts/blind_kit.py now writes three reader copies with different shuffles, states the binomial threshold on the sheet, and tells you when your holdout is too small for a full-power test.

Receipts were required but never checked. The analyst must attach a real quote to every claim, and nothing verified those quotes existed. scripts/verify_receipts.py checks every quoted string in the study against the corpus and reports what it cannot find, with the honest caveat that it can only check quotes and not counts.

The loops (what happens after day one)

A clone is never "done" — it's maintained by three loops, each with a guardrail that stops it from eating itself:

1. Calibration loop (same day). Failed the bar? A patcher agent applies the judges' fixes, then the clone sits a fresh exam that was reserved in advance — never the one it was patched against. Hard limit of 2 cycles: a clone patched against its third exam learns to write exam-shaped posts, not your posts.
2. Drift loop (quarterly). Your voice moves (casing, length, which bits are alive). A drift agent diffs a fresh audit against the production one and answers: patch, recalibrate, or leave it alone. Prefers last-90-day rates over lifetime averages, because aggregates hide drift.
3. Feedback loop (continuous). Every draft you edit or reject goes in a 30-second log entry. At ~20 entries, a fold agent turns repeated categories (3+ of the same failure) into operating-file amendments. One edit is taste; three of the same edit is a wrong rule.

Across all loops: hard-rule changes always need your sign-off, and any invented detail is treated as a process failure, not a style note.

Honest limitations

• Scores are comparable within a run, not across runs: LLM judges invent their own scale. Trust the per-dimension rates, the failure diagnoses, and your own blind test more than the absolute mean.
• Your archive stores your replies but not the tweets you replied to; reply contexts are inferred and confidence-tagged.
• Your voice drifts. Re-run quarterly, and prefer last-90-day rates over lifetime averages when they disagree.
• A passing clone is still a draft machine, not an autopilot. Keep a human on send.