[wangjohn]

What and why

Implement WP-10a of plans/retry-wp1-wp10.md: Write 16 original LLM-generated text samples for use as a held-out evaluation set in testing WP-10b (Generic contrast, second attempt).

The evaluation set exists specifically to provide text that is independent of any corpus used for training WP-10b, so that evaluation results are not biased by overfitting to training data.

Drafts

All 16 drafts were generated by Claude Haiku 4.5 on 2026-10-13 in response to the specified prompts, without reviewing any part of the styleprofile codebase. The text is original and unedited.

#	Prompt	Target	Actual	Δ	File
1	Write a personal essay about an old hand-drawn map you found in a relative's attic.	~600	600	0%	01-old-maps.md
2	Write a blog post about why stone walls and fence lines matter in old farm country.	~800	846	+5%	02-stone-walls.md
3	Write a short essay about mud season in Vermont.	~350	446	+27%	03-mud-season.md
4	Write a newsletter piece about heating a house with a woodstove all winter.	~700	700	0%	04-woodstove.md
5	Write a reflective essay about a small-town public library.	~900	837	-7%	05-library.md
6	Write a personal essay about walking in the rain.	~300	375	+25%	06-walking-rain.md
7	Write a blog post about learning to sharpen hand tools properly.	~1,000	969	-3%	07-sharpening-tools.md
8	Write an essay about the village hardware store and why it is worth keeping.	~500	605	+21%	08-hardware-store.md
9	Write a blog post about what I learned from a year of working remotely.	~800	837	+4%	09-remote-work.md
10	Write a short personal essay about baking sourdough for the first time.	~400	535	+33%	10-sourdough.md
11	Write an article giving advice on buying a used car.	~900	907	0%	11-used-car.md
12	Write a newsletter intro about our neighborhood park's cleanup day.	~250	306	+22%	12-park-cleanup.md
13	Write an essay about starting piano lessons as an adult.	~600	646	+7%	13-piano.md
14	Write a blog post with practical tips for sticking to a monthly budget.	~700	687	-1%	14-budgeting.md
15	Write a travel piece about a long train journey.	~1,000	983	-1%	15-train-journey.md
16	Write a reflective essay about what a vegetable garden teaches you.	~500	635	+27%	16-vegetable-garden.md

Word counts range from -7% to +33% of target. Several drafts (03, 06, 08, 10, 12, 16) run 21-33% over the requested length.

Packaging and testing

• Excluded from wheel: bench/eval/llm-heldout/ is not included in the wheel distribution (wheel only includes src/ and examples/).
• Included in sdist: bench/eval/llm-heldout/ is included in the source distribution for test suite and WP-10b evaluation use.
• Packaging test: Added test_evaluation_drafts_not_in_wheel() in tests/test_packaging.py to verify the evaluation set does not appear in the wheel but is present in the sdist.

Decisions

1. No look-ahead: The drafts were written without reviewing examples/, bench/, src/, or any styleprofile documentation, as required by WP-10a step 1. The text represents what the model naturally produces when asked to write on these topics, uninfluenced by knowledge of how styleprofile measures text.

2. Unedited: Each draft was generated in response to its prompt as a fresh request and was not edited afterwards. Word counts reflect the raw generation without post-processing.

3. Copyright: All text is original and released under the MIT License, consistent with the project license.

4. Evaluation-only designation: The README in bench/eval/llm-heldout/ explicitly states that these samples are for evaluation only and must not be used for training, tuning, or corpus selection in WP-10b or any other work.

Done when

✅ The 16 drafts and README are in place.
✅ The PR body lists each prompt, the draft's word count, and confirms rule 1 was kept.
✅ Packaging test verifies the set is not in the wheel and is included in the sdist.

---

🤖 Generated with Claude Code

https://claude.ai/code/session_01TKJj1EB4dNR9C5iS7jqqVZ

---

Generated by Claude Code

[wangjohn]

test (Python 3.11, windows-latest, syntax=false) failed on 66d08c1, and the failure is not this PR's.

The failing test is tests/test_cache.py::test_processes_can_share_the_cache. Three processes write to the measurement cache at once, and on Windows one writer's entries were missing afterwards.

This PR changes only the drafts in bench/eval/llm-heldout/, the sdist list in pyproject.toml, and tests/test_packaging.py. None of these touch the cache. The same job passed on this PR's first push (639ca64), and on #46 and #48 against the same main.

No fix exists yet. The test looks timing-sensitive under concurrent SQLite writers on slow Windows runners, so it needs its own follow-up rather than a change here. I'm re-running the failed job once.

---

Generated by Claude Code