Repository navigation
Plan a second attempt at WP-1 and WP-10 - #46
Merged
Merged
Conversation
WP-1b replaces the global z-score cap, which cost short-text detection and sparse-reference calibration in #30, with typography invariance: on 0.2.0, curly quotes alone move the writer's own essays to somewhat or clearly different through one encoding-only metric. WP-10a has a different model write a held-out LLM test set first; WP-10b retries the generic contrast with drafts varied in length and register, a leave-one-register-out anchor, and bars written down before the test. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01TrPKUqNFs1JKzqZmNE84oK
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
This adds
plans/retry-wp1-wp10.md: a second attempt at the two onboarding work packages that were deferred in 0.2.0, redesigned from what #30 and #37 found. It also adds a pointer to it inplans/onboarding.md. Only documentation changes; no code.WP-1b: typography never changes a verdict. This replaces the global z-score cap. In Cap metric z-scores in Delta and LLM-likeness (onboarding plan WP-1) #30 the cap fixed its target case, but it also cut Delta's flags on 150-word LLM pieces from 19 of 20 to 5–6 of 20, and it made a two-document reference too narrow. The target case is already fixed in 0.2.0, by WP-5. A new failure has turned up on 0.2.0: converting the writer's own essays to curly quotes moves all seven to "somewhat different" or "clearly different" (Delta 1.22–1.46). One encoding-only metric,
curly_apostrophe_pct, causes it, and straightening the LLM drafts' quotes changes none of their scores. The fix is a suite of 11 reformatting transforms that must not change any verdict, with no cap and no change to calibration math.WP-10a: a held-out test set. A different model (Haiku 4.5) writes 16 LLM drafts in response to fixed prompts, without first reading the project. WP-10b gets tested against them.
WP-10b: generic contrast, second attempt. It fixes the two causes of Preserve deferred generic contrast candidate (onboarding plan WP-10) #37's failure:
The design and the pass bars are committed before the test set is used, and the test runs once.
All three follow the onboarding plan's rules for every work package. No existing acceptance threshold changes.
🤖 Generated with Claude Code
https://claude.ai/code/session_01TrPKUqNFs1JKzqZmNE84oK
Generated by Claude Code