Skip to content

Add rhetorical metrics and personal feedback profiles - #51

Draft
wangjohn wants to merge 2 commits into
mainfrom
codex/personal-feedback-metrics
Draft

wangjohn wants to merge 2 commits into
mainfrom
codex/personal-feedback-metrics

Conversation

@wangjohn

@wangjohn wangjohn commented Oct 16, 2026 •

Copy link
Copy Markdown
Owner

A passage can sound unlike a writer even when its aggregate style-distance label says “close.” This adds sentence-level rhetorical evidence and an optional personal profile that reuses existing passage ratings.

The four measurements cover hypothetical reader prompts, observation prefaces, short/long sentence pairs, and explicit lesson statements. Each returns counts, rates per 1,000 prose words, and exact evidence spans in normalized text. Baseline Delta/contrast features score the original draft, preserving its Markdown structure.

feedback-build learns separate voice and revision-effort associations from rated JSONL passages, the existing scores, and the new measurements. feedback-score reports those estimates with evidence. Profiles embed their base reference and preserve rating IDs, notes, grouping, and text hashes so later judgments can build on earlier feedback. Ordinary build/score behavior is unchanged; there are no new runtime dependencies.

Topic metadata is resolved consistently across every passage from a source before fitting; missing topic labels inherit the source’s supplied topic, conflicting assignments are rejected, and source/topic namespaces cannot collide. Grouped development diagnostics hold related passages out together and fit scaling on training groups only. Each target is disabled when it does not beat a mean-only baseline, and estimates are withheld below 75 words.

File I/O uses explicit UTF-8. Non-object JSON profiles produce a normal invalid-profile error. The experimental feedback format is version 2: earlier version-1 feedback profiles must be rebuilt from saved ratings and the base reference because baseline feature semantics changed. Existing judgments need not be repeated. Estimates remain experimental personal associations, not AI-authorship probabilities; feedback used to design features is not independent validation.

Validation:

  • Full suite without spaCy: pytest -n 4 — 895 passed, 46 skipped.
  • Focused feedback/CLI tests: 30 passed, including source identity with missing/conflicting topic metadata, namespace collisions, Markdown baseline preservation, simulated CP1252 file I/O, invalid JSON roots, and old-format rejection.
  • Evidence offsets, false-positive controls, grouped holdouts, target gating, serialization, and CLI round trips remain covered.
  • Ruff lint and formatting checks pass across the repository.
  • Pyright passes using the isolated development environment.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant