Repository navigation
0.8.0: real-text measurement and precision rules — wild FP 20% → 4.2%, verified FP 8.8% → 1.7%, no retrain - #27
Merged
Merged
Conversation
…run the shipped weights over them Everything the repo measures today comes from the generator or from gold we wrote ourselves. sankhya.wild adds the other half: a source registry (Dakshina romanised Hindi + native-script hi/mr/gu Wikipedia, CMU DoG Hinglish, with licence and redistributability per source), a lexicon-only amount-likelihood score (unit words, prefix words, currency markers, digit+unit shapes -- the model never sees a line before it is sampled), seeded deduplicated sampling of candidate positives and no-signal negatives per language, and a run mode that applies exactly eval_gold's gates and reports any-span share, strict (verified) coverage, the calibrated-confidence distribution and the common token patterns. Also adds tokens_from_lexicon(), which splits a span into core.evaluate tokens using only the pack lexicon -- the way the Marathi/Gujarati gold authors checked their labels by hand. Raw corpora and samples are gitignored (the sources are share-alike); the aggregate metrics JSONs are not.
100 sampled real lines per language (50 the model predicted a span on, 25 lexicon-positive lines it stayed silent on, 25 no-signal negatives), labelled by hand into tests/gold_wild_<lang>.jsonl. Every positive value was checked with core.evaluate on the tokens the pack lexicon implies; the five spans the lexicon cannot tokenize on its own were checked by hand. Sentences come from CC BY-SA sources, attributed in tests/GOLD_WILD_SOURCES.md. tests/test_gold_wild.py is deliberately NOT a gate (test_gates.py is untouched): it only checks the files load, that core.evaluate reproduces every label, and that the shipped weights still evaluate over them. Headline, shipped int8 weights, wild vs. synthetic gold: value accuracy 0.899 vs 0.962, strict coverage 0.869 vs 0.909, strict value accuracy 1.000 vs 1.000 -- but the FP rate on real negatives is 0.200 (0.088 even in strict mode) against 0.000 on our own negatives, and span precision drops 0.972 -> 0.685. REPORT.md has the per-language tables, the top 10 failure shapes with examples, and what real text contains that the generator never produces.
…gates
Cuts the wild-gold false-positive rate from 0.200 to 0.042 and the strict
(verified) FP rate from 0.088 to 0.017 with no measured cost on the
synthetic gold (value accuracy and strict coverage unchanged to 4 dp).
New, mirrored 1:1 in python/sankhya and src/:
R11 core.should_drop_lone_prefix lone PFX_ span ("ढाई साल")
R12 verify.should_drop_ambiguous_form span held up only by pack
`ambiguous_forms` surfaces
("so", "sath", "arab", "अरब")
R13b verify.should_drop_unsupported_bare_cardinal
bare cardinal that is not an
exact lexicon form
R14 verify blocked surfaces a pack `blocked_surfaces` entry
never verifies and is dropped
("हजारे", "अरबी", "સવાઈ")
R15 per-pack suffix stripping one pack's case endings can no
longer inflect another's head;
head must stay >= 3 chars
R16 verify.should_drop_loose_symbol_unit 1-2 char symbol unit must be
glued to its digits ("1996 k")
R17 verify.should_drop_unknown_unit UNIT_ surface that is not a
lexicon form nor within edit
distance 1 of one, next to a
digits-only coefficient
R18 decode boundary trimming trailing year ("दस हज़ार 1987")
and leading digits restatement
("5 panch hajar")
`ambiguous_forms` is a new LanguagePack field, exported (with
`blocked_surfaces`) to src/data/lexicon.json so both runtimes see it.
- python/tests/test_wild_rules.py and test/wild-rules.test.ts: the same cases in both runtimes, named after the real sentence each rule fixes. - python/tests/test_eval_long_lines.py: a 200-character gold line (with the amount past the 128-char window) evaluates instead of raising IndexError; make_windows matches src/charset.ts; normalisation is length-preserving on every shipped gold file, so gold offsets carry over. - tests/test_gates.py: strict FP rate on the wild gold <= 0.02 (actual 4/240 = 0.0167), overall wild FP <= 0.05 (actual 10/240 = 0.0417), and strict value accuracy on real text == 1.0. - tests/test_gold_wild.py: non-gating per-language wild table (value acc, strict coverage, FP, strict FP) printed under -s.
README.md: 'what counts as a span' plus R11-R18 with the real sentence behind each; R18 next to the incomplete-amount merge it constrains. python/README.md: the same convention, the >128-char windowing fix, the new pack fields (ambiguous_forms, blocked_surfaces at runtime), and the fact that real-text PRECISION is gated while accuracy floors are not. data_wild/REPORT.md: a section 6 with the before/after table, what each rule fixed, and the four verified false positives that still need a model change rather than a lexicon one.
…ision rules R11–R18, evaluator windowing, real-text release gates
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First measurement on real text, and the precision work it demanded. No weights change.
Measurement (
python -m sankhya.wild,python/data_wild/REPORT.md)Openly licensed sources (Dakshina Wikipedia sentences hi/mr/gu and romanised Hindi, CMU Hinglish DoG chat; all CC BY-SA, attributed in
python/tests/GOLD_WILD_SOURCES.md). Amounts appear in ~1% of real sentences, so precision is the metric that matters in the wild. 400 hand-labelled real lines (100 per language) intests/gold_wild_*.jsonl.Failure shapes: unit words inside names ("अण्णा हजारे"), the ethnonym "अरब"/"અરબ" → 1e9, Hinglish clitics ("k", "so", "sath"), digits followed by a non-unit ("3 hours" → 3000), lone prefixes, years merged into amounts.
Precision rules R11–R18 (both runtimes, 85 new tests, parity on 1,148 fixture rows)
Ambiguous-form gating (pack
ambiguous_forms, exported to the JS lexicon), blocked proper nouns in verifier and decoder, lone-prefix drop, bare cardinals only when exactly verified, per-pack suffix stripping with a 3-char head, glued symbol units, unknown unit word next to digits within edit distance 1 of a lexicon form or dropped, year/restated-digit trimming at span boundaries. Span convention documented ("what counts as a span").Evaluator
eval_goldnow windows long texts exactly like the JS runtime (128-char windows, 16 overlap) instead of raising past 128 chars, and decodes on normalised text likeparse()does. Fixtures regenerated from all 8 gold files.Gates
New release gates on real text: strict FP ≤ 2% (actual 1.7%), FP ≤ 5% (4.2%), strict value accuracy == 1.0.
Remaining verified FPs (4/240) need the model, not rules: "mil" as miles, English "teen", "हजारों-हजार". A model-side experiment (real negatives in training, masked-character pretraining) is running separately.
Test plan
python -m pytest tests/377 passing; all 6 gates passnpm test215 passing,npm run build🤖 Generated with Claude Code
https://claude.ai/code/session_014m4us3Nx2Z2X5b7WbAw4iR
Generated by Claude Code