From 3a24cf09ac7cfbafe543e34dff4defce723afd72 Mon Sep 17 00:00:00 2001 From: i Date: Tue, 25 Aug 2026 17:56:47 -0400 Subject: [PATCH 1/2] Clarify inclusion learning and browsing context --- README.md | 5 +- docs/architecture.md | 12 +- docs/developer-workbench.md | 5 + docs/inference-and-learning.md | 48 +- docs/personal-workbench.md | 31 +- docs/prefetch-and-reading.md | 15 +- docs/renderer-interface.md | 4 +- docs/resource-constrained-rendering.md | 5 + docs/tab-categorization.md | 12 +- experiments/category-hyperplanes/README.md | 106 ++++ experiments/category-hyperplanes/probe.py | 662 +++++++++++++++++++++ 11 files changed, 876 insertions(+), 29 deletions(-) create mode 100644 experiments/category-hyperplanes/README.md create mode 100755 experiments/category-hyperplanes/probe.py diff --git a/README.md b/README.md index bf4db52..75ce748 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ An experimental personal browser and task-workbench substrate built around durab IB's immediate target is one person's real browsing corpus and workflows, not general-purpose web compatibility. It optimizes the task behind navigation: learning a documentation set, recovering a fact, finding and sharing an image, comparing delivered prices, or resuming an investigation after the live browser processes are gone. -The substrate supports multiple frontends over the same browser-owned state. A conventional page surface, a small phone frontend, a text-and-action workbench, and developer inspectors may coexist. Renderers, acquisition adapters, extractors, and models remain replaceable; none owns tabs, history, tasks, or accepted organization. +The first two user frontends share the same browser-owned state: a visual frontend that immediately pre-paints the cheapest useful source-backed view, and a ChatGPT-like text-only-by-default task frontend that answers questions and offers actions over the browsing corpus. Additional page surfaces and developer inspectors may coexist later. Renderers, acquisition adapters, extractors, and models remain replaceable; none owns tabs, history, tasks, or accepted organization. The browser core owns resource, tab, event, and task identity; sleeping and waking; snapshots; organization; indexes; inference acceptance; and renderer selection. Only roughly 3–10 renderer working sets should normally be resident even when the known corpus reaches 10,000 resources. @@ -14,9 +14,10 @@ The browser core owns resource, tab, event, and task identity; sleeping and waki - `docs/personal-workbench.md` — personal scope, task frontend, user stories, and latency targets - `docs/prefetch-and-reading.md` — durable investigation frontiers, disposable fetches, and `~/reading` - `docs/tab-categorization.md` — overlapping personal categories and adaptive refinement -- `docs/inference-and-learning.md` — local-model proposals, validation, ensembles, and correction events +- `docs/inference-and-learning.md` — configured-model proposals, explicit hyperplanes, ensembles, and human supervision - `docs/storage-model.md` — identity levels and canonical, proposed, and derived state - `docs/developer-workbench.md` — fixture and memory-pressure harness +- `experiments/category-hyperplanes/README.md` — disposable embedding and explicit affine-separator probe ## Implementation languages diff --git a/docs/architecture.md b/docs/architecture.md index d45dc42..23c738d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -6,7 +6,7 @@ A tab is not a renderer process. It is a persistent navigation thread that may currently have a renderer attached. A task may span several tabs, resources, searches, and actions. -IB's present product target is a personal browser/workbench, not universal web compatibility. The substrate still supports multiple first-class frontends and renderer adapters so a broader browser can be built over it without owning or changing the stored model. See `docs/personal-workbench.md`. +IB's present product target is a personal browser/workbench, not universal web compatibility. Its first two first-class frontends are a progressively augmented visual pre-paint and a ChatGPT-like text-only-by-default task workbench. The substrate still permits additional frontends and renderer adapters so a broader browser can be built over it without owning or changing the stored model. See `docs/personal-workbench.md`. ## Ownership @@ -31,16 +31,16 @@ A frontend projects browser and task state and issues commands. It does not beco ## Main layers ```text -page frontend task workbench inspector/commands - \ | / - +--------- browser and task core -----+ +visual pre-paint text task frontend inspector/commands + \ | / + +----------- browser and task core -------+ / | \ persistent store acquisition renderer adapters / extraction | | | HTTP, parsers Servo WebView text/etc. ``` -The page frontend, text-first task workbench, inspector, information extractor, and text renderer are distinct roles. In particular, a text-oriented renderer is not the ChatGPT-like workbench frontend. +The visual pre-paint frontend, text-first task frontend, inspector, information extractor, and text renderer are distinct roles. In particular, a text-oriented renderer is not the ChatGPT-like workbench frontend. The persistent store remains intelligible and useful without a rendering engine or language model installed. @@ -139,7 +139,7 @@ IB is implemented in Idriç, with Grease for operating-system and process orches The current work does not promise: - universal web, MIME, renderer, or malformed-input compatibility; -- one mandatory frontend; +- one singular frontend that owns browser state; - faithful reproduction of interfaces irrelevant to the user's task; - preserving a JavaScript heap across renderer changes; - automatic understanding of every private application protocol; diff --git a/docs/developer-workbench.md b/docs/developer-workbench.md index c5c1949..0a74344 100644 --- a/docs/developer-workbench.md +++ b/docs/developer-workbench.md @@ -100,6 +100,8 @@ Other required distinctions include: - a malformed or unavailable model cannot mutate canonical history or block browsing; - adding a category membership does not remove another membership; - removing a category from `_active` creates no negative training event. +- a focus-priority hint may reorder safe prefetch work but creates no speculative tab or renderer; +- the configured assistant receives only the explicitly scoped, inspectable task-context bundle. If RAM grows approximately with known-resource count, or rebuilding a derived view loses a human correction, the architecture has coupled state classes that must remain separate. @@ -113,3 +115,6 @@ If RAM grows approximately with known-resource count, or rebuilding a derived vi 6. Add operative-document-link and shared-child documentation fixtures. 7. Add proposal, validation, correction, and reversible materialization fixtures. 8. Continue live or recorded scientific-media fixtures through Grease. +9. Add a GitLab-shaped seventeen-link fixture: changing visual focus reprioritizes safe links, creates zero speculative tabs or renderers, and supports a cited text answer. +10. Add a multi-paper arXiv fixture: early per-paper summaries and one cross-paper answer require no renderer per paper. +11. Add a mock video fixture: captions and playback position enter an authorized assistant context bundle without fetching video bytes or exposing secrets. diff --git a/docs/inference-and-learning.md b/docs/inference-and-learning.md index be60d5d..41bdaac 100644 --- a/docs/inference-and-learning.md +++ b/docs/inference-and-learning.md @@ -1,6 +1,6 @@ # Inference and learning boundary -IB should interface cleanly with one or more local language or embedding models without allowing a probabilistic model to become the owner of browser state. +IB should interface cleanly with user-configured local or remote language, embedding, reranking, and summarization models without allowing a probabilistic model or provider to become the owner of browser state. ## Observations and proposals, not mutations @@ -46,13 +46,22 @@ Deterministic code may validate, compare, reduce, accept, or reject proposals. V When several models or resampled classifiers are useful, preserve their individual outputs before reducing them. Bagging, voting, or rank aggregation should not erase disagreement. An aggregate records its inputs, quorum, reducer, and reducer version. A missing model is not a negative vote, and disagreement is itself evidence that a category boundary or ranking is uncertain. -The adapter should identify the model and task explicitly so local models can be replaced, compared, or run as an ensemble without changing canonical records or callers. Browsing must remain usable when every model is absent, slow, or crashes. Local models receive only explicitly selected corpus material; secret and session storage are excluded by default. +The adapter should identify the model and task explicitly so models can be replaced, compared, or run as an ensemble without changing canonical records or callers. Browsing must remain usable when every model is absent, slow, or crashes. Local models receive only task-relevant corpus material. Remote models receive only an explicitly scoped, inspectable context export; secret and session storage and unrelated private browsing state are excluded by default. ## Classification baseline -Category membership is multilabel. A useful first baseline is one scored binary classifier per category rather than a single exclusive multiclass classifier. +Category membership is multilabel. A useful first baseline is one independent inclusion scorer per category rather than a single exclusive multiclass classifier. It is binary relevance only in the sense of asking one category question at a time; material outside category `c` is not automatically a negative example for `c`. -A linear baseline can use one separator per category over persistent Float32 embeddings and cheap structured features. Train from explicit or trusted positive and negative evidence; where absence is merely unlabeled, use a positive-unlabeled treatment rather than declaring every other object negative. Useful features include: +For category `c`, an explicit linear baseline is an affine score and a separately recorded policy threshold: + +```text +s_c(x) = w_c dot phi(x) + b_c +propose c when s_c(x) >= tau_c +``` + +`b_c` is normally learned, so the decision surface is affine and is not forced through the origin. `tau_c` need not be zero: it should reflect the cost of hiding relevant material. There is no argmax across categories. An authoritative human membership remains included and an authoritative category-scoped exclusion remains excluded regardless of a later model score; passing the threshold is still a proposal, not silent acceptance. + +Train from explicit or trusted positive and negative evidence; where absence is merely unlabeled, use a positive-unlabeled treatment rather than declaring every other object negative. Useful features include: - URL, host, title, MIME type, and source; - extracted text or image description; @@ -61,13 +70,32 @@ A linear baseline can use one separator per category over persistent Float32 emb - prior accepted memberships and explicit corrections; - neighborhood or vector similarity. -The literal classifier remains replaceable. A margin is a score, not an ontology. Slack, support examples, and disagreement among plausible separators should remain inspectable where they help explain uncertainty. +In a soft-margin SVM, each labeled example has a scalar slack variable measuring violation of the desired margin. The collection of those scalars may be called a slack vector. Support vectors are instead the training examples with nonzero dual weight that determine the separator; some lie on the margin and some violate it. The signed geometric distance to the fitted zero surface is `s_c(x) / norm(w_c)`; distance to the model-policy threshold surface is `(s_c(x) - tau_c) / norm(w_c)`. Raw score, normalized distance, slack, support-vector status, held-out error, and ensemble disagreement are separate diagnostics; none is automatically a calibrated probability or an inclusion band. + +One-class SVM is an origin-related construction that separates examples from the feature-space origin with an offset. It is a possible positive-only probe, not the ordinary soft-margin binary SVM and not the default once explicit negative corrections exist. + +IB may fit another affine separator inside a coherent region or against residual errors from an earlier separator. That yields a collection of binary decisions—possibly an oblique tree, a boosted ensemble, or overlapping category scorers—not a compulsory single hierarchy. A parent and a narrower category may both remain true. + +Bagging may fit planes over row, feature, or provisional-unlabeled resamples and retain every plane before voting or averaging; this is a useful positive-unlabeled baseline. Boosting may fit later learners against earlier errors. A sum of unthresholded linear scores collapses algebraically to one linear score, while thresholded-plane voting or tree structure can represent a more elaborate boundary. Vote fraction still is not automatically a probability. Zero, one, or many categories may pass their per-category decisions. Retrieval should generally prefer an extra plausible membership to hiding material because another category won. -## Corrections are training events +## Human organization is supervision + +Machine learning should learn from intentional human organization, not merely from corrections made after a bad proposal. + +- creating or naming a category supplies category semantics; +- adding membership supplies an authoritative category-scoped positive; +- removing membership supplies an authoritative negative for that category only; +- accepting a split or merge supplies a structural constraint; +- grouping resources into a task or reading bundle supplies relationship and ranking evidence; +- explicitly meaningful pinning or ordering may supply attention evidence. + +Each signal retains its original event, target kind, scope, and authority instead of being flattened into a universal label. Incidental filesystem order, passive visibility, `_active` removal, and unaccepted model output are not negative classification evidence. + +## Corrections and assertions are training events -A drag, drop, rename, membership addition, or membership removal is an explicit human correction. Record the correction as an event and update derived models conservatively; do not overwrite the model proposal that prompted it. +A drag, drop, rename, membership addition, membership removal, or accepted structural change is a typed human assertion. Record the event and update derived models conservatively; do not overwrite the model proposal that prompted it. Dropping an object into category `B` is an authoritative positive assertion for `B` and changes that view immediately. It is an add, not a move: existing membership in `A` remains because categories overlap. Explicitly removing `B` is negative evidence only for `B`. Removing `B` from `_active` is attention control and produces no classification or training event. Never train on a model's own unaccepted labels. @@ -81,7 +109,7 @@ A history-analysis view may ask: *what is this a record of about the user?* It c Those are evidence-backed interpretations, not facts about identity or belief. Every interpretation should retain links to the searches, visits, tabs, or accepted categories that support it. A repeated URL or duplicate tab is evidence of salience or revisitation, not redundant noise and not by itself proof of endorsement. -Private source material and credentials remain subject to the storage and export boundaries even when inference runs locally. A public fixture should not acquire private URLs merely because a model could classify them. +Private source material and credentials remain subject to the storage and export boundaries whether inference is local or remote. A public fixture should not acquire private URLs merely because a model could classify them. ## Implementation status @@ -91,7 +119,7 @@ The generic proposal, validation, aggregation, and correction records described Settled boundaries: -- local-model adapters are replaceable; +- local and remote model adapters are replaceable and provider-neutral; - model output is append-only evidence, never direct canonical mutation; - proposals retain provenance, model identity, scores, and source references; - deterministic code owns validation and acceptance; @@ -101,7 +129,7 @@ Settled boundaries: Current baselines and evaluation ideas: - Float32 embeddings and exact vector search at the current 10,000-URL scale; -- one scored linear decision per category using explicit or trusted labels; +- one explicit affine inclusion score and a separately recorded threshold per category using positive, explicit-negative, and unlabeled evidence correctly; - ensembles when disagreement is useful; - measurable improvement after a few nearby corrections. diff --git a/docs/personal-workbench.md b/docs/personal-workbench.md index d70ebf8..403b0df 100644 --- a/docs/personal-workbench.md +++ b/docs/personal-workbench.md @@ -4,7 +4,12 @@ IB's immediate target is a personal browser: a workbench over one person's durable browsing corpus, tasks, and habits. It is not presently a general-purpose compatibility browser and should not spend its effort reproducing every web input or every site's preferred interface. -The architecture must nevertheless support multiple first-class frontends over the same browser-owned state. A conventional page-oriented browser, a small phone surface, and a ChatGPT-like text-and-action workbench may coexist. The text-first workbench is **a frontend**, not **the frontend**, and no frontend becomes the owner of tabs, history, tasks, or saved material. +IB's first two first-class user frontends are: + +1. a visual browser frontend that immediately pre-paints the cheapest useful source-backed representation, then progressively augments it or attaches a live renderer when needed; and +2. a ChatGPT-like, text-only-by-default task frontend that answers questions and offers direct actions over the same browsing corpus. + +These are two projections of the same browser-owned tabs, tasks, history, organization, and source material. Neither frontend owns that state, and the architecture may support additional frontends later. Renderer adapters and capability negotiation remain first-class architecture. That makes a broader compatibility browser possible without making broad compatibility IB's current product requirement. An unsupported or irrelevant input may be marked unknown or discarded after a safe failure. MIME type, source, and renderer requirements remain useful signals even when IB does not support the input itself. @@ -33,7 +38,13 @@ A task record may retain: This is a conceptual record, not a frozen schema. -## One text-first frontend +## Visual pre-paint frontend + +The visual frontend should paint useful extracted material as soon as it exists. A page may begin as title, headings, text, operative links, forms, tables, and already-fetched images, then gain richer styling or a live renderer only when the task needs them. Pre-paint is a user-facing frontend behavior, not merely a cache artifact or developer diagnostic. + +The current focus region is also task evidence. Safe operative links at or near its top are strong short-lived candidates for bounded background acquisition; changing focus reprioritizes work without manufacturing tabs. + +## Text-only task frontend The ChatGPT-like frontend should keep a searchable text history of requests, evidence, results, and actions. Its default output is the smallest useful combination for the current task, not a faithful miniature of a corporate webpage. @@ -66,6 +77,20 @@ The depth and priority policy remain open. The invariant is that the durable tas A deterministic fixture should keep the root document usable while a bounded set of early safe links from its operative content becomes warm. Selecting a warmed target should add no predictable foreground network wait and should not require a renderer session for every warmed document. +## Acceptance story: GitLab or a comparable software page + +Given a project, README, documentation, issue, or source page with roughly seventeen plausible operative links, IB should not require the user to open seventeen background tabs and manually wade back and forth through them. It records those links as task candidates, warms a bounded safe subset, extracts and summarizes useful material in the background, and lets the user ask, “just tell me what I need to know,” against the accumulated source-backed corpus. + +A discovered candidate is not a tab. A navigation thread or renderer is created only when interaction actually requires one. The same behavior applies to a batch of arXiv links: the question motivating the browsing may be asked before every paper has been opened and manually read. + +## Direct assistant context + +IB should expose a provider-neutral, user-authorized task-context interface to the person's configured assistant. The user should not have to copy URLs, open every candidate tab, or manually restate what they are browsing. A context bundle may include the current resource and navigation thread, visible and selected source spans, the current task question, roots and frontier, extracted views, retained summaries, and source provenance. + +For an actively watched YouTube video or comparable media resource, the bundle should include available title, channel, chapters, captions or transcript, and current playback position without requiring an automatic full-media download. For a batch of arXiv links, it should include the task-linked papers, their extracted text, and current summaries. + +The automatic low-cost summarizer and the user's preferred interactive assistant are separate configurable roles, although one model may fill both. A remote provider receives only the explicitly scoped context bundle; credentials, session storage, and unrelated private browsing state remain excluded. Context assembly and export are visible, inspectable actions governed by user policy, not a hidden exfiltration path. + ## Acceptance story: find and send an image For a query such as finding a young Larry Wall in a loud 1970s shirt to show a friend, success is not merely displaying image-search results. The requested terminal action is sending a suitable image. @@ -90,8 +115,10 @@ Settled boundaries: - personal-browser priorities first; - multiple frontends over shared browser-owned state; +- visual pre-paint and text-only task interaction as the first two frontends; - task completion rather than page reproduction as the success measure; - durable task intent separated from disposable renderer and response caches; +- a scoped provider-neutral bridge from the active browsing task to the configured assistant; - safe failure for unsupported inputs rather than general compatibility work. Current heuristics and benchmarks: diff --git a/docs/prefetch-and-reading.md b/docs/prefetch-and-reading.md index 151529b..6aa72da 100644 --- a/docs/prefetch-and-reading.md +++ b/docs/prefetch-and-reading.md @@ -88,6 +88,8 @@ For a documentation or research task, an acquisition adapter may discover and pr The operative part is not simply the first links in the HTML or body. It is the relevant document content after global navigation, branding, and unrelated chrome have been excluded. +Within safe operative content, links at or near the top of the current visual-focus region are strong short-lived prefetch candidates. The visual frontend updates that priority as focus or scrolling changes so useful work can begin before a click. “Top” means the current laid-out viewport or focus region, not the first links in HTML. Global navigation, hidden links, logout controls, form actions, and token-bearing URLs remain excluded. A focus hint may cause a safe discovered link to become queued under the active prefetch budget; it does not imply that bytes already exist, manufacture a tab, or erase durable source edges. + Expansion remains bounded by explicit depth, bytes, storage, CPU, network, and task budgets. It warms response or extraction state on disk; it does not create a live renderer session for every target and it is not a promise to crawl arbitrary sites. ## Programming-documentation behavior @@ -96,10 +98,14 @@ If fifty saved documentation tabs belong to one documentation task and each cont When several roots link to the same page, keep one reusable resource plus all source edges. Continue bounded background fetching and extraction while the foreground remains usable. A later question should search and combine the already accumulated material with per-source provenance instead of requiring the user to open each link again. +A GitLab project, README, issue, source page, or comparable software page may expose roughly seventeen plausible operative links. Those are candidates in one task frontier, not an instruction to create seventeen background tabs. IB should warm a bounded subset, extract useful facts and code spans, produce early orientation summaries, and let the text frontend answer “just tell me what I need to know” from the accumulated evidence. An interactive tab is created only when the requested action requires one. + After a week and a reboot, the browser may have lost every cached response and renderer process. It should still know the roots, discovered frontier, completed work, and missing work. Resume or refetch according to current policy rather than starting the investigation from nothing. Summaries and rankings are versioned derived artifacts keyed to immutable source-representation hashes. They retain page and section references, generator or model identity, prompt or policy revision, creation time, completeness, and staleness. A new result appends or supersedes; it does not overwrite the source or the prior result. A page summary may be reused by several tasks, while a cross-page synthesis remains task-specific. +A compact orientation summary is an intended early pre-paint artifact. Once enough operative text has arrived, the lowest-cost configured model satisfying the summary policy may run without waiting for a complete page or complete investigation. Later bytes may append or supersede that proposal. Extraction and navigation remain usable when no model is configured or the model fails. + For a resumable investigation, the accepted or last-complete task summary—or its complete raw proposal—plus source references is durable enough to show immediately after a week and a reboot. Presentation indexes and intermediate or rejected attempts remain disposable. The task, user selections and corrections, and source graph are durable browser state. ## Renderer path @@ -132,10 +138,12 @@ For an arXiv identifier, the reading layout begins as: image_tags/ ``` -The HTML and external figures may also remain in the disposable response cache. The files under `~/reading` are the semantic corpus: a person or local model can query the paper text, captions, image manifest, and figure files without depending on a live renderer cache. +The HTML and external figures may also remain in the disposable response cache. The files under `~/reading` are the semantic corpus: a person or configured model can query the paper text, captions, image manifest, and figure files without depending on a live renderer cache. If usable arXiv HTML is unavailable, the existing scientific-media policy may fall back to PDF. +A batch of arXiv links encountered during one research task is one investigation frontier, not a request for one tab or renderer per paper. Within task budgets, IB may acquire HTML and explicit figures, produce compact per-paper orientation summaries as text becomes available, and answer or synthesize across the batch with paper-and-span provenance. The user should be able to ask the motivating question before manually opening and inspecting every paper. + ## Test surfaces The fixture harness should expose these independently: @@ -153,13 +161,16 @@ A regression should clear response and extraction caches, reconstruct the same t Minimum deterministic cases should also prove: - header, navigation, form, logout, and token links do not outrank safe links in operative documentation content; +- changing the visual-focus region reprioritizes safe operative links without creating tabs or renderers; - two roots targeting one child yield one compatible fetch and two retained edges; - a stronger later retention claim is not lost; - killing the process after queueing or during a fetch never exposes a torn body/metadata pair, and pending work resumes; - a fake-clock return one week later shows the last complete view immediately and revalidates stale material in the background; - relative URLs, ``, fragments, redirects, and incompatible auth/`Vary` partitions preserve the correct identity distinctions; - a changed source marks the old summary stale and appends a new source-backed version; -- duplicate tab intentions and visit events survive while compatible response bytes remain shared. +- duplicate tab intentions and visit events survive while compatible response bytes remain shared; +- a GitLab-shaped page with seventeen operative links yields no speculative tabs or renderers and can produce a cited answer from bounded background work; +- an arXiv batch yields early per-paper summaries and one cross-paper answer without a renderer per paper. ## Commitment level diff --git a/docs/renderer-interface.md b/docs/renderer-interface.md index c0ee665..fe69039 100644 --- a/docs/renderer-interface.md +++ b/docs/renderer-interface.md @@ -4,7 +4,7 @@ A renderer is a replaceable live-page implementation behind a browser-owned inte The contract should be narrow enough that Servo, Chromium/WebView, a text-oriented engine, or a future renderer can coexist without becoming the source of truth for browser state. -A renderer is not a frontend. A page frontend may attach one, while a task frontend may complete useful work through acquisition and extraction without attaching any renderer. A text-oriented page renderer is likewise not the ChatGPT-like task workbench. +A renderer is not a frontend. The visual pre-paint frontend may attach one, while the text task frontend may complete useful work through acquisition and extraction without attaching any renderer. A text-oriented page renderer is likewise not the ChatGPT-like task frontend. ## Core responsibilities @@ -36,6 +36,8 @@ renderer_failed The core decides what becomes durable. +The visual frontend may also send ephemeral visual-focus and visible-operative-link hints to the acquisition scheduler. These hints can reprioritize bounded safe work as the user scrolls or changes focus. A focus hint may cause a safe discovered link to become queued under the active prefetch budget; it does not itself imply that bytes already exist, create a tab, or bypass unsafe-link policy. Raw layout geometry need not become canonical browsing state. + ## Conceptual interface ```text diff --git a/docs/resource-constrained-rendering.md b/docs/resource-constrained-rendering.md index 3c4363a..aab6369 100644 --- a/docs/resource-constrained-rendering.md +++ b/docs/resource-constrained-rendering.md @@ -39,6 +39,8 @@ Useful early material includes: The first view may be incomplete. Better information may replace or augment it without losing task, source, tab, or history identity. +A compact source-backed orientation summary is intended early pre-paint, not optional finishing polish. Once enough operative text has arrived, IB should schedule the lowest-cost configured model that satisfies the summary policy rather than wait for the complete page or investigation. Later source material may append or supersede the proposal. Extraction remains usable when the model is absent or fails, and every summary retains source spans and model provenance. + ### arXiv integration demonstration `bin/prepaint_arxiv_progressively.grease` joins the scientific-media fetch boundary to the Idriç information projection without starting a heavyweight renderer. It selects arXiv HTML first and gives the information core progressive 4, 16, and 32 KiB prefixes. @@ -94,6 +96,8 @@ The constrained path may suppress or defer work not needed for the task: JavaScript is an escalation, not an assumption. If source bytes or a structured response already supply the needed information, JavaScript need not run merely because the site normally expects it. +Deferring autoplay or full video bytes does not discard task-relevant media context. For an actively watched video, title, channel, chapters, available captions or transcript, and current playback position may remain available to the task and configured assistant without prefetching the media stream. + An unsupported input may safely remain unknown, be discarded, or be handed to an available full renderer. Its existence does not create a general compatibility obligation for IB. ## Relation to renderer capabilities @@ -127,6 +131,7 @@ It must not remove durable resources, tabs, history events, task frontiers, user 7. Cached extracted view: pre-paint immediately and revalidate in the background. 8. Cache cleared: durable task and history survive and the view can be regenerated. 9. Renderer absent or crashed: extracted results remain usable. +10. Active video context: expose transcript metadata and playback position without fetching video bytes or exporting secrets. ## Non-goals for the first slices diff --git a/docs/tab-categorization.md b/docs/tab-categorization.md index 6ad0fdc..2a4be68 100644 --- a/docs/tab-categorization.md +++ b/docs/tab-categorization.md @@ -12,17 +12,17 @@ This is a personal taxonomy learned from one person's corpus. Fixture-derived ca ## Classification is multilabel -Membership is represented as non-exclusive per-category scores or proposals rather than one leaf in a tree. Conceptually, each category gets its own decision surface over tab, resource, event, and task features. A simple baseline may use one binary separator per category, trained from explicit or trusted positive and negative evidence and treated as positive-unlabeled where negatives are missing. Richer or ensembled classifiers can replace it without changing the contract. This does not assert that categories or scores are statistically independent, and a signed margin is not automatically a calibrated probability. +Membership is represented as non-exclusive per-category scores or proposals rather than one leaf in a tree. Conceptually, each category gets its own inclusion surface over tab, resource, event, and task features. A simple baseline may use one affine score `s_c(x) = w_c dot phi(x) + b_c` and separately recorded inclusion threshold `tau_c` per category. It learns from positive, explicit-negative, and unlabeled evidence without treating every object outside the category as negative. Richer or ensembled classifiers can replace it without changing the contract. This does not assert that categories or scores are statistically independent. The central semantic question is roughly: why is this here? Purpose categories such as repair, research, ordinary curiosity, morbid curiosity, applying, scheduling, transacting, reading, and acquiring are useful signals and memberships. They are not gating branches through which every object must descend. -An object may land on the positive side of several decisions at once. A small signed margin means the decision is fragile under that fitted classifier; it is not by itself a probability or full epistemic uncertainty. Slack examples, repeated fits, and disagreement among plausible separators can expose additional instability without demanding one winner. +An object may pass several inclusion thresholds at once. For a soft-margin SVM, slack variables measure labeled examples' margin violations; support vectors are the examples that determine the fitted separator. Normalized geometric distance, held-out errors, and disagreement among resampled planes are useful but different diagnostics. None is by itself a probability or full epistemic uncertainty. Missing membership is normally unlabeled, not an automatic negative example. Explicit removal supplies negative evidence for that category only. -“CART-like” is only an analogy for adaptive refinement. When one region contains a coherent recurring subcluster, IB may propose an additional narrower category with its own decision surface. Members need not leave the broader category, and the broader category may remain useful. This is not a CART tree and not a single hierarchy. +When one region contains a coherent recurring subcluster, IB may fit another affine separator to that region or to residual errors from an earlier separator. Several planes may be retained as an oblique tree, bagged vote, boosted ensemble, or independent overlapping category scorers. Members need not leave the broader category, and the broader category may remain useful. This is not necessarily CART and not a compulsory single hierarchy. -Classifier output is evidence for zero, one, or many memberships. Prefer a useful extra membership to making material disappear from retrieval because one supposedly exclusive classification won. +Classifier output is evidence for zero, one, or many memberships. High-recall proposal and retrieval policy should prefer a useful extra candidate to making material disappear because one supposedly exclusive classification won. A proposal does not silently become accepted durable membership. ## Do not conflate identity levels @@ -131,7 +131,7 @@ The pass separated, among other things, book acquisition from book reading or re A Brecht listing can still belong simultaneously to `book-shopping`, `Brecht`, and `theatre`; a mathematical book can belong to an acquisition view and several mathematical subjects. The exercise supports overlapping refinement precisely because the forced partition loses useful memberships. -If a reproducible labeled fixture is later derived from that pass, it should retain explicit corrections rather than plausible guesses from names alone: `cruz.jp` was kids'-toy shopping, and Daniel Litt's “Problems I Like” was a broken mathematics site. +If a reproducible labeled fixture is later derived from that pass, it should retain explicit corrections rather than plausible guesses from names alone: the fixture URL `crux.jp` (earlier transcribed as `cruz.jp`) was kids'-toy shopping, and Daniel Litt's “Problems I Like” was a broken mathematics site. Two ingestion lessons are settled: @@ -158,7 +158,7 @@ Settled boundaries: Current heuristics and baselines: - roughly 5–10 relevant objects as a category-promotion signal; -- one scored binary classifier per category using explicit or trusted labels; +- one affine inclusion scorer and separate threshold per category using positives, explicit negatives, and unlabeled material correctly; - over-inclusion when the alternative is failed retrieval; - `_active` as a filesystem-shaped working-set control. diff --git a/experiments/category-hyperplanes/README.md b/experiments/category-hyperplanes/README.md new file mode 100644 index 0000000..89a6c3e --- /dev/null +++ b/experiments/category-hyperplanes/README.md @@ -0,0 +1,106 @@ +# Category hyperplane probe + +This is a disposable comparison outside the IB runtime. It does not make Python a browser implementation layer. Its purpose is to test frozen embedding models and explicit affine category separators before reproducing the selected policy in Idriç, Grease, or a narrow native adapter. + +## Question + +Given human organization and an embedding matrix, can independent inclusion-oriented planes recover useful accepted memberships without forcing one winning category? + +For category `c`, each fitted plane records: + +```text +s_c(x) = w_c dot x + b_c +``` + +The experiment also records a proposal-policy threshold separately from the fitted SVM zero surface. Several categories may propose the same row. Human assertions are never reversed by the probe, and a candidate produced by a model never silently becomes accepted durable membership. + +## Candidate embedding models + +Start with meaningfully different small encoders rather than treating one model as ground truth: + +| Candidate | Output | Initial role | +| --- | ---: | --- | +| [`mixedbread-ai/mxbai-embed-xsmall-v1`](https://huggingface.co/mixedbread-ai/mxbai-embed-xsmall-v1) | 384, with Matryoshka truncation | default; Apache-2.0; official INT8 ONNX is roughly 24 MB | +| [`intfloat/e5-small-v2`](https://huggingface.co/intfloat/e5-small-v2) | 384 | independent linear-probe challenger; MIT; prefix inputs consistently with `query: ` | +| [`sentence-transformers/static-retrieval-mrl-en-v1`](https://huggingface.co/sentence-transformers/static-retrieval-mrl-en-v1) | 1024, truncatable to 512/256/128 | compute-speed floor; Apache-2.0; its roughly 125 MB lookup table is not the smallest memory footprint, and token-vector averaging loses word order | +| [`Snowflake/snowflake-arctic-embed-xs`](https://huggingface.co/Snowflake/snowflake-arctic-embed-xs) | 384 | optional browser-oriented challenger; Apache-2.0; CLS pooling and query prefix policy matter | + +The first run should use float normalized output vectors even if model weights are INT8. Binary output-vector quantization can wait until the float decision surfaces are understood. + +Pin and record the exact model revision, files and checksums, backend, weight precision, tokenizer, input prefix, pooling, maximum token count, truncation side and strategy, output dimension, input-field grammar, and normalization. A model name alone is not reproducible provenance. A weak-device comparison should normally begin with a fixed 128- or 256-token input budget even when the encoder permits more. + +Canonical input text should use stable labeled fields such as decoded host and path, title, and source-backed snippet. Do not silently fetch private pages merely to improve an experiment. + +## Input + +`probe.py` expects a NumPy archive containing: + +- `ids`: a one-dimensional string array of stable resource, tab, or event ids; +- `vectors`: a two-dimensional float array in the same order. + +It also requires the exact UTF-8 input-text artifact used to create those vectors. This is a two-column TSV, in the same row order as the archive: + +```text +id text +``` + +The probe verifies the ids and records the artifact hash plus the immutable input-builder revision. This makes cross-model comparisons use identical text rather than merely similar preprocessing descriptions. + +The labels file is UTF-8 TSV with this header: + +```text +id category polarity authority +``` + +`polarity` is `positive` or `negative`. `authority` must be `human_assertion` or `accepted_decision`. A missing row is unlabeled, not negative. The same id may have positive membership in several categories. A separate file with the same grammar may be supplied through `--evaluation-labels`; those rows are excluded from fitting and provisional-negative sampling. + +The current repository does not contain the complete accepted labels from the conversational 491-row exercise or the earlier 227-row reading/non-reading split. Do not reconstruct them from domain stereotypes. Exact replication begins only when the ordered rows, stable identities, accepted memberships, and explicit corrections are available together. + +## Run + +```text +python3 experiments/category-hyperplanes/probe.py \ + --vectors /path/to/embeddings.npz \ + --input-texts /path/to/canonical-inputs.tsv \ + --labels /path/to/accepted-labels.tsv \ + --evaluation-labels /path/to/held-out-labels.tsv \ + --model-id mixedbread-ai/mxbai-embed-xsmall-v1 \ + --model-revision <40-character-commit> \ + --model-file-sha256 onnx/model_quantized.onnx= \ + --backend onnxruntime \ + --backend-version \ + --weight-precision int8 \ + --tokenizer-revision <40-character-commit> \ + --tokenizer-sha256 \ + --pooling mean \ + --input-prefix '' \ + --input-grammar 'host: path: title: text: <snippet>' \ + --input-builder-revision <40-character-commit> \ + --max-input-tokens 256 \ + --truncation-side right \ + --truncation-strategy longest_first \ + --truncation-dimension 384 \ + --output /tmp/ib-category-probe.json +``` + +The default requests bagged positive-unlabeled SVMs. Each retained plane has a unique provisional-unlabeled sample and preserves its explicit `w`, `b`, support examples, and margin violations before reduction. If no resampling diversity is possible, the probe fits one plane instead of presenting repeated identical fits as an ensemble. Explicit human negatives are always included in the training set; sampled unlabeled rows are provisional negatives for that plane only and never become assertions. The report preserves the untouched model proposal separately from a candidate/retrieval view after authoritative positive and negative overrides. Neither view mutates accepted durable membership. + +Run the network-free in-memory contract check with: + +```text +python3 experiments/category-hyperplanes/probe.py --self-test +``` + +## Evaluation + +Do not report training fit as reproduction. Construct the separate evaluation-label file by navigation thread, resource, domain, or time so repeated URLs cannot leak across train and test. Evaluation rows are excluded globally from every category's fitting and provisional-negative pool. A category with assertions but no fitted positive set is rejected rather than silently omitted. The probe reports held-out positive recall and held-out explicit-negative false-proposal rate on the raw model proposal; it then applies those held-out assertions when showing the candidate/retrieval view. Report at least: + +- recall and false omissions for held-out human positives; +- false proposals on explicit human negatives; +- recovered overlap, rather than forced primary-label accuracy; +- each plane's support examples and margin violations; +- examples nearest each model-policy proposal boundary; +- disagreement across bags and embedding models; +- the effect of two or three new human assertions without erasing older categories. + +Vote fraction, fitted margin, normalized distance, and calibrated probability are different quantities. Preserve them separately. diff --git a/experiments/category-hyperplanes/probe.py b/experiments/category-hyperplanes/probe.py new file mode 100755 index 0000000..f7bdb8c --- /dev/null +++ b/experiments/category-hyperplanes/probe.py @@ -0,0 +1,662 @@ +#!/usr/bin/env python3 +"""Disposable explicit-hyperplane comparison for IB category design.""" + +from __future__ import annotations + +import argparse +import csv +import hashlib +import itertools +import json +import math +import platform +import re +from dataclasses import dataclass +from pathlib import Path +from typing import Iterable + +import numpy as np +import sklearn +from sklearn.svm import SVC + + +ALLOWED_AUTHORITIES = {"human_assertion", "accepted_decision"} +ALLOWED_POLARITIES = {"positive", "negative"} +SHA256_PATTERN = re.compile(r"[0-9a-f]{64}") +REVISION_PATTERN = re.compile(r"[0-9a-f]{40}") + + +@dataclass(frozen=True) +class Label: + row_id: str + category: str + polarity: str + authority: str + + +def sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + return digest.hexdigest() + + +def parse_model_file_checksums(values: list[str]) -> dict[str, str]: + result: dict[str, str] = {} + for value in values: + name, separator, checksum = value.rpartition("=") + if not separator or not name or not SHA256_PATTERN.fullmatch(checksum): + raise ValueError("--model-file-sha256 must be NAME=64-lowercase-hex-digits") + if name in result: + raise ValueError(f"duplicate model-file checksum name {name!r}") + result[name] = checksum + return dict(sorted(result.items())) + + +def normalize_rows(vectors: np.ndarray) -> np.ndarray: + vectors = np.asarray(vectors, dtype=np.float32) + if vectors.ndim != 2 or vectors.shape[0] == 0 or vectors.shape[1] == 0: + raise ValueError("vectors must be a nonempty two-dimensional matrix") + if not np.isfinite(vectors).all(): + raise ValueError("vectors contain a non-finite value") + norms = np.linalg.norm(vectors, axis=1, keepdims=True) + if np.any(norms == 0): + raise ValueError("vectors contain a zero row") + return vectors / norms + + +def load_vectors(path: Path) -> tuple[list[str], np.ndarray]: + with np.load(path, allow_pickle=False) as archive: + if set(archive.files) != {"ids", "vectors"}: + raise ValueError("vector archive must contain exactly ids and vectors") + ids = [str(value) for value in archive["ids"].tolist()] + vectors = normalize_rows(archive["vectors"]) + if len(ids) != vectors.shape[0] or len(ids) != len(set(ids)): + raise ValueError("ids must be unique and align one-to-one with vectors") + return ids, vectors + + +def validate_input_texts(path: Path, expected_ids: list[str]) -> None: + actual_ids: list[str] = [] + with path.open("r", encoding="utf-8", newline="") as stream: + reader = csv.DictReader(stream, delimiter="\t") + expected_header = ["id", "text"] + if reader.fieldnames != expected_header: + raise ValueError(f"input-text header must be {expected_header!r}") + for line_number, row in enumerate(reader, start=2): + if not row["id"] or not row["text"]: + raise ValueError(f"input-text line {line_number}: empty id or text") + actual_ids.append(row["id"]) + if actual_ids != expected_ids: + raise ValueError("input-text ids must exactly match vector ids and row order") + + +def load_labels(path: Path, known_ids: set[str]) -> list[Label]: + labels: list[Label] = [] + seen: set[tuple[str, str]] = set() + with path.open("r", encoding="utf-8", newline="") as stream: + reader = csv.DictReader(stream, delimiter="\t") + expected = ["id", "category", "polarity", "authority"] + if reader.fieldnames != expected: + raise ValueError(f"labels header must be {expected!r}") + for line_number, row in enumerate(reader, start=2): + label = Label(row["id"], row["category"], row["polarity"], row["authority"]) + if not label.row_id or not label.category: + raise ValueError(f"labels line {line_number}: empty id or category") + if label.row_id not in known_ids: + raise ValueError(f"labels line {line_number}: unknown id {label.row_id!r}") + if label.polarity not in ALLOWED_POLARITIES: + raise ValueError(f"labels line {line_number}: invalid polarity") + if label.authority not in ALLOWED_AUTHORITIES: + raise ValueError(f"labels line {line_number}: invalid authority") + key = (label.row_id, label.category) + if key in seen: + raise ValueError(f"labels line {line_number}: duplicate category assertion for id") + seen.add(key) + labels.append(label) + if not labels: + raise ValueError("labels file is empty") + return labels + + +def choose_threshold( + scores: np.ndarray, positive_indices: np.ndarray, target_recall: float +) -> float: + positive_scores = np.sort(scores[positive_indices]) + allowed_misses = int(math.floor((1.0 - target_recall) * len(positive_scores))) + allowed_misses = min(max(allowed_misses, 0), len(positive_scores) - 1) + return float(positive_scores[allowed_misses] - 1e-7) + + +def fit_category( + category: str, + ids: list[str], + vectors: np.ndarray, + labels: list[Label], + evaluation_labels: list[Label], + global_evaluation_indices: set[int], + bags: int, + unlabeled_per_positive: float, + c_value: float, + positive_weight: float, + target_positive_recall: float, + seed: int, +) -> dict: + index = {row_id: offset for offset, row_id in enumerate(ids)} + positives = np.array( + sorted( + index[label.row_id] + for label in labels + if label.category == category and label.polarity == "positive" + ), + dtype=np.int64, + ) + negatives = np.array( + sorted( + index[label.row_id] + for label in labels + if label.category == category and label.polarity == "negative" + ), + dtype=np.int64, + ) + if len(positives) < 2: + raise ValueError(f"category {category!r} needs at least two authoritative positives") + + evaluation_positives = np.array( + sorted( + index[label.row_id] + for label in evaluation_labels + if label.category == category and label.polarity == "positive" + ), + dtype=np.int64, + ) + evaluation_negatives = np.array( + sorted( + index[label.row_id] + for label in evaluation_labels + if label.category == category and label.polarity == "negative" + ), + dtype=np.int64, + ) + positive_ids = set(positives.tolist()) + negative_ids = set(negatives.tolist()) + evaluation_positive_ids = set(evaluation_positives.tolist()) + evaluation_negative_ids = set(evaluation_negatives.tolist()) + fit_ids = positive_ids | negative_ids + held_out_ids = evaluation_positive_ids | evaluation_negative_ids + if fit_ids & held_out_ids: + raise ValueError(f"category {category!r} has an id in both fit and evaluation labels") + + fixed = fit_ids | global_evaluation_indices + unlabeled = np.array( + [offset for offset in range(len(ids)) if offset not in fixed], dtype=np.int64 + ) + if len(negatives) == 0 and len(unlabeled) == 0: + raise ValueError(f"category {category!r} has no negative or unlabeled comparison rows") + + requested_count = int(math.ceil(len(positives) * unlabeled_per_positive)) + sample_count = min(len(unlabeled), requested_count) + if len(negatives) == 0 and sample_count == 0: + raise ValueError( + f"category {category!r} needs an explicit negative or a positive " + "unlabeled sampling ratio" + ) + + rng = np.random.default_rng(seed) + if sample_count == 0: + provisional_samples = [tuple()] + elif sample_count == len(unlabeled): + provisional_samples = [tuple(unlabeled.tolist())] + else: + possible_samples = math.comb(len(unlabeled), sample_count) + retained_count = min(bags, possible_samples) + if possible_samples <= bags: + provisional_samples = list(itertools.combinations(unlabeled.tolist(), sample_count)) + rng.shuffle(provisional_samples) + else: + unique_samples: set[tuple[int, ...]] = set() + while len(unique_samples) < retained_count: + sample = rng.choice(unlabeled, size=sample_count, replace=False) + unique_samples.add(tuple(sorted(sample))) + provisional_samples = sorted(unique_samples) + + normalized_scores: list[np.ndarray] = [] + planes: list[dict] = [] + + for bag_number, provisional_values in enumerate(provisional_samples): + provisional = np.asarray(provisional_values, dtype=np.int64) + bag_negatives = np.unique(np.concatenate([negatives, provisional])) + train_indices = np.concatenate([positives, bag_negatives]) + y = np.concatenate([np.ones(len(positives)), -np.ones(len(bag_negatives))]) + + classifier = SVC( + C=c_value, + kernel="linear", + class_weight={1.0: positive_weight, -1.0: 1.0}, + ) + classifier.fit(vectors[train_indices], y) + w = np.asarray(classifier.coef_[0], dtype=np.float64) + b = float(classifier.intercept_[0]) + w_norm = float(np.linalg.norm(w)) + if not math.isfinite(w_norm) or w_norm == 0: + raise ValueError(f"category {category!r} bag {bag_number}: degenerate plane") + + raw_all = vectors @ w + b + normalized_scores.append(raw_all / w_norm) + raw_train = raw_all[train_indices] + slack = np.maximum(0.0, 1.0 - y * raw_train) + support_indices = train_indices[classifier.support_] + violations = np.flatnonzero(slack > 1e-9) + largest = violations[np.argsort(slack[violations])[::-1][:10]] if len(violations) else [] + + planes.append( + { + "bag": bag_number, + "w": w.tolist(), + "b": b, + "w_norm": w_norm, + "positive_ids": [ids[offset] for offset in positives], + "explicit_negative_ids": [ids[offset] for offset in negatives], + "provisional_unlabeled_ids": [ids[offset] for offset in provisional], + "support_ids": [ids[offset] for offset in support_indices], + "slack_nonzero_count": int(len(violations)), + "largest_margin_violations": [ + {"id": ids[int(train_indices[offset])], "slack": float(slack[offset])} + for offset in largest + ], + } + ) + + score_matrix = np.vstack(normalized_scores) + aggregate = score_matrix.mean(axis=0) + votes = (score_matrix >= 0.0).mean(axis=0) + threshold = choose_threshold(aggregate, positives, target_positive_recall) + model_proposed = aggregate >= threshold + candidate_included = model_proposed.copy() + asserted_positive_ids = positive_ids | evaluation_positive_ids + asserted_negative_ids = negative_ids | evaluation_negative_ids + candidate_included[list(asserted_positive_ids)] = True + candidate_included[list(asserted_negative_ids)] = False + positive_overrides = np.zeros(len(ids), dtype=bool) + negative_overrides = np.zeros(len(ids), dtype=bool) + positive_overrides[list(asserted_positive_ids)] = ~model_proposed[list(asserted_positive_ids)] + negative_overrides[list(asserted_negative_ids)] = model_proposed[list(asserted_negative_ids)] + + ranking = np.argsort(aggregate)[::-1] + nearest = np.argsort(np.abs(aggregate - threshold))[: min(20, len(ids))] + + def recall(indices: np.ndarray) -> float | None: + return float(model_proposed[indices].mean()) if len(indices) else None + + def false_proposal_rate(indices: np.ndarray) -> float | None: + return float(model_proposed[indices].mean()) if len(indices) else None + + return { + "category": category, + "positive_count": int(len(positives)), + "explicit_negative_count": int(len(negatives)), + "fit_eligible_unlabeled_count": int(len(unlabeled)), + "requested_plane_count": int(bags), + "retained_unique_plane_count": int(len(planes)), + "provisional_unlabeled_per_plane": int(sample_count), + "fit_mode": ( + "bagged linear soft-margin SVM with provisional-unlabeled resampling" + if len(planes) > 1 + else "single linear soft-margin SVM; no provisional-resample diversity" + ), + "evaluation_positive_count": int(len(evaluation_positives)), + "evaluation_negative_count": int(len(evaluation_negatives)), + "aggregate_score_meaning": "mean signed geometric distance across retained planes", + "inclusion_threshold": threshold, + "target_training_positive_recall": target_positive_recall, + "in_sample_model_positive_recall_before_override": recall(positives), + "candidate_asserted_positive_inclusion_recall": float( + candidate_included[list(asserted_positive_ids)].mean() + ), + "candidate_asserted_negative_exclusion_recall": float( + (~candidate_included[list(asserted_negative_ids)]).mean() + ) if asserted_negative_ids else None, + "explicit_negative_false_proposal_rate": false_proposal_rate(negatives), + "held_out_positive_recall": recall(evaluation_positives), + "held_out_negative_false_proposal_rate": false_proposal_rate(evaluation_negatives), + "planes": planes, + "ranking": [ + { + "id": ids[offset], + "score": float(aggregate[offset]), + "zero_surface_vote_fraction": float(votes[offset]), + "model_proposed": bool(model_proposed[offset]), + "candidate_included_after_assertion_overrides": bool(candidate_included[offset]), + "human_positive_override_applied": bool(positive_overrides[offset]), + "human_negative_override_applied": bool(negative_overrides[offset]), + "fit_positive": bool(offset in positive_ids), + "fit_negative": bool(offset in negative_ids), + "evaluation_positive": bool(offset in evaluation_positive_ids), + "evaluation_negative": bool(offset in evaluation_negative_ids), + } + for offset in ranking + ], + "nearest_model_policy_boundary": [ + {"id": ids[offset], "score": float(aggregate[offset])} for offset in nearest + ], + } + + +def run_probe(args: argparse.Namespace) -> dict: + vector_path = Path(args.vectors) + input_text_path = Path(args.input_texts) + label_path = Path(args.labels) + ids, vectors = load_vectors(vector_path) + validate_input_texts(input_text_path, ids) + labels = load_labels(label_path, set(ids)) + evaluation_path = Path(args.evaluation_labels) if args.evaluation_labels else None + evaluation_labels = load_labels(evaluation_path, set(ids)) if evaluation_path else [] + fit_row_ids = {label.row_id for label in labels} + evaluation_row_ids = {label.row_id for label in evaluation_labels} + overlap = fit_row_ids & evaluation_row_ids + if overlap: + raise ValueError( + "fit and evaluation labels must be row-disjoint; overlapping ids: " + + ", ".join(sorted(overlap)) + ) + id_to_index = {row_id: offset for offset, row_id in enumerate(ids)} + global_evaluation_indices = {id_to_index[row_id] for row_id in evaluation_row_ids} + model_files = parse_model_file_checksums(args.model_file_sha256) + if vectors.shape[1] != args.truncation_dimension: + raise ValueError( + f"vector dimension {vectors.shape[1]} does not match --truncation-dimension " + f"{args.truncation_dimension}" + ) + known_dimensions = { + "mixedbread-ai/mxbai-embed-xsmall-v1": {128, 256, 384}, + "intfloat/e5-small-v2": {384}, + "sentence-transformers/static-retrieval-mrl-en-v1": {128, 256, 512, 1024}, + "Snowflake/snowflake-arctic-embed-xs": {384}, + } + allowed = known_dimensions.get(args.model_id) + if allowed is not None and vectors.shape[1] not in allowed: + raise ValueError(f"dimension {vectors.shape[1]} is not declared for {args.model_id}") + categories = sorted({label.category for label in labels if label.polarity == "positive"}) + all_fit_categories = {label.category for label in labels} + missing_positive_categories = all_fit_categories - set(categories) + if missing_positive_categories: + raise ValueError( + "fit labels contain categories without a fitted positive set: " + + ", ".join(sorted(missing_positive_categories)) + ) + evaluation_categories = {label.category for label in evaluation_labels} + missing_fit_categories = evaluation_categories - set(categories) + if missing_fit_categories: + raise ValueError( + "evaluation labels contain categories without a fitted positive set: " + + ", ".join(sorted(missing_fit_categories)) + ) + return { + "format": "ib-category-hyperplane-probe-v1", + "warning": "in-sample fit is not evidence of held-out reproduction", + "vector_source": { + "path": str(vector_path), + "sha256": sha256(vector_path), + "rows": len(ids), + "dimensions": int(vectors.shape[1]), + "l2_normalized_by_probe": True, + }, + "embedding_input": { + "path": str(input_text_path), + "sha256": sha256(input_text_path), + "rows": len(ids), + "encoding": "UTF-8 TSV", + "input_builder_revision": args.input_builder_revision, + }, + "labels": {"path": str(label_path), "sha256": sha256(label_path)}, + "evaluation_labels": ( + {"path": str(evaluation_path), "sha256": sha256(evaluation_path)} + if evaluation_path + else None + ), + "embedding": { + "model_id": args.model_id, + "model_revision": args.model_revision, + "model_files_sha256": model_files, + "backend": args.backend, + "backend_version": args.backend_version, + "weight_precision": args.weight_precision, + "tokenizer_revision": args.tokenizer_revision, + "tokenizer_sha256": args.tokenizer_sha256, + "pooling": args.pooling, + "input_prefix": args.input_prefix, + "input_grammar": args.input_grammar, + "max_input_tokens": args.max_input_tokens, + "truncation_side": args.truncation_side, + "truncation_strategy": args.truncation_strategy, + "truncation_dimension": args.truncation_dimension, + }, + "probe_runtime": { + "python": platform.python_version(), + "numpy": np.__version__, + "scikit_learn": sklearn.__version__, + }, + "fit": { + "kind": ( + "linear soft-margin SVM planes with category-specific " + "provisional-unlabeled resampling" + ), + "requested_planes_per_category": args.bags, + "C": args.c_value, + "positive_weight": args.positive_weight, + "unlabeled_per_positive": args.unlabeled_per_positive, + "seed": args.seed, + }, + "categories": [ + fit_category( + category, + ids, + vectors, + labels, + evaluation_labels, + global_evaluation_indices, + args.bags, + args.unlabeled_per_positive, + args.c_value, + args.positive_weight, + args.target_positive_recall, + args.seed + offset * 1009, + ) + for offset, category in enumerate(categories) + ], + } + + +def self_test() -> None: + ids = ["a", "b", "c", "d", "overlap", "denied-a", "denied-b", "outside"] + vectors = normalize_rows( + np.array( + [ + [3.0, 0.2, 1.0], + [2.8, -0.1, 1.0], + [0.2, 3.0, 1.0], + [-0.1, 2.8, 1.0], + [2.4, 2.4, 1.0], + [3.0, 0.2, 1.0], + [0.2, 3.0, 1.0], + [-2.0, -2.0, 1.0], + ], + dtype=np.float32, + ) + ) + labels = [ + Label("a", "A", "positive", "human_assertion"), + Label("b", "A", "positive", "human_assertion"), + Label("outside", "A", "negative", "human_assertion"), + Label("c", "B", "positive", "human_assertion"), + Label("d", "B", "positive", "human_assertion"), + Label("outside", "B", "negative", "human_assertion"), + ] + evaluation_labels = [ + Label("overlap", "A", "positive", "accepted_decision"), + Label("overlap", "B", "positive", "accepted_decision"), + Label("denied-a", "A", "negative", "accepted_decision"), + Label("denied-b", "B", "negative", "accepted_decision"), + ] + global_evaluation_indices = { + ids.index("overlap"), + ids.index("denied-a"), + ids.index("denied-b"), + } + results = { + category: fit_category( + category, + ids, + vectors, + labels, + evaluation_labels, + global_evaluation_indices, + 3, + 0.0, + 1.0, + 2.0, + 1.0, + 7, + ) + for category in ("A", "B") + } + for category, result in results.items(): + assert result["in_sample_model_positive_recall_before_override"] == 1.0 + assert result["held_out_positive_recall"] == 1.0 + assert result["held_out_negative_false_proposal_rate"] == 1.0 + assert result["candidate_asserted_negative_exclusion_recall"] == 1.0 + assert result["fit_eligible_unlabeled_count"] == 2 + assert result["retained_unique_plane_count"] == 1 + assert all(not plane["provisional_unlabeled_ids"] for plane in result["planes"]) + assert any(abs(plane["b"]) > 1e-9 for plane in result["planes"]), category + by_id = {row["id"]: row for row in result["ranking"]} + assert by_id["overlap"]["model_proposed"], category + assert not by_id["overlap"]["human_positive_override_applied"], category + denied_id = "denied-a" if category == "A" else "denied-b" + assert by_id[denied_id]["model_proposed"], category + assert not by_id[denied_id]["candidate_included_after_assertion_overrides"], category + assert by_id[denied_id]["human_negative_override_applied"], category + + resampled = fit_category( + "A", + ids, + vectors, + labels, + evaluation_labels, + global_evaluation_indices, + 10, + 0.5, + 1.0, + 2.0, + 1.0, + 11, + ) + assert resampled["requested_plane_count"] == 10 + assert resampled["retained_unique_plane_count"] == 2 + provisional_sets = { + tuple(plane["provisional_unlabeled_ids"]) for plane in resampled["planes"] + } + assert provisional_sets == {("c",), ("d",)} + print("category hyperplane self-test: ok") + + +def parser() -> argparse.ArgumentParser: + result = argparse.ArgumentParser(description=__doc__) + result.add_argument("--self-test", action="store_true") + result.add_argument("--vectors") + result.add_argument("--input-texts") + result.add_argument("--labels") + result.add_argument("--evaluation-labels") + result.add_argument("--output") + result.add_argument("--model-id") + result.add_argument("--model-revision") + result.add_argument("--model-file-sha256", action="append") + result.add_argument("--backend") + result.add_argument("--backend-version") + result.add_argument("--weight-precision") + result.add_argument("--tokenizer-revision") + result.add_argument("--tokenizer-sha256") + result.add_argument("--pooling") + result.add_argument("--input-prefix") + result.add_argument("--input-grammar") + result.add_argument("--input-builder-revision") + result.add_argument("--max-input-tokens", type=int) + result.add_argument("--truncation-side", choices=("left", "right")) + result.add_argument("--truncation-strategy") + result.add_argument("--truncation-dimension", type=int) + result.add_argument("--bags", type=int, default=16) + result.add_argument("--unlabeled-per-positive", type=float, default=2.0) + result.add_argument("--c-value", type=float, default=1.0) + result.add_argument("--positive-weight", type=float, default=2.0) + result.add_argument("--target-positive-recall", type=float, default=1.0) + result.add_argument("--seed", type=int, default=1729) + return result + + +def main(argv: Iterable[str] | None = None) -> int: + args = parser().parse_args(argv) + if args.self_test: + self_test() + return 0 + required = ( + "vectors", + "input_texts", + "labels", + "output", + "model_id", + "model_revision", + "model_file_sha256", + "backend", + "backend_version", + "weight_precision", + "tokenizer_revision", + "tokenizer_sha256", + "pooling", + "input_prefix", + "input_grammar", + "input_builder_revision", + "max_input_tokens", + "truncation_side", + "truncation_strategy", + "truncation_dimension", + ) + missing = [name for name in required if getattr(args, name) is None] + if missing: + arguments = ", ".join("--" + name for name in missing) + raise SystemExit("missing required arguments: " + arguments) + if ( + args.bags < 1 + or args.unlabeled_per_positive < 0 + or args.c_value <= 0 + or args.positive_weight <= 0 + ): + raise SystemExit("invalid fit parameter") + if not 0 < args.target_positive_recall <= 1: + raise SystemExit("--target-positive-recall must be in (0, 1]") + if args.truncation_dimension < 1: + raise SystemExit("--truncation-dimension must be positive") + if args.max_input_tokens < 1: + raise SystemExit("--max-input-tokens must be positive") + if not REVISION_PATTERN.fullmatch(args.model_revision): + raise SystemExit("--model-revision must be an immutable 40-character lowercase commit id") + if not REVISION_PATTERN.fullmatch(args.tokenizer_revision): + raise SystemExit( + "--tokenizer-revision must be an immutable 40-character lowercase commit id" + ) + if not REVISION_PATTERN.fullmatch(args.input_builder_revision): + raise SystemExit( + "--input-builder-revision must be an immutable 40-character lowercase commit id" + ) + if not SHA256_PATTERN.fullmatch(args.tokenizer_sha256): + raise SystemExit("--tokenizer-sha256 must be 64 lowercase hexadecimal characters") + report = run_probe(args) + Path(args.output).write_text( + json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8" + ) + print(args.output) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From 7ff3b3dd1e6de7faecb998b1ce08f192d091bfe9 Mon Sep 17 00:00:00 2001 From: i <isomorphisms@sdf.org> Date: Tue, 25 Aug 2026 19:16:37 -0400 Subject: [PATCH 2/2] Recover authoritative category assertions --- bin/ci_browser_foundation.grease | 8 ++ experiments/category-hyperplanes/README.md | 39 +++++- .../category-hyperplanes/build_url_inputs.py | 56 +++++++++ experiments/category-hyperplanes/probe.py | 119 +++++++++++++++--- .../recovered-authoritative-assertions.tsv | 14 +++ .../recovered-reading-evaluation.tsv | 3 + .../recovered-reading-fit.tsv | 4 + .../validate_recovered_labels.py | 107 ++++++++++++++++ 8 files changed, 326 insertions(+), 24 deletions(-) create mode 100644 experiments/category-hyperplanes/build_url_inputs.py create mode 100644 experiments/category-hyperplanes/recovered-authoritative-assertions.tsv create mode 100644 experiments/category-hyperplanes/recovered-reading-evaluation.tsv create mode 100644 experiments/category-hyperplanes/recovered-reading-fit.tsv create mode 100644 experiments/category-hyperplanes/validate_recovered_labels.py diff --git a/bin/ci_browser_foundation.grease b/bin/ci_browser_foundation.grease index 105a255..6eba5d3 100755 --- a/bin/ci_browser_foundation.grease +++ b/bin/ci_browser_foundation.grease @@ -60,6 +60,14 @@ exercise_workbench() { cd "$repository_root" sh -n tests/test_real_world_url_fixture.grease sh tests/test_real_world_url_fixture.grease + python3 -m py_compile \ + experiments/category-hyperplanes/build_url_inputs.py \ + experiments/category-hyperplanes/validate_recovered_labels.py + python3 experiments/category-hyperplanes/validate_recovered_labels.py + python3 experiments/category-hyperplanes/build_url_inputs.py \ + --source tests/fixtures/real_world_urls.txt \ + --output /tmp/ib-recovered-inputs.tsv + test "$(wc -l < /tmp/ib-recovered-inputs.tsv)" = 220 cd "$repository_root/src" "$idric_prefix/bin/idris2" Workbench.idric -o ib-workbench 2>&1 | tee /tmp/idric-workbench-compile.txt diff --git a/experiments/category-hyperplanes/README.md b/experiments/category-hyperplanes/README.md index 89a6c3e..f00845f 100644 --- a/experiments/category-hyperplanes/README.md +++ b/experiments/category-hyperplanes/README.md @@ -49,21 +49,48 @@ The probe verifies the ids and records the artifact hash plus the immutable inpu The labels file is UTF-8 TSV with this header: ```text -id category polarity authority +id category polarity authority asserted_at assertion_source source_artifact_sha256 source_row ``` `polarity` is `positive` or `negative`. `authority` must be `human_assertion` or `accepted_decision`. A missing row is unlabeled, not negative. The same id may have positive membership in several categories. A separate file with the same grammar may be supplied through `--evaluation-labels`; those rows are excluded from fitting and provisional-negative sampling. -The current repository does not contain the complete accepted labels from the conversational 491-row exercise or the earlier 227-row reading/non-reading split. Do not reconstruct them from domain stereotypes. Exact replication begins only when the ordered rows, stable identities, accepted memberships, and explicit corrections are available together. +Every label retains the user-assertion time and locator plus the checksum and one-based row in the object-source artifact. The report copies this provenance instead of retaining only the label-file checksum. `--category` may be repeated to fit a deliberate subset of categories while preserving one-off assertions separately. + +## Recovered prior assertions + +The recovery audit did **not** promote either old assistant output to ground truth: + +- The 491-row primary-purpose partition was assistant-generated, initially listed only 478 rows, and was later reconciled by adding rows 37–44 and 46–50 to its AbeBooks run. The count repair does not turn the classifications into human assertions. +- The 227-row reading/non-reading output was also assistant-generated. Conversation evidence establishes its reported 129/98 counts and shows that files named `Reading Links.txt` and `Non Reading Links.txt` existed, but their complete bytes and row-by-row provenance were not recovered. +- A later 98-row grouped output was assistant-generated too. Only explicit user corrections from its review were recovered as authoritative. + +`recovered-authoritative-assertions.tsv` is therefore deliberately small. It contains only explicit user assignments that could be tied to an exact URL: Problems I Like and the Rezchikov visualization as reading and short reading; Benn Peifert as reading and long-form self-education; Haaretz as news/reading; The Decent One as morbid-curiosity reading; the exact AbeBooks basket as reading; Open Syllabus as Black Ball; and `crux.jp` as shopping. Compound statements are retained as overlapping positive memberships. No omitted row becomes a negative, and no assignment is generalized to neighboring URLs on the same domain. + +The sanitized fixture has 219 rows, not the complete private 227-row source. Its exact bytes are pinned by SHA-256 in `build_url_inputs.py` and every probe label. Stable ids include the source row so duplicate URLs remain separate. Build the URL-only canonical input artifact with: + +```text +python3 experiments/category-hyperplanes/build_url_inputs.py \ + --source tests/fixtures/real_world_urls.txt \ + --output /tmp/ib-recovered-inputs.tsv +``` + +Five recovered reading positives occur in that pinned fixture. `recovered-reading-fit.tsv` contains three and `recovered-reading-evaluation.tsv` contains two row-disjoint held-out positives. This is enough to execute a positive-unlabeled `reading` probe, but it is not enough to claim reproduction of the 227- or 491-row exercises. The one recovered Black Ball assertion, one shopping assertion, and the AbeBooks basket absent from the sanitized fixture remain in the audit ledger rather than being padded into fit sets. + +Validate the ledger, source-row joins, and split with: + +```text +python3 experiments/category-hyperplanes/validate_recovered_labels.py +``` ## Run ```text python3 experiments/category-hyperplanes/probe.py \ --vectors /path/to/embeddings.npz \ - --input-texts /path/to/canonical-inputs.tsv \ - --labels /path/to/accepted-labels.tsv \ - --evaluation-labels /path/to/held-out-labels.tsv \ + --input-texts /tmp/ib-recovered-inputs.tsv \ + --labels experiments/category-hyperplanes/recovered-reading-fit.tsv \ + --evaluation-labels experiments/category-hyperplanes/recovered-reading-evaluation.tsv \ + --category reading \ --model-id mixedbread-ai/mxbai-embed-xsmall-v1 \ --model-revision <40-character-commit> \ --model-file-sha256 onnx/model_quantized.onnx=<sha256> \ @@ -74,7 +101,7 @@ python3 experiments/category-hyperplanes/probe.py \ --tokenizer-sha256 <sha256> \ --pooling mean \ --input-prefix '' \ - --input-grammar 'host: <host> path: <path> title: <title> text: <snippet>' \ + --input-grammar 'url: <url>' \ --input-builder-revision <40-character-commit> \ --max-input-tokens 256 \ --truncation-side right \ diff --git a/experiments/category-hyperplanes/build_url_inputs.py b/experiments/category-hyperplanes/build_url_inputs.py new file mode 100644 index 0000000..baf7209 --- /dev/null +++ b/experiments/category-hyperplanes/build_url_inputs.py @@ -0,0 +1,56 @@ +#!/usr/bin/env python3 +"""Build pinned URL-only probe inputs from the sanitized browsing fixture.""" + +from __future__ import annotations + +import argparse +import csv +import hashlib +from pathlib import Path + + +EXPECTED_SOURCE_SHA256 = "4c71746f994ef02959cef9baf9787a74c60e6fe51309d096e6398d56bef0c850" +EXPECTED_ROWS = 219 +ID_PREFIX = "real-world-20260825" + + +def sha256(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def row_id(line_number: int) -> str: + return f"{ID_PREFIX}-{line_number:06d}" + + +def build(source: Path, output: Path) -> None: + checksum = sha256(source) + if checksum != EXPECTED_SOURCE_SHA256: + raise ValueError( + f"source checksum {checksum} does not match pinned {EXPECTED_SOURCE_SHA256}" + ) + urls = source.read_text(encoding="utf-8").splitlines() + if len(urls) != EXPECTED_ROWS: + raise ValueError(f"source has {len(urls)} rows, expected {EXPECTED_ROWS}") + if any(not url or "\t" in url for url in urls): + raise ValueError("source contains an empty row or tab") + + output.parent.mkdir(parents=True, exist_ok=True) + with output.open("w", encoding="utf-8", newline="") as stream: + writer = csv.writer(stream, delimiter="\t", lineterminator="\n") + writer.writerow(("id", "text")) + for line_number, url in enumerate(urls, start=1): + writer.writerow((row_id(line_number), f"url: {url}")) + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--source", required=True) + parser.add_argument("--output", required=True) + args = parser.parse_args() + build(Path(args.source), Path(args.output)) + print(args.output) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/experiments/category-hyperplanes/probe.py b/experiments/category-hyperplanes/probe.py index f7bdb8c..8e3e52a 100755 --- a/experiments/category-hyperplanes/probe.py +++ b/experiments/category-hyperplanes/probe.py @@ -11,7 +11,7 @@ import math import platform import re -from dataclasses import dataclass +from dataclasses import asdict, dataclass from pathlib import Path from typing import Iterable @@ -32,6 +32,16 @@ class Label: category: str polarity: str authority: str + asserted_at: str + assertion_source: str + source_artifact_sha256: str + source_row: int + + +def label_record(label: Label) -> dict: + result = asdict(label) + result["id"] = result.pop("row_id") + return result def sha256(path: Path) -> str: @@ -97,11 +107,33 @@ def load_labels(path: Path, known_ids: set[str]) -> list[Label]: seen: set[tuple[str, str]] = set() with path.open("r", encoding="utf-8", newline="") as stream: reader = csv.DictReader(stream, delimiter="\t") - expected = ["id", "category", "polarity", "authority"] + expected = [ + "id", + "category", + "polarity", + "authority", + "asserted_at", + "assertion_source", + "source_artifact_sha256", + "source_row", + ] if reader.fieldnames != expected: raise ValueError(f"labels header must be {expected!r}") for line_number, row in enumerate(reader, start=2): - label = Label(row["id"], row["category"], row["polarity"], row["authority"]) + try: + source_row = int(row["source_row"]) + except ValueError as error: + raise ValueError(f"labels line {line_number}: invalid source row") from error + label = Label( + row["id"], + row["category"], + row["polarity"], + row["authority"], + row["asserted_at"], + row["assertion_source"], + row["source_artifact_sha256"], + source_row, + ) if not label.row_id or not label.category: raise ValueError(f"labels line {line_number}: empty id or category") if label.row_id not in known_ids: @@ -110,6 +142,12 @@ def load_labels(path: Path, known_ids: set[str]) -> list[Label]: raise ValueError(f"labels line {line_number}: invalid polarity") if label.authority not in ALLOWED_AUTHORITIES: raise ValueError(f"labels line {line_number}: invalid authority") + if not label.asserted_at or not label.assertion_source: + raise ValueError(f"labels line {line_number}: missing assertion provenance") + if not SHA256_PATTERN.fullmatch(label.source_artifact_sha256): + raise ValueError(f"labels line {line_number}: invalid source artifact checksum") + if label.source_row < 1: + raise ValueError(f"labels line {line_number}: invalid source row") key = (label.row_id, label.category) if key in seen: raise ValueError(f"labels line {line_number}: duplicate category assertion for id") @@ -377,15 +415,28 @@ def run_probe(args: argparse.Namespace) -> dict: allowed = known_dimensions.get(args.model_id) if allowed is not None and vectors.shape[1] not in allowed: raise ValueError(f"dimension {vectors.shape[1]} is not declared for {args.model_id}") - categories = sorted({label.category for label in labels if label.polarity == "positive"}) - all_fit_categories = {label.category for label in labels} + positive_categories = {label.category for label in labels if label.polarity == "positive"} + categories = sorted(set(args.category) if args.category else positive_categories) + unknown_categories = set(categories) - positive_categories + if unknown_categories: + raise ValueError( + "requested categories lack a fitted positive set: " + + ", ".join(sorted(unknown_categories)) + ) + all_fit_categories = { + label.category + for label in labels + if not args.category or label.category in categories + } missing_positive_categories = all_fit_categories - set(categories) if missing_positive_categories: raise ValueError( "fit labels contain categories without a fitted positive set: " + ", ".join(sorted(missing_positive_categories)) ) - evaluation_categories = {label.category for label in evaluation_labels} + evaluation_categories = { + label.category for label in evaluation_labels if label.category in categories + } missing_fit_categories = evaluation_categories - set(categories) if missing_fit_categories: raise ValueError( @@ -409,9 +460,23 @@ def run_probe(args: argparse.Namespace) -> dict: "encoding": "UTF-8 TSV", "input_builder_revision": args.input_builder_revision, }, - "labels": {"path": str(label_path), "sha256": sha256(label_path)}, + "labels": { + "path": str(label_path), + "sha256": sha256(label_path), + "assertions": [ + label_record(label) for label in labels if label.category in categories + ], + }, "evaluation_labels": ( - {"path": str(evaluation_path), "sha256": sha256(evaluation_path)} + { + "path": str(evaluation_path), + "sha256": sha256(evaluation_path), + "assertions": [ + label_record(label) + for label in evaluation_labels + if label.category in categories + ], + } if evaluation_path else None ), @@ -485,19 +550,31 @@ def self_test() -> None: dtype=np.float32, ) ) + def label(row_id: str, category: str, polarity: str, authority: str) -> Label: + return Label( + row_id, + category, + polarity, + authority, + "2026-08-25T00:00:00Z", + "self-test", + "0" * 64, + 1, + ) + labels = [ - Label("a", "A", "positive", "human_assertion"), - Label("b", "A", "positive", "human_assertion"), - Label("outside", "A", "negative", "human_assertion"), - Label("c", "B", "positive", "human_assertion"), - Label("d", "B", "positive", "human_assertion"), - Label("outside", "B", "negative", "human_assertion"), + label("a", "A", "positive", "human_assertion"), + label("b", "A", "positive", "human_assertion"), + label("outside", "A", "negative", "human_assertion"), + label("c", "B", "positive", "human_assertion"), + label("d", "B", "positive", "human_assertion"), + label("outside", "B", "negative", "human_assertion"), ] evaluation_labels = [ - Label("overlap", "A", "positive", "accepted_decision"), - Label("overlap", "B", "positive", "accepted_decision"), - Label("denied-a", "A", "negative", "accepted_decision"), - Label("denied-b", "B", "negative", "accepted_decision"), + label("overlap", "A", "positive", "accepted_decision"), + label("overlap", "B", "positive", "accepted_decision"), + label("denied-a", "A", "negative", "accepted_decision"), + label("denied-b", "B", "negative", "accepted_decision"), ] global_evaluation_indices = { ids.index("overlap"), @@ -568,6 +645,12 @@ def parser() -> argparse.ArgumentParser: result.add_argument("--input-texts") result.add_argument("--labels") result.add_argument("--evaluation-labels") + result.add_argument( + "--category", + action="append", + default=[], + help="fit only this category; repeat for several independent planes", + ) result.add_argument("--output") result.add_argument("--model-id") result.add_argument("--model-revision") diff --git a/experiments/category-hyperplanes/recovered-authoritative-assertions.tsv b/experiments/category-hyperplanes/recovered-authoritative-assertions.tsv new file mode 100644 index 0000000..4e1cc0b --- /dev/null +++ b/experiments/category-hyperplanes/recovered-authoritative-assertions.tsv @@ -0,0 +1,14 @@ +id url category polarity authority asserted_at assertion_source recovery_basis source_artifact_sha256 source_row +real-world-20260825-000076 http://problemsilike.com/ reading positive human_assertion 2026-08-25T16:46:28Z conversation-user-message@2026-08-25T16:46:28Z context-index-explicit-user-constraint 4c71746f994ef02959cef9baf9787a74c60e6fe51309d096e6398d56bef0c850 76 +real-world-20260825-000076 http://problemsilike.com/ intellectual-fun-short-reading positive human_assertion 2026-08-25T16:49:44Z conversation-user-message@2026-08-25T16:49:44Z context-index-explicit-user-constraint 4c71746f994ef02959cef9baf9787a74c60e6fe51309d096e6398d56bef0c850 76 +real-world-20260825-000083 https://www.opensyllabus.org/ black-ball positive human_assertion 2026-08-25T16:46:28Z conversation-user-message@2026-08-25T16:46:28Z context-index-explicit-user-constraint 4c71746f994ef02959cef9baf9787a74c60e6fe51309d096e6398d56bef0c850 83 +real-world-20260825-000089 https://x.com/bennpeifert/status/2004518534471422095 reading positive human_assertion 2026-08-25T16:49:44Z conversation-user-message@2026-08-25T16:49:44Z context-index-explicit-user-constraint 4c71746f994ef02959cef9baf9787a74c60e6fe51309d096e6398d56bef0c850 89 +real-world-20260825-000089 https://x.com/bennpeifert/status/2004518534471422095 long-form-reading-self-education positive human_assertion 2026-08-25T16:49:44Z conversation-user-message@2026-08-25T16:49:44Z context-index-explicit-user-constraint 4c71746f994ef02959cef9baf9787a74c60e6fe51309d096e6398d56bef0c850 89 +real-world-20260825-000102 https://www.thedecentonefilm.com/ reading positive human_assertion 2026-08-25T16:49:44Z conversation-user-message@2026-08-25T16:49:44Z context-index-explicit-user-constraint 4c71746f994ef02959cef9baf9787a74c60e6fe51309d096e6398d56bef0c850 102 +real-world-20260825-000102 https://www.thedecentonefilm.com/ morbid-curiosity positive human_assertion 2026-08-25T16:49:44Z conversation-user-message@2026-08-25T16:49:44Z context-index-explicit-user-constraint 4c71746f994ef02959cef9baf9787a74c60e6fe51309d096e6398d56bef0c850 102 +real-world-20260825-000136 https://crux.jp/ shopping positive human_assertion 2026-08-25T19:23:52Z conversation-user-message@2026-08-25T19:23:52Z verbatim-user-correction 4c71746f994ef02959cef9baf9787a74c60e6fe51309d096e6398d56bef0c850 136 +real-world-20260825-000185 https://promotion.haaretz.com/offers reading positive human_assertion 2026-08-25T16:46:28Z conversation-user-message@2026-08-25T16:46:28Z context-index-explicit-user-constraint 4c71746f994ef02959cef9baf9787a74c60e6fe51309d096e6398d56bef0c850 185 +real-world-20260825-000185 https://promotion.haaretz.com/offers news-reading positive human_assertion 2026-08-25T16:46:28Z conversation-user-message@2026-08-25T16:46:28Z context-index-explicit-user-constraint 4c71746f994ef02959cef9baf9787a74c60e6fe51309d096e6398d56bef0c850 185 +real-world-20260825-000197 https://www.rezchikov.me/jacobian_counterexample_visualization.html reading positive human_assertion 2026-08-25T16:46:28Z conversation-user-message@2026-08-25T16:46:28Z context-index-explicit-user-constraint 4c71746f994ef02959cef9baf9787a74c60e6fe51309d096e6398d56bef0c850 197 +real-world-20260825-000197 https://www.rezchikov.me/jacobian_counterexample_visualization.html intellectual-fun-short-reading positive human_assertion 2026-08-25T16:49:44Z conversation-user-message@2026-08-25T16:49:44Z context-index-explicit-user-constraint 4c71746f994ef02959cef9baf9787a74c60e6fe51309d096e6398d56bef0c850 197 +interactive-98-abebooks-checkout https://www.abebooks.com/checkout/basket reading positive human_assertion 2026-08-25T16:46:28Z conversation-user-message@2026-08-25T16:46:28Z context-index-explicit-user-constraint unavailable unavailable diff --git a/experiments/category-hyperplanes/recovered-reading-evaluation.tsv b/experiments/category-hyperplanes/recovered-reading-evaluation.tsv new file mode 100644 index 0000000..1e1f4e1 --- /dev/null +++ b/experiments/category-hyperplanes/recovered-reading-evaluation.tsv @@ -0,0 +1,3 @@ +id category polarity authority asserted_at assertion_source source_artifact_sha256 source_row +real-world-20260825-000102 reading positive human_assertion 2026-08-25T16:49:44Z conversation-user-message@2026-08-25T16:49:44Z 4c71746f994ef02959cef9baf9787a74c60e6fe51309d096e6398d56bef0c850 102 +real-world-20260825-000197 reading positive human_assertion 2026-08-25T16:46:28Z conversation-user-message@2026-08-25T16:46:28Z 4c71746f994ef02959cef9baf9787a74c60e6fe51309d096e6398d56bef0c850 197 diff --git a/experiments/category-hyperplanes/recovered-reading-fit.tsv b/experiments/category-hyperplanes/recovered-reading-fit.tsv new file mode 100644 index 0000000..822a6c4 --- /dev/null +++ b/experiments/category-hyperplanes/recovered-reading-fit.tsv @@ -0,0 +1,4 @@ +id category polarity authority asserted_at assertion_source source_artifact_sha256 source_row +real-world-20260825-000076 reading positive human_assertion 2026-08-25T16:46:28Z conversation-user-message@2026-08-25T16:46:28Z 4c71746f994ef02959cef9baf9787a74c60e6fe51309d096e6398d56bef0c850 76 +real-world-20260825-000089 reading positive human_assertion 2026-08-25T16:49:44Z conversation-user-message@2026-08-25T16:49:44Z 4c71746f994ef02959cef9baf9787a74c60e6fe51309d096e6398d56bef0c850 89 +real-world-20260825-000185 reading positive human_assertion 2026-08-25T16:46:28Z conversation-user-message@2026-08-25T16:46:28Z 4c71746f994ef02959cef9baf9787a74c60e6fe51309d096e6398d56bef0c850 185 diff --git a/experiments/category-hyperplanes/validate_recovered_labels.py b/experiments/category-hyperplanes/validate_recovered_labels.py new file mode 100644 index 0000000..08894fa --- /dev/null +++ b/experiments/category-hyperplanes/validate_recovered_labels.py @@ -0,0 +1,107 @@ +#!/usr/bin/env python3 +"""Check recovered assertions against their pinned source rows and probe splits.""" + +from __future__ import annotations + +import csv +from pathlib import Path + +from build_url_inputs import EXPECTED_ROWS, EXPECTED_SOURCE_SHA256, row_id, sha256 +ROOT = Path(__file__).resolve().parents[2] +SOURCE = ROOT / "tests/fixtures/real_world_urls.txt" +DIRECTORY = Path(__file__).resolve().parent +LEDGER = DIRECTORY / "recovered-authoritative-assertions.tsv" +FIT = DIRECTORY / "recovered-reading-fit.tsv" +EVALUATION = DIRECTORY / "recovered-reading-evaluation.tsv" +LEDGER_HEADER = [ + "id", + "url", + "category", + "polarity", + "authority", + "asserted_at", + "assertion_source", + "recovery_basis", + "source_artifact_sha256", + "source_row", +] +PROBE_HEADER = [ + "id", + "category", + "polarity", + "authority", + "asserted_at", + "assertion_source", + "source_artifact_sha256", + "source_row", +] + + +def read_tsv(path: Path, expected_header: list[str]) -> list[dict[str, str]]: + with path.open("r", encoding="utf-8", newline="") as stream: + reader = csv.DictReader(stream, delimiter="\t") + if reader.fieldnames != expected_header: + raise ValueError(f"{path}: unexpected header") + return list(reader) + + +def main() -> int: + if sha256(SOURCE) != EXPECTED_SOURCE_SHA256: + raise ValueError("sanitized source fixture checksum changed") + urls = SOURCE.read_text(encoding="utf-8").splitlines() + if len(urls) != EXPECTED_ROWS: + raise ValueError("sanitized source fixture row count changed") + + ledger = read_tsv(LEDGER, LEDGER_HEADER) + by_key: dict[tuple[str, str], dict[str, str]] = {} + for record in ledger: + key = (record["id"], record["category"]) + if key in by_key: + raise ValueError(f"duplicate recovered assertion {key!r}") + by_key[key] = record + if record["polarity"] != "positive" or record["authority"] != "human_assertion": + raise ValueError(f"non-authoritative recovered assertion {key!r}") + if not record["asserted_at"] or not record["assertion_source"]: + raise ValueError(f"missing assertion provenance {key!r}") + if record["source_row"] != "unavailable": + source_row = int(record["source_row"]) + if record["source_artifact_sha256"] != EXPECTED_SOURCE_SHA256: + raise ValueError(f"wrong source checksum for {key!r}") + if record["id"] != row_id(source_row): + raise ValueError(f"wrong stable row id for {key!r}") + if urls[source_row - 1] != record["url"]: + raise ValueError(f"source URL mismatch for {key!r}") + elif record["source_artifact_sha256"] != "unavailable": + raise ValueError(f"unavailable source row has an artifact checksum for {key!r}") + + known_ids = {row_id(number) for number in range(1, EXPECTED_ROWS + 1)} + fit = read_tsv(FIT, PROBE_HEADER) + evaluation = read_tsv(EVALUATION, PROBE_HEADER) + fit_ids = {label["id"] for label in fit} + evaluation_ids = {label["id"] for label in evaluation} + if fit_ids & evaluation_ids: + raise ValueError("fit and evaluation rows overlap") + if len(fit_ids) != 3 or len(evaluation_ids) != 2: + raise ValueError("expected three fit and two evaluation reading assertions") + for label in fit + evaluation: + if label["id"] not in known_ids: + raise ValueError(f"probe label has unknown source id: {label!r}") + record = by_key.get((label["id"], label["category"])) + if record is None: + raise ValueError(f"probe label absent from recovered ledger: {label!r}") + if ( + label["polarity"] != record["polarity"] + or label["authority"] != record["authority"] + or label["asserted_at"] != record["asserted_at"] + or label["assertion_source"] != record["assertion_source"] + or label["source_artifact_sha256"] != record["source_artifact_sha256"] + or label["source_row"] != record["source_row"] + ): + raise ValueError(f"probe label provenance drifted: {label!r}") + + print("recovered authoritative labels: ok") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main())