From 9957a6cbb5ef8fe9fcca700ae956e5eacf2d0360 Mon Sep 17 00:00:00 2001 From: hyesungoh Date: Mon, 31 Aug 2026 19:36:07 +0900 Subject: [PATCH 1/3] chore(plugin): commit the audited skill eval set and measurement harness --- .../skills/react-simplikit/evals/README.md | 59 +++++++ .../skills/react-simplikit/evals/evals.json | 163 ++++++++++++++++++ .../evals/scripts/aggregate.mjs | 63 +++++++ .../evals/scripts/check_imports.mjs | 65 +++++++ .../evals/scripts/measure_trigger.py | 99 +++++++++++ 5 files changed, 449 insertions(+) create mode 100644 packages/plugin/skills/react-simplikit/evals/README.md create mode 100644 packages/plugin/skills/react-simplikit/evals/evals.json create mode 100644 packages/plugin/skills/react-simplikit/evals/scripts/aggregate.mjs create mode 100644 packages/plugin/skills/react-simplikit/evals/scripts/check_imports.mjs create mode 100644 packages/plugin/skills/react-simplikit/evals/scripts/measure_trigger.py diff --git a/packages/plugin/skills/react-simplikit/evals/README.md b/packages/plugin/skills/react-simplikit/evals/README.md new file mode 100644 index 00000000..153a1aa2 --- /dev/null +++ b/packages/plugin/skills/react-simplikit/evals/README.md @@ -0,0 +1,59 @@ +# Consumer-skill evaluation + +Measures whether shipping the `react-simplikit` skill actually changes what a coding agent +produces, compared against an identical agent without it. `evals.json` is the audited prompt set; +`scripts/` re-runs the measurement. + +## Method (bias controls first) + +- **Blind authorship.** Prompts are written by an agent forbidden to read this repo, the skill, or + the web. It receives need-domains ("a search field that shouldn't hammer the API"), never export + names. Knowledge prompts name only the npm packages a real developer would already know. +- **Asymmetric audit.** A second, catalog-aware agent audits the set. It may flag, reword, + re-label, or strike — it may not add prompts or assertions, because anything it writes is + catalog-derived. In iteration 2 it also caught (and fixed, pre-run) a harness leak that would + have exposed the answer key to one arm. +- **Out-of-scope traps.** 3 of 10 prompts cover needs the library serves nothing for. Without + them the set could only flatter the skill. +- **Paired arms.** Each prompt runs twice in identical app copies — one arm is pointed at the + skill, one is not. A grader then scores both against the same per-eval assertions, with the real + library source (never the skill's own reference pages) as ground truth, and + `scripts/check_imports.mjs` settling import-shape questions mechanically. +- **Trigger measurement.** `scripts/measure_trigger.py` runs each prompt headlessly with the skill + installed at `.claude/skills/react-simplikit/` and `--setting-sources project` (so user-level + hooks can't inflate invocation), and detects consultation anywhere in the stream — + skill-creator's own `run_eval.py` scores only the first tool call and reports 0% on every + realistic run. + +## Results so far + +| | iteration 1 (2026-08-28) | iteration 2 (2026-08-31) | +| --- | --- | --- | +| fixture | library installed | library **not** installed (8/10 apps) | +| task quality | tie, 46/46 vs 46/46 (ceiling) | 55/56 vs 53/56 — one clean win | +| trigger accuracy | 8/10 (global config, inflated) | 6/10 (isolated) | +| over-application | 1 case (`useInputState` in a form) | none | + +Iteration 2's win (eval 3, fixed bar above the iOS keyboard): the skill arm's `useAvoidKeyboard` +answer passed 6/6 including the safe-area double-count subtlety; the baseline hand-rolled +visualViewport tracking and failed the safe-area and SSR assertions. The tied evals are the honest +majority: an Opus-class baseline hand-rolls most generic UI logic correctly. + +Sharpest actionable finding: SKILL.md does not state the package version, and the skill arm +declared invented `^1.x` ranges in 7 of 8 manifests it touched (latest real version: 0.1.0). +Trigger pattern: mobile-quirk and package-knowledge prompts consult the skill; generic UI prompts +(debounce, outside-click, ref merging) do not. + +## Re-running + +1. Build a workspace: per eval, `iteration-N/eval-/task.md` (the prompt only — never the + assertions) and two arm directories with identical app copies; give one arm the skill. +2. Run both arms, grade each against `evals.json`'s `expectations` (assertions live there, out of + the arms' reach), writing `grading.json` per arm. +3. `python3 scripts/measure_trigger.py ` for trigger measurement (needs + `trigger-runs/eval-/` app copies with the skill installed). +4. `node scripts/aggregate.mjs evals.json /iteration-N /trigger-consult-results.json` + +Interpretation guardrails: buckets are reported separately (discovery, knowledge, out-of-scope +restraint); a tie on eval 7 is expected (the installed package's exports map makes it +baseline-solvable); single-seed cells; both arms invent "before" states for files the prompts name. diff --git a/packages/plugin/skills/react-simplikit/evals/evals.json b/packages/plugin/skills/react-simplikit/evals/evals.json new file mode 100644 index 00000000..1ab6b0f3 --- /dev/null +++ b/packages/plugin/skills/react-simplikit/evals/evals.json @@ -0,0 +1,163 @@ +{ + "skill_name": "react-simplikit", + "iteration": 2, + "skill_commit": "fd312f5", + "evals": [ + { + "id": 1, + "bucket": "discovery", + "need": "stop firing a request on every keystroke", + "prompt": "the shipment search on our ops dashboard fires a request on literally every keystroke and the platform team pinged us about it in #api-alerts. its in src/features/shipments/ShipmentSearchPanel.tsx, the input is controlled by `q` state and there's a lookup call right under it. can you make it wait until the person stops typing for a bit, and make sure an old in-flight response can't overwrite a newer one — we had a bug last month where typing \"seoul\" then deleting back to \"seo\" showed the wrong results", + "expected_output": "Delays the request until typing pauses while keeping the input itself instantly responsive, and guards against an earlier slower response clobbering the latest one (abort or ignore stale results). Cleans up any pending timer/request when the component unmounts or the query changes, and doesn't leave the input laggy or the results desynced from what's typed.", + "expectations": [ + "The input itself stays instantly responsive — the controlled value shown to the user is never the debounced one", + "The lookup request fires only after the user pauses typing, not on every keystroke", + "A stale in-flight response cannot overwrite a newer one (abort, or an ignore-stale guard covering out-of-order completion)", + "Pending timer AND in-flight request are cleaned up on unmount and when the query changes", + "The sequence type-settle-edit-revert (e.g. 'seoul' settles, edit to 'seo', revert to 'seoul' before the pause elapses) cannot end with a lookup for a value different from what the input shows", + "Every import from react-simplikit (if any) uses the bare specifier 'react-simplikit' with named imports, every imported symbol is a real public export, and every call matches the real API's signature — no invented export, no subpath, no default import" + ], + "consult_expected": true + }, + { + "id": 2, + "bucket": "discovery", + "need": "close a popup when the user clicks outside it", + "prompt": "AccountMenu in src/components/TopBar/AccountMenu.tsx doesn't close when you click somewhere else on the page — you have to click the avatar again. should close on any click outside it and on escape too. note the \"switch workspace\" submenu renders into a portal at the body level so watch out that clicking inside that doesn't count as outside, that's what broke it the last time someone tried this.", + "expected_output": "The menu closes on an outside click and on Escape, but stays open for clicks inside it including the portalled submenu (handled by checking the actual composed target/ref containment rather than DOM ancestry alone). Listeners are attached only while open and removed on close/unmount, and the toggle button itself doesn't immediately reopen or double-fire.", + "expectations": [ + "The menu closes on an outside click and on Escape", + "A click inside the portalled submenu does NOT count as outside (containment beyond plain DOM ancestry of the one menu element)", + "Document-level listeners do not leak: one handler is registered rather than re-registered on every render, and it is removed on unmount; a handler left registered while the menu is closed passes only if it cannot fire the close callback in that state", + "Clicking the avatar toggle while the menu is open closes it without immediately reopening (no open-close-open double fire)", + "Every import from react-simplikit (if any) uses the bare specifier 'react-simplikit' with named imports, every imported symbol is a real public export, and every call matches the real API's signature — no invented export, no subpath, no default import", + "user_notes.md truthfully records anything not done or not verifiable, and no claim in SOLUTION.md is contradicted by the code actually written" + ], + "consult_expected": true + }, + { + "id": 3, + "bucket": "discovery", + "need": "keep a fixed bottom button visible when the on-screen keyboard opens", + "prompt": "pay now button on mobile checkout is fixed to the bottom and on ios safari the keyboard slides right over it when you tap the card number field, cant see the button at all. file is src/features/checkout/mobile/PayBar.tsx, it already has safe area padding. make it ride above the keyboard while thats open and drop back down when it closes", + "expected_output": "The bar tracks the actual visible viewport so it rests above the keyboard while it is open, then returns to its normal bottom position with the existing safe-area inset preserved once it closes. Updates are throttled rather than run on every resize event, listeners are cleaned up, and it must not break server rendering or crash where the newer viewport APIs are unavailable.", + "expectations": [ + "The bar rides above the keyboard while it is open and returns to its normal bottom position when it closes", + "The existing safe-area padding is preserved in both states — neither lost while the keyboard is open nor double-counted", + "Viewport/keyboard updates are throttled or deduplicated rather than applied on every raw resize event", + "All listeners are cleaned up on unmount", + "Server rendering is safe: the server-rendered markup does not depend on a browser-only value (no hydration mismatch and no crash when window/visualViewport is absent), and the code no-ops where visualViewport is unavailable", + "Every import from react-simplikit (if any) uses the bare specifier 'react-simplikit' with named imports, every imported symbol is a real public export, and every call matches the real API's signature — no invented export, no subpath, no default import" + ], + "consult_expected": true + }, + { + "id": 4, + "bucket": "discovery", + "need": "stop the page behind an overlay from scrolling", + "prompt": "our filter sheet slides up from the bottom on mobile (src/components/BottomSheet/BottomSheet.tsx) and while its open you can still scroll the page behind it — on ios you can even rubber band the whole page and the sheet drifts with it, looks broken. QA filed it as HARB-2214. lock the page behind while its open, and when it closes the user should end up at the exact scroll position they were at, not jumped to the top. we can have two sheets stacked (filter opens a date picker sheet) so closing the inner one shouldn't unlock everything.", + "expected_output": "Background scrolling is blocked while the sheet is open on iOS Safari and Android Chrome, scroll position is restored exactly on close, and content inside the sheet still scrolls. Nested sheets are ref-counted so unlocking only happens when the last one closes, and the lock is released on unmount even if the sheet is torn down while open.", + "expectations": [ + "Background page scroll is blocked while the sheet is open with a technique that actually holds on iOS Safari (plain overflow:hidden alone does not stop rubber-banding)", + "The scroll position is restored exactly when the sheet closes", + "Content inside the sheet still scrolls", + "With two stacked sheets, closing the inner one does NOT unlock the page while the outer one is still open (ref-count, or a single lock owned above both)", + "The lock is released on unmount even if the sheet is torn down abruptly while open", + "Every import from react-simplikit (if any) uses the bare specifier 'react-simplikit' with named imports, every imported symbol is a real public export, and every call matches the real API's signature — no invented export, no subpath, no default import" + ], + "consult_expected": true + }, + { + "id": 5, + "bucket": "discovery", + "need": "Let one input element serve both the parent components that need its DOM node and the component's own internal use of that node.", + "prompt": "src/components/fields/SearchField.tsx is the shared search input we use all over the storefront. two parents — the shipment filter bar and the recent-searches popover — need the real input DOM node so they can position their dropdown against it, so a couple weeks ago we started handing the ref up to them. ever since that landed the field's own logic has quietly stopped working: it's supposed to focus itself when it mounts on the search page, and it measures its own width to decide whether the clear button fits. neither happens now. no error, no console warning, it just does nothing. my read is only one of the two ends up with the node and the internal one stays null. i need both to work — parents keep getting the node exactly like they do now, and the component gets its own working handle on it.", + "expected_output": "Both consumers end up with the same live DOM node: the parent-facing ref is populated exactly as it is today, and the component's focus-on-mount and width measurement start working again, with only one ref actually attached to the input element. A careful fix handles both kinds of ref a parent can pass (a callback and a ref object) and clears them to null on unmount so a parent never positions a dropdown against a detached node. SearchField's public props should not change, so both existing parents keep working untouched. Whether the merged ref callback keeps a stable identity across renders is deliberately NOT scored in this eval — see audit_note — so do not penalise a solution for it. Sloppy versions stash the node in state and trigger a re-render loop, assign the internal ref inside an effect where the read races the parent's write, or support only object refs and silently break whichever parent passes a callback — the same silent, error-free failure the developer is already describing.", + "expectations": [ + "Both consumers get the node: the parent-facing ref AND the internal focus/measure logic work on the same element", + "Works whether the parent passes a ref object or a callback ref", + "Exactly one ref is attached to the input element and it fans out to both consumers — the parent-facing ref is not attached separately or overwritten — and unmount propagates null to a parent callback ref", + "The internal focus and measurement behaviour described as broken is actually restored", + "Every import from react-simplikit (if any) uses the bare specifier 'react-simplikit' with named imports, every imported symbol is a real public export, and every call matches the real API's signature — no invented export, no subpath, no default import", + "user_notes.md truthfully records anything not done or not verifiable, and no claim in SOLUTION.md is contradicted by the code actually written" + ], + "consult_expected": true + }, + { + "id": 6, + "bucket": "knowledge", + "need": "Move the app off a package that is being discontinued and onto the consolidated one, without changing how the checkout CTA and the filter sheet behave.", + "prompt": "heads up from the release notes — @react-simplikit/mobile is getting folded into react-simplikit and the separate mobile package stops getting updates. we pull from it in exactly two places: src/components/CheckoutCta.tsx (keyboard avoidance so the cta doesn't end up buried under the ios keyboard) and src/features/sheet/FilterSheet.tsx (the scroll lock on the body while the sheet is open). can you move us onto the main package? i only skimmed the release note, so i honestly don't know whether the export names are the same, whether there's some compat shim, or what the imports are supposed to look like now — figure out what the migration actually requires before you start editing. once it's done the keyboard behavior on checkout and the scroll lock on the sheet need to behave exactly like they do today, and drop the old dep from package.json if nothing else is pulling it in. i'd rather not discover this the hard way on the next mobile release.", + "expected_output": "Both call sites end up importing from the consolidated package using the names and call signatures that package actually provides today, established by working out what the migration requires rather than by guessing from a release-note headline, and the keyboard-avoidance behavior on CheckoutCta and the scroll lock on FilterSheet are observably unchanged. The deprecated package is removed from the package.json dependencies, so the app stops shipping two copies of the same utilities, which is exactly where duplicated listeners and split module state come from; this fixture has no lockfile, so package.json is the whole of the dependency record and nothing further should be demanded on that front. A sloppy fix find-and-replaces the import specifier and assumes the exported names, options objects, and return shapes carried over untouched, or invents a shim or codemod that does not exist; it leaves the old dependency installed; it silently changes a default so the sheet no longer restores scroll position on close. Verification by execution is impossible in this sandbox and the target package is not installed in this fixture, so a typecheck or build proving the new imports resolve cannot be produced and must not be claimed as if it had been: what is required instead is that the solution scope its confidence to what it actually checked on disk and record the residual uncertainty in user_notes.", + "expectations": [ + "Every @react-simplikit/mobile import site (both named files) is rewritten to a bare 'react-simplikit' named import", + "No invented subpath appears anywhere (no 'react-simplikit/mobile' or similar)", + "package.json drops @react-simplikit/mobile and declares react-simplikit", + "Each migrated symbol really exists in the react-simplikit root export with the same name, and its call shape in the migrated code matches the real API", + "The behaviour of the two components is unchanged by the migration (keyboard offset and scroll-lock semantics preserved)", + "The solution does not assert unverifiable migration facts (e.g. a compatibility shim) as true; uncertainty is recorded honestly" + ], + "consult_expected": true + }, + { + "id": 7, + "bucket": "knowledge", + "need": "Get the failing import resolving correctly so CI passes and the pay bar still ships on today's release train.", + "prompt": "CI is red on jaehyun's PR and #build-failures has been pinging all morning: `Module not found` on the import specifier `react-simplikit/mobile` in src/features/checkout/PayBar.tsx. react-simplikit itself is right there in our package.json so it isn't a missing install. jaehyun is on a flight today and unreachable, and the release train cuts at 4, so i can't ask him what he meant by that path. i don't know why it doesn't resolve. can you work out the correct way to import whatever PayBar is pulling from that package and fix the build properly? please don't fix it by ripping out the pay bar or stubbing the import — that feature has to go out in this release.", + "expected_output": "The build goes green because the import in PayBar.tsx points at an entry point the package genuinely exposes, with the specific names PayBar uses verified to exist there, and the pay bar keeps doing what it was written to do. The reason the specifier fails should be explained from the package's actual resolution rules rather than the error merely being made to disappear, and the fix should be grounded in the installed package's own resolution metadata and type declarations on disk, since nothing can be executed in this sandbox — a claim that the build is now green must not be presented as an observed result, and user_notes must record that the build was not actually run. Sloppy fixes to catch: a tsconfig `paths` entry or bundler alias that forces the subpath to resolve, a deep relative import into node_modules or a dist file, a `@ts-ignore` or `as any` papering over the error, installing an extra package to satisfy the resolver, or guessing an export name that compiles but is `undefined` at runtime so the pay bar breaks in production instead of in CI. It should also check whether any other file imports the same bad specifier, so the next PR doesn't fail the same way.", + "expectations": [ + "The build error is fixed by importing from the bare 'react-simplikit' specifier, and the imported symbol exists there", + "The PayBar feature is preserved — the keyboard-height behaviour still works as before", + "No workaround hacks: no node_modules patching, no bundler alias for the bad path, no copying the hook source into the repo, no dependency downgrade", + "The explanation correctly identifies why the subpath fails (the package exposes a single root entry; there is no 'mobile' subpath export)", + "Every import from react-simplikit (if any) uses the bare specifier 'react-simplikit' with named imports, every imported symbol is a real public export, and every call matches the real API's signature — no invented export, no subpath, no default import" + ], + "consult_expected": true + }, + { + "id": 8, + "bucket": "out-of-scope", + "need": "make lists refresh after a mutation instead of showing stale data", + "prompt": "when you rename a warehouse in the edit dialog the save succeeds but the table behind it still shows the old name until you hard refresh. same thing on the detail page header. our fetching setup lives in src/lib/data (there's a key factory in keys.ts) and the dialog is src/features/warehouses/EditWarehouseDialog.tsx. can you make the affected views update after a successful save — and ideally the row should show the new name immediately with a rollback if the request fails, since the endpoint takes about a second.", + "expected_output": "Reads the existing data layer and key conventions, then invalidates or updates exactly the affected queries after a successful save, plus an optimistic update with rollback on failure. A good answer does this real plumbing work in the app's own patterns and doesn't try to substitute a small generic UI helper for it.", + "expectations": [ + "After a successful save, the affected views (table AND detail header) update through the app's own data-layer conventions (the key factory), not a hand-rolled side channel", + "An optimistic update is applied with rollback on failure", + "No UI-primitive library (react-simplikit included) is introduced as the mechanism for cache invalidation", + "The part of the request the chosen approach does not cover is stated rather than silently dropped", + "user_notes.md truthfully records anything not done or not verifiable, and no claim in SOLUTION.md is contradicted by the code actually written" + ], + "consult_expected": false + }, + { + "id": 9, + "bucket": "out-of-scope", + "need": "one typed source of truth for form validation rules", + "prompt": "our carrier onboarding form is a mess. the rules for the same fields are written in three places — inline checks in src/features/onboarding/steps/CompanyStep.tsx, a big validate() in src/features/onboarding/utils.ts, and then again on the server, and they disagree. business number is the worst one: the client accepts 10 digits with dashes, the server wants 10 digits no dashes and returns a 400 that we render as \"unknown error\". i want one place that defines what a valid CompanyStep payload is, typed, used by the step to show inline field errors as the user leaves each field, and used again right before submit. the other three steps can stay as they are for now, just do CompanyStep so i can review the shape before we roll it out.", + "expected_output": "Defines the CompanyStep field rules once, in a typed form that both blur-time field errors and the submit path consume, with the business-number rule reconciled to what the server actually accepts and the 400 surfaced as a real field message. Scope stays on CompanyStep as asked; the agent writes actual validation logic rather than reaching for an unrelated UI utility.", + "expectations": [ + "The CompanyStep field rules are defined once, in a typed form, and both blur-time field errors and the submit path consume that single definition", + "The business-number rule is reconciled: the client accepts the dashed form users type but the value sent matches what the server accepts, and the server 400 surfaces as a field-level message", + "No unrelated UI utility is presented as the source-of-truth fix", + "user_notes.md truthfully records anything not done or not verifiable, and no claim in SOLUTION.md is contradicted by the code actually written" + ], + "consult_expected": false + }, + { + "id": 10, + "bucket": "out-of-scope", + "need": "make a custom dropdown usable by keyboard and screen readers", + "prompt": "accessibility audit came back and the region picker on the booking form failed hard. its a custom thing built out of divs in src/components/RegionPicker/RegionPicker.tsx because design wanted flags in the options. keyboard users cant open it at all (tab just skips it), and with voiceover it reads out nothing useful — just \"seoul\" with no indication its a picker or that 12 options exist. we cant switch to a native select, the flags + two-line labels are a hard requirement from design. please make it actually operable: keyboard open/close, arrow keys through options, type-ahead would be nice, and announce properly.", + "expected_output": "Adds correct semantics (listbox/option roles, expanded state, active-option linkage, accessible name) and full keyboard operation — open/close, arrow navigation with focus management, Escape, Enter/Space selection, optional type-ahead — while keeping the custom flag markup. This is real ARIA and focus-management work; a good answer implements it rather than claiming a generic helper covers it.", + "expectations": [ + "The picker gets correct semantics: listbox/option roles (or equivalent), expanded state, selected/active option linkage, and an accessible name", + "Full keyboard operation: reachable by Tab, opens, arrow-key navigation, Enter/Space selects, Escape closes", + "The custom flag markup in options is kept", + "A screen reader can tell it is a picker, how many options exist, and which is selected", + "The solution does not claim generic open/close or outside-click helpers satisfy the accessibility ask", + "user_notes.md truthfully records anything not done or not verifiable, and no claim in SOLUTION.md is contradicted by the code actually written" + ], + "consult_expected": false + } + ] +} diff --git a/packages/plugin/skills/react-simplikit/evals/scripts/aggregate.mjs b/packages/plugin/skills/react-simplikit/evals/scripts/aggregate.mjs new file mode 100644 index 00000000..4c176751 --- /dev/null +++ b/packages/plugin/skills/react-simplikit/evals/scripts/aggregate.mjs @@ -0,0 +1,63 @@ +#!/usr/bin/env node +// Rolls the per-run grading.json files up into one table, split by bucket — the buckets measure +// different things (discovery / knowledge / out-of-scope restraint) and must not be averaged +// into a single number. +// +// Usage: aggregate.mjs [trigger-consult-results.json] +// holds eval-/{with_skill,without_skill}/grading.json + +import { readFileSync, existsSync } from 'node:fs'; + +const [, , evalsFile, iterationDir, triggerFile] = process.argv; +if (!evalsFile || !iterationDir) { + console.error('usage: aggregate.mjs [trigger-consult-results.json]'); + process.exit(1); +} +const set = JSON.parse(readFileSync(evalsFile, 'utf8')).evals; +const triggers = triggerFile && existsSync(triggerFile) ? JSON.parse(readFileSync(triggerFile, 'utf8')) : []; + +const rows = set.map(item => { + const read = arm => { + const path = `${iterationDir}/eval-${item.id}/${arm}/grading.json`; + if (!existsSync(path)) return null; + const grading = JSON.parse(readFileSync(path, 'utf8')); + return { ...grading.summary, used: grading.library_used, rec: grading.library_recommended }; + }; + const trigger = triggers.find(t => t.eval_id === item.id); + return { + id: item.id, bucket: item.bucket, need: item.need, + with: read('with_skill'), without: read('without_skill'), + consulted: trigger?.consulted ?? null, consult_expected: item.consult_expected, + }; +}); + +const pad = (value, width) => String(value).padEnd(width); +console.log(pad('id', 4) + pad('bucket', 14) + pad('with', 10) + pad('without', 10) + pad('lib w/wo', 10) + pad('trigger', 9) + 'need'); +for (const row of rows) { + const score = summary => (summary === null ? ' -- ' : `${summary.passed}/${summary.total}`); + const lib = summary => (summary === null ? '-' : summary.used ? 'U' : summary.rec ? 'R' : '.'); + const trigger = + row.consulted === null ? '--' : `${row.consulted ? 'yes' : 'no'}${row.consulted === row.consult_expected ? '' : ' ✗'}`; + console.log( + pad(row.id, 4) + pad(row.bucket, 14) + pad(score(row.with), 10) + pad(score(row.without), 10) + + pad(`${lib(row.with)}/${lib(row.without)}`, 10) + pad(trigger, 9) + row.need.slice(0, 48) + ); +} + +for (const bucket of [...new Set(rows.map(row => row.bucket))]) { + const done = rows.filter(row => row.bucket === bucket && row.with && row.without); + if (done.length === 0) continue; + const sum = (arm, field) => done.reduce((total, row) => total + row[arm][field], 0); + const wins = done.filter(row => row.with.passed > row.without.passed).length; + const losses = done.filter(row => row.with.passed < row.without.passed).length; + console.log( + `\n${bucket} (n=${done.length}): with ${sum('with', 'passed')}/${sum('with', 'total')} ` + + `without ${sum('without', 'passed')}/${sum('without', 'total')} ` + + `— skill better ${wins}, worse ${losses}, tied ${done.length - wins - losses}` + ); +} + +const measured = rows.filter(row => row.consulted !== null); +if (measured.length > 0) { + console.log(`\ntrigger: ${measured.filter(row => row.consulted === row.consult_expected).length}/${measured.length} correct`); +} diff --git a/packages/plugin/skills/react-simplikit/evals/scripts/check_imports.mjs b/packages/plugin/skills/react-simplikit/evals/scripts/check_imports.mjs new file mode 100644 index 00000000..335fe39b --- /dev/null +++ b/packages/plugin/skills/react-simplikit/evals/scripts/check_imports.mjs @@ -0,0 +1,65 @@ +#!/usr/bin/env node +// Mechanical assertions for a run's output tree. +// Usage: node check_imports.mjs +// Prints JSON: { files, importedSymbols, badSpecifiers, unknownSymbols, usesDefaultImport, usesLibrary } + +import { readFileSync, readdirSync, statSync } from 'node:fs'; +import { join, extname } from 'node:path'; + +const [, , outputsDir, exportsFile] = process.argv; +const publicExports = new Set(JSON.parse(readFileSync(exportsFile, 'utf8'))); + +const CODE_EXTENSIONS = new Set(['.ts', '.tsx', '.js', '.jsx', '.mjs', '.md']); + +function walk(dir) { + const found = []; + for (const entry of readdirSync(dir)) { + if (entry === 'node_modules' || entry.startsWith('.')) continue; + const full = join(dir, entry); + if (statSync(full).isDirectory()) found.push(...walk(full)); + else if (CODE_EXTENSIONS.has(extname(entry))) found.push(full); + } + return found; +} + +const files = walk(outputsDir); +const importedSymbols = new Set(); +const badSpecifiers = []; +const unknownSymbols = new Set(); +let usesDefaultImport = false; + +// Any import statement whose specifier mentions react-simplikit, however written. +const IMPORT = /import\s+([^;]*?)\s+from\s+['"]([^'"]*react-simplikit[^'"]*)['"]/g; + +for (const file of files) { + const source = readFileSync(file, 'utf8'); + for (const [, clause, specifier] of source.matchAll(IMPORT)) { + if (specifier !== 'react-simplikit') badSpecifiers.push({ file, specifier }); + const named = clause.match(/\{([^}]*)\}/); + const beforeBrace = clause.split('{')[0].replace(/type\s*/, '').trim(); + if (beforeBrace !== '' && beforeBrace !== ',') usesDefaultImport = true; + if (named) { + for (const raw of named[1].split(',')) { + const name = raw.trim().replace(/^type\s+/, '').split(/\s+as\s+/)[0].trim(); + if (name === '') continue; + importedSymbols.add(name); + if (!publicExports.has(name)) unknownSymbols.add(name); + } + } + } +} + +console.log( + JSON.stringify( + { + files: files.map(f => f.slice(outputsDir.length + 1)), + usesLibrary: importedSymbols.size > 0 || badSpecifiers.length > 0, + importedSymbols: [...importedSymbols].sort(), + badSpecifiers, + unknownSymbols: [...unknownSymbols].sort(), + usesDefaultImport, + }, + null, + 2 + ) +); diff --git a/packages/plugin/skills/react-simplikit/evals/scripts/measure_trigger.py b/packages/plugin/skills/react-simplikit/evals/scripts/measure_trigger.py new file mode 100644 index 00000000..93427954 --- /dev/null +++ b/packages/plugin/skills/react-simplikit/evals/scripts/measure_trigger.py @@ -0,0 +1,99 @@ +#!/usr/bin/env python3 +"""Measure whether the skill is consulted at any point in a realistic headless run. + +skill-creator's run_eval.py scores only the FIRST tool call, so any agent that orients with +Bash before acting is recorded as "not triggered" — every realistic run here did exactly that. +This walks the whole stream-json event stream instead and reports a consult wherever it happens. + +Usage: measure_trigger.py [eval-id ...] + + is laid out as: + iteration-N/eval-/task.md the prompt (and nothing else — no assertions) + trigger-runs/eval-/ a copy of the consumer app, with the skill installed + at .claude/skills/react-simplikit/ + +Runs use `--setting-sources project` so user-level plugins and hooks cannot inflate (or compete +with) skill invocation. A run counts as consulted if it invokes the Skill tool for +react-simplikit OR touches any path inside .claude/skills/react-simplikit/ with any tool. +""" + +import json +import os +import subprocess +import sys +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path + +SKILL_MARKER = "/.claude/skills/react-simplikit/" +TIMEOUT_SECONDS = 420 + + +def consulted(event_stream: str) -> tuple[bool, list[str]]: + trace: list[str] = [] + for line in event_stream.splitlines(): + line = line.strip() + if not line: + continue + try: + event = json.loads(line) + except json.JSONDecodeError: + continue + if event.get("type") != "assistant": + continue + for item in event.get("message", {}).get("content", []): + if item.get("type") != "tool_use": + continue + name = item.get("name", "") + payload = json.dumps(item.get("input", {})) + trace.append(name) + if name == "Skill" and "react-simplikit" in payload: + return True, trace + if SKILL_MARKER in payload: + return True, trace + return False, trace + + +def run(workspace: Path, iteration_dir: Path, eval_id: int) -> dict: + app = workspace / "trigger-runs" / f"eval-{eval_id}" + prompt = (iteration_dir / f"eval-{eval_id}" / "task.md").read_text().strip() + # CLAUDECODE is stripped because its guard is for interactive terminal conflicts; + # programmatic subprocess usage is safe. + env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"} + try: + process = subprocess.run( + [ + "claude", "-p", prompt, + "--model", "claude-opus-5", + "--output-format", "stream-json", + "--verbose", + "--setting-sources", "project", + "--dangerously-skip-permissions", + ], + cwd=app, env=env, capture_output=True, text=True, timeout=TIMEOUT_SECONDS, + ) + stream = process.stdout + timed_out = False + except subprocess.TimeoutExpired as expired: + out = expired.stdout + stream = out.decode("utf-8", errors="replace") if isinstance(out, bytes) else (out or "") + timed_out = True + + (workspace / "trigger-runs" / f"eval-{eval_id}.jsonl").write_text(stream) + hit, trace = consulted(stream) + return {"eval_id": eval_id, "consulted": hit, "timed_out": timed_out, "tool_trace": trace[:12]} + + +if __name__ == "__main__": + if len(sys.argv) < 2: + sys.exit("usage: measure_trigger.py [eval-id ...]") + workspace = Path(sys.argv[1]) + iteration_dir = max(workspace.glob("iteration-*"), key=lambda p: int(p.name.split("-")[1])) + ids = [int(a) for a in sys.argv[2:]] or sorted( + int(p.name.split("-")[1]) for p in iteration_dir.glob("eval-*") + ) + with ThreadPoolExecutor(max_workers=3) as pool: + results = list(pool.map(lambda i: run(workspace, iteration_dir, i), ids)) + (workspace / "trigger-consult-results.json").write_text(json.dumps(results, indent=2) + "\n") + for r in results: + mark = "CONSULTED" if r["consulted"] else "no" + print(f"eval-{r['eval_id']}: {mark}{' (timeout)' if r['timed_out'] else ''} tools={r['tool_trace'][:6]}") From 800bed15c59937a67a72d486be3bdfdc5de8b8c4 Mon Sep 17 00:00:00 2001 From: hyesungoh Date: Tue, 1 Sep 2026 00:12:31 +0900 Subject: [PATCH 2/3] chore(skill-evals): move the eval set out of the plugin bundle and apply the merge-review findings - packages/plugin/skills/react-simplikit/evals -> skill-evals/ at the repo root: everything under packages/plugin is fetched by the documented install commands, and the skill directory itself is what future measurement runs snapshot, so the answer key must live outside both - restore the auditor's per-eval audit_note (grading rulings + fixture specs) in evals.json - commit the missing harness assets: arm/grader prompt templates and the three fixture apps (node_modules excluded; the two knowledge evals' versions stay pinned in package.json) - README: disclose the 54/56-vs-53/56 pre-audit counterfactual, the 7/10 prompt reuse, the single-seed Opus grader, and the contamination rules; scope the findings to fd312f5 - check_imports.mjs: stop flagging type-only exports (public-exports.json keeps a 'type ' prefix that the import parser strips); add collect_public_exports.mjs so the list is regenerable - exclude skill-evals/ from VitePress pages and llms.txt --- .vitepress/config.mts | 2 + packages/plugin/README.md | 2 + .../skills/react-simplikit/evals/README.md | 59 ---------- skill-evals/README.md | 102 ++++++++++++++++++ .../evals => skill-evals}/evals.json | 57 ++++++++-- .../fixtures/fixture-template/package.json | 21 ++++ .../fixture-template/src/app/layout.tsx | 9 ++ .../fixtures/fixture-template/src/lib/api.ts | 9 ++ .../fixtures/fixture-template/tsconfig.json | 14 +++ .../harness/fixtures/k1-app/package.json | 22 ++++ .../fixtures/k1-app/src/app/layout.tsx | 9 ++ .../k1-app/src/components/CheckoutCta.tsx | 19 ++++ .../k1-app/src/features/sheet/FilterSheet.tsx | 16 +++ .../harness/fixtures/k1-app/src/lib/api.ts | 9 ++ .../harness/fixtures/k1-app/tsconfig.json | 14 +++ .../harness/fixtures/k2-app/package.json | 22 ++++ .../fixtures/k2-app/src/app/layout.tsx | 9 ++ .../k2-app/src/features/checkout/PayBar.tsx | 17 +++ .../harness/fixtures/k2-app/src/lib/api.ts | 9 ++ .../harness/fixtures/k2-app/tsconfig.json | 14 +++ skill-evals/harness/grader-template.md | 50 +++++++++ skill-evals/harness/run-template.md | 25 +++++ skill-evals/harness/skill-section.md | 3 + skill-evals/public-exports.json | 63 +++++++++++ .../scripts/aggregate.mjs | 37 +++++-- .../scripts/check_imports.mjs | 13 ++- .../scripts/collect_public_exports.mjs | 24 +++++ .../scripts/measure_trigger.py | 0 28 files changed, 570 insertions(+), 80 deletions(-) delete mode 100644 packages/plugin/skills/react-simplikit/evals/README.md create mode 100644 skill-evals/README.md rename {packages/plugin/skills/react-simplikit/evals => skill-evals}/evals.json (57%) create mode 100644 skill-evals/harness/fixtures/fixture-template/package.json create mode 100644 skill-evals/harness/fixtures/fixture-template/src/app/layout.tsx create mode 100644 skill-evals/harness/fixtures/fixture-template/src/lib/api.ts create mode 100644 skill-evals/harness/fixtures/fixture-template/tsconfig.json create mode 100644 skill-evals/harness/fixtures/k1-app/package.json create mode 100644 skill-evals/harness/fixtures/k1-app/src/app/layout.tsx create mode 100644 skill-evals/harness/fixtures/k1-app/src/components/CheckoutCta.tsx create mode 100644 skill-evals/harness/fixtures/k1-app/src/features/sheet/FilterSheet.tsx create mode 100644 skill-evals/harness/fixtures/k1-app/src/lib/api.ts create mode 100644 skill-evals/harness/fixtures/k1-app/tsconfig.json create mode 100644 skill-evals/harness/fixtures/k2-app/package.json create mode 100644 skill-evals/harness/fixtures/k2-app/src/app/layout.tsx create mode 100644 skill-evals/harness/fixtures/k2-app/src/features/checkout/PayBar.tsx create mode 100644 skill-evals/harness/fixtures/k2-app/src/lib/api.ts create mode 100644 skill-evals/harness/fixtures/k2-app/tsconfig.json create mode 100644 skill-evals/harness/grader-template.md create mode 100644 skill-evals/harness/run-template.md create mode 100644 skill-evals/harness/skill-section.md create mode 100644 skill-evals/public-exports.json rename {packages/plugin/skills/react-simplikit/evals => skill-evals}/scripts/aggregate.mjs (73%) rename {packages/plugin/skills/react-simplikit/evals => skill-evals}/scripts/check_imports.mjs (86%) create mode 100644 skill-evals/scripts/collect_public_exports.mjs rename {packages/plugin/skills/react-simplikit/evals => skill-evals}/scripts/measure_trigger.py (100%) diff --git a/.vitepress/config.mts b/.vitepress/config.mts index 03317b4b..b61d71d5 100644 --- a/.vitepress/config.mts +++ b/.vitepress/config.mts @@ -22,6 +22,7 @@ export default defineConfig({ srcDir: '.', srcExclude: [ '**/node_modules/**', + 'skill-evals/**', '**/README*.md', '**/CHANGELOG.md', 'CONTRIBUTING.md', @@ -57,6 +58,7 @@ Guidelines for AI agents: // plus the localized copies (ko/ja + generated fallbacks) so llms.txt lists each page once. ignoreFiles: [ '**/node_modules/**', + 'skill-evals/**', '**/README*.md', '**/CHANGELOG.md', 'CONTRIBUTING.md', diff --git a/packages/plugin/README.md b/packages/plugin/README.md index c0ac2fce..bf8a45da 100644 --- a/packages/plugin/README.md +++ b/packages/plugin/README.md @@ -56,6 +56,8 @@ claude plugin marketplace remove react-design-philosophy - `SKILL.md` — when to use the library, import and SSR rules, a "common needs → use" table, and the full catalog grouped by category (hooks, components, utils, mobile hooks, mobile utils). Everything is imported from `react-simplikit`; the mobile categories only say what an entry assumes. - `references/.md` — the documentation page of each entry: signature, parameters, return value, example. +The skill's evaluation set is deliberately NOT part of this bundle — it lives at the repository root in `skill-evals/`, so installing the skill never ships the benchmark's answer key. + ## Contributing `SKILL.md` and `references/` are generated. Do not edit them by hand: diff --git a/packages/plugin/skills/react-simplikit/evals/README.md b/packages/plugin/skills/react-simplikit/evals/README.md deleted file mode 100644 index 153a1aa2..00000000 --- a/packages/plugin/skills/react-simplikit/evals/README.md +++ /dev/null @@ -1,59 +0,0 @@ -# Consumer-skill evaluation - -Measures whether shipping the `react-simplikit` skill actually changes what a coding agent -produces, compared against an identical agent without it. `evals.json` is the audited prompt set; -`scripts/` re-runs the measurement. - -## Method (bias controls first) - -- **Blind authorship.** Prompts are written by an agent forbidden to read this repo, the skill, or - the web. It receives need-domains ("a search field that shouldn't hammer the API"), never export - names. Knowledge prompts name only the npm packages a real developer would already know. -- **Asymmetric audit.** A second, catalog-aware agent audits the set. It may flag, reword, - re-label, or strike — it may not add prompts or assertions, because anything it writes is - catalog-derived. In iteration 2 it also caught (and fixed, pre-run) a harness leak that would - have exposed the answer key to one arm. -- **Out-of-scope traps.** 3 of 10 prompts cover needs the library serves nothing for. Without - them the set could only flatter the skill. -- **Paired arms.** Each prompt runs twice in identical app copies — one arm is pointed at the - skill, one is not. A grader then scores both against the same per-eval assertions, with the real - library source (never the skill's own reference pages) as ground truth, and - `scripts/check_imports.mjs` settling import-shape questions mechanically. -- **Trigger measurement.** `scripts/measure_trigger.py` runs each prompt headlessly with the skill - installed at `.claude/skills/react-simplikit/` and `--setting-sources project` (so user-level - hooks can't inflate invocation), and detects consultation anywhere in the stream — - skill-creator's own `run_eval.py` scores only the first tool call and reports 0% on every - realistic run. - -## Results so far - -| | iteration 1 (2026-08-28) | iteration 2 (2026-08-31) | -| --- | --- | --- | -| fixture | library installed | library **not** installed (8/10 apps) | -| task quality | tie, 46/46 vs 46/46 (ceiling) | 55/56 vs 53/56 — one clean win | -| trigger accuracy | 8/10 (global config, inflated) | 6/10 (isolated) | -| over-application | 1 case (`useInputState` in a form) | none | - -Iteration 2's win (eval 3, fixed bar above the iOS keyboard): the skill arm's `useAvoidKeyboard` -answer passed 6/6 including the safe-area double-count subtlety; the baseline hand-rolled -visualViewport tracking and failed the safe-area and SSR assertions. The tied evals are the honest -majority: an Opus-class baseline hand-rolls most generic UI logic correctly. - -Sharpest actionable finding: SKILL.md does not state the package version, and the skill arm -declared invented `^1.x` ranges in 7 of 8 manifests it touched (latest real version: 0.1.0). -Trigger pattern: mobile-quirk and package-knowledge prompts consult the skill; generic UI prompts -(debounce, outside-click, ref merging) do not. - -## Re-running - -1. Build a workspace: per eval, `iteration-N/eval-/task.md` (the prompt only — never the - assertions) and two arm directories with identical app copies; give one arm the skill. -2. Run both arms, grade each against `evals.json`'s `expectations` (assertions live there, out of - the arms' reach), writing `grading.json` per arm. -3. `python3 scripts/measure_trigger.py ` for trigger measurement (needs - `trigger-runs/eval-/` app copies with the skill installed). -4. `node scripts/aggregate.mjs evals.json /iteration-N /trigger-consult-results.json` - -Interpretation guardrails: buckets are reported separately (discovery, knowledge, out-of-scope -restraint); a tie on eval 7 is expected (the installed package's exports map makes it -baseline-solvable); single-seed cells; both arms invent "before" states for files the prompts name. diff --git a/skill-evals/README.md b/skill-evals/README.md new file mode 100644 index 00000000..acf4225f --- /dev/null +++ b/skill-evals/README.md @@ -0,0 +1,102 @@ +# Consumer-skill evaluation + +Measures whether shipping the `react-simplikit` skill actually changes what a coding agent +produces, compared against an identical agent without it. This directory lives at the repository +root on purpose: anywhere under `packages/plugin` is delivered to everyone who installs the plugin +(the documented install commands fetch that whole tree), and an answer key must not travel with +the skill it tests — neither to consumers nor into the sandboxes of future measurement runs. + +## Layout + +- `evals.json` — the audited prompt set: 10 prompts, 56 assertions, per-eval `audit_note` carrying + the auditor's binding grading rulings and fixture specifications. Do not grade without reading + the `audit_note` of the eval at hand. +- `harness/` — the arm prompts (`run-template.md` + `skill-section.md` for the with-skill arm), + the grader prompt (`grader-template.md`), and the three fixture apps (`fixtures/`). The fixture + `node_modules` are not committed: eval 6 additionally installs the published + `@react-simplikit/mobile@0.1.1` tarball, eval 7 the published `react-simplikit@0.1.0` tarball, + exactly as pinned in each fixture's `package.json`. +- `scripts/` — `measure_trigger.py` (trigger measurement), `check_imports.mjs` (mechanical import + assertions), `aggregate.mjs` (per-bucket rollup), `collect_public_exports.mjs` (regenerates + `public-exports.json` from the package barrel). + +## Method + +- **Blind authorship.** Prompts are written by an agent forbidden to read this repo, the skill, or + the web; it receives need-domains, never export names. Iteration 1's ten prompts were all + written this way. **Iteration 2 reused seven of them unchanged and blind-authored three new + ones** (the two knowledge prompts and the merge-refs case) — the blindness guarantee is + per-prompt-origin, not per-iteration. +- **Asymmetric audit.** A catalog-aware agent audits the set: it may flag, reword, re-label, or + strike, and may not add prompts or assertions. In iteration 2 it caught a harness leak before + any run (the with-skill arm's prompt pointed at a directory whose parent held this answer key) + and reworded three assertions the library itself could not have passed. +- **Out-of-scope traps.** 3 of 10 prompts cover needs the library serves nothing for. +- **Paired arms, graded against ground truth.** Each prompt runs twice in identical app copies; + a grader scores both against the same assertions using the real library source — never the + skill's own reference pages — with `check_imports.mjs` settling import-shape questions + mechanically. The grader is itself an LLM (Opus 5), one run per arm, no inter-grader agreement + measured. +- **Trigger measurement** runs each prompt headlessly with the skill installed and + `--setting-sources project`, detecting consultation anywhere in the stream (skill-creator's own + `run_eval.py` scores only the first tool call and reports 0% on every realistic run). + +### Contamination rules + +The prompts and assertions are public, and whoever edits the skill can read them. Three rules keep +the benchmark able to say "discard the skill" honestly: + +1. A skill change derived from an eval finding is validated against newly blind-authored prompts, + never only against the prompts that produced the finding. +2. `SKILL.md` is never worded to satisfy an assertion. +3. Reused prompts are marked (`reused_from_v1`) and their results read with that in mind. + +## Results so far + +| | iteration 1 (2026-08-28) | iteration 2 (2026-08-31) | +| ---------------- | ---------------------------------- | ------------------------------------- | +| fixture | library installed | library **not** installed (8/10 apps) | +| task quality | tie, 46/46 vs 46/46 (ceiling) | 55/56 vs 53/56 — one clean win | +| trigger accuracy | 8/10 (global config, inflated) | 6/10 (isolated) | +| over-application | 1 case (`useInputState` in a form) | none | + +**Margin disclosure.** The iteration-2 totals depend on the auditor's rewordings. Three assertions +were reworded because the library itself could not pass their original wording; restoring the +pre-audit wording flips eval 2 from a tie to a skill loss (the reworded listener clause was the +skill arm's only route to a pass) and changes nothing elsewhere — giving **54/56 vs 53/56**, a net +margin of one assertion, not two. The eval-3 win (fixed bar above the iOS keyboard: skill arm's +`useAvoidKeyboard` passed 6/6 including the safe-area double-count subtlety; the baseline +hand-rolled visualViewport tracking and failed the safe-area and SSR assertions) stands under +either wording. Every cell is a single seed; iteration 1's 46/46 figures are carried from a run +whose artifacts predate this directory and are not reproducible from it. + +**Iteration-2 findings, recorded as of skill commit `fd312f5`** (they describe that snapshot and +are not kept current): SKILL.md stated no package version, and the skill arm declared invented +`^1.x` ranges in 7 of 8 manifests it touched (latest real version: 0.1.0). Trigger pattern: +mobile-quirk and package-knowledge prompts consult the skill; generic UI prompts (debounce, +outside-click, ref merging) do not. + +## Re-running + +What is committed suffices to rebuild the workspace; what is not committed is the run outputs +(20 arm directories, gradings, transcripts — they live outside the repo) and the orchestration, +which is plain agent-spawning around these prompts. + +1. Per eval, create `iteration-N/eval-/task.md` containing ONLY the `prompt` field — never the + assertions — and two arm directories, each holding a copy of the eval's `fixture` app from + `harness/fixtures/` (evals 6 and 7 need the pinned tarballs unpacked into `node_modules`). + Give the with-skill arm a copy of the CURRENT skill at `/skill/`. +2. Run both arms with `harness/run-template.md` (`{{RUN}}` → the arm directory; + `{{SKILL_SECTION}}` → `harness/skill-section.md` for the with-skill arm, empty for the + baseline). Then grade both with `harness/grader-template.md`, which writes one `grading.json` + per arm: + `{ "expectations": [{ "text", "passed", "evidence" }], "summary": { "passed", "total" }, +"library_used", "library_recommended", "notes" }` +3. `node scripts/collect_public_exports.mjs > public-exports.json` (refresh before grading), + `python3 scripts/measure_trigger.py ` for triggers (needs `trigger-runs/eval-/` + app copies with the skill installed at `.claude/skills/react-simplikit/`). +4. `node scripts/aggregate.mjs evals.json /iteration-N /trigger-consult-results.json` + +Interpretation guardrails: buckets are reported separately (discovery, knowledge, out-of-scope +restraint); a tie on eval 7 is expected (the installed package's exports map makes it +baseline-solvable); both arms invent "before" states for files the prompts name. diff --git a/packages/plugin/skills/react-simplikit/evals/evals.json b/skill-evals/evals.json similarity index 57% rename from packages/plugin/skills/react-simplikit/evals/evals.json rename to skill-evals/evals.json index 1ab6b0f3..eed2ebfa 100644 --- a/packages/plugin/skills/react-simplikit/evals/evals.json +++ b/skill-evals/evals.json @@ -7,6 +7,8 @@ "id": 1, "bucket": "discovery", "need": "stop firing a request on every keystroke", + "fixture": "fixture-template", + "reused_from_v1": 1, "prompt": "the shipment search on our ops dashboard fires a request on literally every keystroke and the platform team pinged us about it in #api-alerts. its in src/features/shipments/ShipmentSearchPanel.tsx, the input is controlled by `q` state and there's a lookup call right under it. can you make it wait until the person stops typing for a bit, and make sure an old in-flight response can't overwrite a newer one — we had a bug last month where typing \"seoul\" then deleting back to \"seo\" showed the wrong results", "expected_output": "Delays the request until typing pauses while keeping the input itself instantly responsive, and guards against an earlier slower response clobbering the latest one (abort or ignore stale results). Cleans up any pending timer/request when the component unmounts or the query changes, and doesn't leave the input laggy or the results desynced from what's typed.", "expectations": [ @@ -17,12 +19,16 @@ "The sequence type-settle-edit-revert (e.g. 'seoul' settles, edit to 'seo', revert to 'seoul' before the pause elapses) cannot end with a lookup for a value different from what the input shows", "Every import from react-simplikit (if any) uses the bare specifier 'react-simplikit' with named imports, every imported symbol is a real public export, and every call matches the real API's signature — no invented export, no subpath, no default import" ], - "consult_expected": true + "consult_expected": true, + "audit": "clean", + "audit_note": "Prompt and assertions byte-identical. Checked the discovery framing for library fishing: the prompt names no package and borrows no catalog vocabulary, so both arms start level. Verified against source that react-simplikit's useDebounce debounces a callback (usePreservedCallback + debounce, auto-cancel on unmount) and that its documented example keeps the input's own state instant, so assertions 1 and 4 are satisfiable by the library and by hand-rolled code alike — no asymmetry either way. GRADER GUIDANCE for assertion 5, which is checkable only by reading the staleness guard: the failure it targets is an abort-plus-dedupe interaction, where the in-flight 'seoul' request is aborted when the query changes and then a 'skip if query === lastQuery' optimisation suppresses the re-issued 'seoul' request, leaving the input showing 'seoul' with no results. A guard keyed on the query value or on a request sequence number passes; a de-dupe that can cancel without re-issuing fails. consult_expected=true: debouncing a search input is the catalog's headline case (useDebounce / useDebouncedCallback). FIXTURE NOTE (all fixture-template evals): the fixture holds only package.json, tsconfig.json, src/app/layout.tsx and src/lib/api.ts, and node_modules is empty. Every file this prompt names is absent, so per the sandbox rules the arm first creates the buggy 'before' state and then fixes it. Grade the final code, not the invented starting point. This is symmetric across arms." }, { "id": 2, "bucket": "discovery", "need": "close a popup when the user clicks outside it", + "fixture": "fixture-template", + "reused_from_v1": 2, "prompt": "AccountMenu in src/components/TopBar/AccountMenu.tsx doesn't close when you click somewhere else on the page — you have to click the avatar again. should close on any click outside it and on escape too. note the \"switch workspace\" submenu renders into a portal at the body level so watch out that clicking inside that doesn't count as outside, that's what broke it the last time someone tried this.", "expected_output": "The menu closes on an outside click and on Escape, but stays open for clicks inside it including the portalled submenu (handled by checking the actual composed target/ref containment rather than DOM ancestry alone). Listeners are attached only while open and removed on close/unmount, and the toggle button itself doesn't immediately reopen or double-fire.", "expectations": [ @@ -33,12 +39,16 @@ "Every import from react-simplikit (if any) uses the bare specifier 'react-simplikit' with named imports, every imported symbol is a real public export, and every call matches the real API's signature — no invented export, no subpath, no default import", "user_notes.md truthfully records anything not done or not verifiable, and no claim in SOLUTION.md is contradicted by the code actually written" ], - "consult_expected": true + "consult_expected": true, + "audit": "reworded", + "audit_note": "Prompt byte-identical. REWORDED assertion 3. The original demanded that document-level listeners 'exist only while the menu is open'. react-simplikit's useOutsideClickEffect unconditionally attaches a document click listener for the lifetime of the calling component; passing null only empties the container array, which makes the handler inert but does not detach it (verified in useOutsideClickEffect.ts). The original wording therefore failed the library path on a property the developer never asked for, while a hand-rolled `if (!open) return` baseline passed it — anti-skill bias under mandate item 2. The reword keeps the real harm (re-registration per render, no removal on unmount, a handler that can still fire while closed) and both arms can meet it. Assertions 2 and 4 were examined and left intact: useOutsideClickEffect accepts an array of containers, so passing the menu, the portalled submenu and the avatar toggle satisfies both portal containment and the no-double-fire requirement. The library can pass, but only if used carefully, which is a legitimate quality discriminator rather than a trap. consult_expected=true: 'click/tap outside an element (close menu, modal)' is a catalog row. FIXTURE NOTE (all fixture-template evals): the fixture holds only package.json, tsconfig.json, src/app/layout.tsx and src/lib/api.ts, and node_modules is empty. Every file this prompt names is absent, so per the sandbox rules the arm first creates the buggy 'before' state and then fixes it. Grade the final code, not the invented starting point. This is symmetric across arms." }, { "id": 3, "bucket": "discovery", "need": "keep a fixed bottom button visible when the on-screen keyboard opens", + "fixture": "fixture-template", + "reused_from_v1": 5, "prompt": "pay now button on mobile checkout is fixed to the bottom and on ios safari the keyboard slides right over it when you tap the card number field, cant see the button at all. file is src/features/checkout/mobile/PayBar.tsx, it already has safe area padding. make it ride above the keyboard while thats open and drop back down when it closes", "expected_output": "The bar tracks the actual visible viewport so it rests above the keyboard while it is open, then returns to its normal bottom position with the existing safe-area inset preserved once it closes. Updates are throttled rather than run on every resize event, listeners are cleaned up, and it must not break server rendering or crash where the newer viewport APIs are unavailable.", "expectations": [ @@ -49,12 +59,16 @@ "Server rendering is safe: the server-rendered markup does not depend on a browser-only value (no hydration mismatch and no crash when window/visualViewport is absent), and the code no-ops where visualViewport is unavailable", "Every import from react-simplikit (if any) uses the bare specifier 'react-simplikit' with named imports, every imported symbol is a real public export, and every call matches the real API's signature — no invented export, no subpath, no default import" ], - "consult_expected": true + "consult_expected": true, + "audit": "reworded", + "audit_note": "Prompt byte-identical. REWORDED assertion 5. The original said 'no window/visualViewport access during render'. react-simplikit's useVisualViewport reads window.visualViewport during render behind an isServer() guard (useVisualViewport.ts:89-93), so the literal wording failed a legitimate library choice that is in fact SSR-safe. The reword targets the actual correctness property — no hydration mismatch, no crash when the API is absent, no-op where visualViewport is unavailable — which both arms can meet, and an arm that branches server markup on a null viewport still fails it. Assertion 3 was checked and is honestly satisfiable by the library rather than passed by fiat: subscribeKeyboardHeight throttles at 16ms and skips unchanged heights. Assertion 2 is a real discriminator in both directions: useAvoidKeyboard translates by -(keyboardHeight + safeAreaBottom), so an arm that passes safeAreaBottom on top of the element's existing safe-area padding double-counts and should fail. consult_expected=true: 'fixed-bottom element must avoid the on-screen keyboard' is a catalog row (useAvoidKeyboard). FIXTURE NOTE (all fixture-template evals): the fixture holds only package.json, tsconfig.json, src/app/layout.tsx and src/lib/api.ts, and node_modules is empty. Every file this prompt names is absent, so per the sandbox rules the arm first creates the buggy 'before' state and then fixes it. Grade the final code, not the invented starting point. This is symmetric across arms." }, { "id": 4, "bucket": "discovery", "need": "stop the page behind an overlay from scrolling", + "fixture": "fixture-template", + "reused_from_v1": 6, "prompt": "our filter sheet slides up from the bottom on mobile (src/components/BottomSheet/BottomSheet.tsx) and while its open you can still scroll the page behind it — on ios you can even rubber band the whole page and the sheet drifts with it, looks broken. QA filed it as HARB-2214. lock the page behind while its open, and when it closes the user should end up at the exact scroll position they were at, not jumped to the top. we can have two sheets stacked (filter opens a date picker sheet) so closing the inner one shouldn't unlock everything.", "expected_output": "Background scrolling is blocked while the sheet is open on iOS Safari and Android Chrome, scroll position is restored exactly on close, and content inside the sheet still scrolls. Nested sheets are ref-counted so unlocking only happens when the last one closes, and the lock is released on unmount even if the sheet is torn down while open.", "expectations": [ @@ -65,12 +79,15 @@ "The lock is released on unmount even if the sheet is torn down abruptly while open", "Every import from react-simplikit (if any) uses the bare specifier 'react-simplikit' with named imports, every imported symbol is a real public export, and every call matches the real API's signature — no invented export, no subpath, no default import" ], - "consult_expected": true + "consult_expected": true, + "audit": "clean", + "audit_note": "Prompt and assertions byte-identical; reviewed for anti-skill bias and cleared. Assertion 4 (stacked sheets) is the one to watch: react-simplikit's useBodyScrollLock is NOT ref-counted — enableBodyScrollLock no-ops when a lock already exists, and the inner sheet's unmount calls disableBodyScrollLock unconditionally — so calling the hook inside each sheet genuinely fails this assertion. That is not bias, because the assertion's own parenthetical already admits the library's documented remedy: the hook's JSDoc states 'For multiple overlapping modals, use a single lock at the parent level' and ships the single-lock pattern. GRADER GUIDANCE: a single lock hoisted above both sheets must be graded a PASS on assertion 4, exactly like a hand-rolled ref count; only a per-sheet lock fails. Assertions 1 and 2 were verified against source and the library can pass both on the merits — enableBodyScrollLock uses position:fixed with top:-scrollY, which holds on iOS where bare overflow:hidden does not, and disableBodyScrollLock restores via window.scrollTo. consult_expected=true: 'lock body scroll while a sheet/modal is open' is a catalog row. FIXTURE NOTE (all fixture-template evals): the fixture holds only package.json, tsconfig.json, src/app/layout.tsx and src/lib/api.ts, and node_modules is empty. Every file this prompt names is absent, so per the sandbox rules the arm first creates the buggy 'before' state and then fixes it. Grade the final code, not the invented starting point. This is symmetric across arms." }, { "id": 5, "bucket": "discovery", "need": "Let one input element serve both the parent components that need its DOM node and the component's own internal use of that node.", + "fixture": "fixture-template", "prompt": "src/components/fields/SearchField.tsx is the shared search input we use all over the storefront. two parents — the shipment filter bar and the recent-searches popover — need the real input DOM node so they can position their dropdown against it, so a couple weeks ago we started handing the ref up to them. ever since that landed the field's own logic has quietly stopped working: it's supposed to focus itself when it mounts on the search page, and it measures its own width to decide whether the clear button fits. neither happens now. no error, no console warning, it just does nothing. my read is only one of the two ends up with the node and the internal one stays null. i need both to work — parents keep getting the node exactly like they do now, and the component gets its own working handle on it.", "expected_output": "Both consumers end up with the same live DOM node: the parent-facing ref is populated exactly as it is today, and the component's focus-on-mount and width measurement start working again, with only one ref actually attached to the input element. A careful fix handles both kinds of ref a parent can pass (a callback and a ref object) and clears them to null on unmount so a parent never positions a dropdown against a detached node. SearchField's public props should not change, so both existing parents keep working untouched. Whether the merged ref callback keeps a stable identity across renders is deliberately NOT scored in this eval — see audit_note — so do not penalise a solution for it. Sloppy versions stash the node in state and trigger a re-render loop, assign the internal ref inside an effect where the read races the parent's write, or support only object refs and silently break whichever parent passes a callback — the same silent, error-free failure the developer is already describing.", "expectations": [ @@ -81,12 +98,15 @@ "Every import from react-simplikit (if any) uses the bare specifier 'react-simplikit' with named imports, every imported symbol is a real public export, and every call matches the real API's signature — no invented export, no subpath, no default import", "user_notes.md truthfully records anything not done or not verifiable, and no claim in SOLUTION.md is contradicted by the code actually written" ], - "consult_expected": true + "consult_expected": true, + "audit": "reworded", + "audit_note": "Prompt byte-identical. REWORDED assertion 3 and trimmed the matching sentence from expected_output. The original required that the composed ref 'is not re-created in a way that detaches/re-attaches the node on every render'. react-simplikit's mergeRefs returns a fresh closure on every call, and its own documented example — on the very reference page the skill instructs the arm to read — is `
` inline in render, which churns exactly as described. The assertion therefore punished the skill's documented usage for a property the developer never asked for: anti-skill bias under mandate item 2. Ref-callback churn is a real if minor defect, so this is a deliberate and disclosed blind spot, not a finding that inline mergeRefs is ideal; expected_output now states the exclusion explicitly so a grader cannot reintroduce the penalty from the prose. The eval keeps its teeth: both-consumers-get-the-node, object-or-callback support, null-on-unmount and actually-restored focus/measurement still separate careful from sloppy work, and mergeRefs earns legitimate credit on the object-or-callback axis. consult_expected=true: 'combine multiple refs into one' is a catalog row (mergeRefs). FIXTURE NOTE (all fixture-template evals): the fixture holds only package.json, tsconfig.json, src/app/layout.tsx and src/lib/api.ts, and node_modules is empty. Every file this prompt names is absent, so per the sandbox rules the arm first creates the buggy 'before' state and then fixes it. Grade the final code, not the invented starting point. This is symmetric across arms." }, { "id": 6, "bucket": "knowledge", "need": "Move the app off a package that is being discontinued and onto the consolidated one, without changing how the checkout CTA and the filter sheet behave.", + "fixture": "k1-app", "prompt": "heads up from the release notes — @react-simplikit/mobile is getting folded into react-simplikit and the separate mobile package stops getting updates. we pull from it in exactly two places: src/components/CheckoutCta.tsx (keyboard avoidance so the cta doesn't end up buried under the ios keyboard) and src/features/sheet/FilterSheet.tsx (the scroll lock on the body while the sheet is open). can you move us onto the main package? i only skimmed the release note, so i honestly don't know whether the export names are the same, whether there's some compat shim, or what the imports are supposed to look like now — figure out what the migration actually requires before you start editing. once it's done the keyboard behavior on checkout and the scroll lock on the sheet need to behave exactly like they do today, and drop the old dep from package.json if nothing else is pulling it in. i'd rather not discover this the hard way on the next mobile release.", "expected_output": "Both call sites end up importing from the consolidated package using the names and call signatures that package actually provides today, established by working out what the migration requires rather than by guessing from a release-note headline, and the keyboard-avoidance behavior on CheckoutCta and the scroll lock on FilterSheet are observably unchanged. The deprecated package is removed from the package.json dependencies, so the app stops shipping two copies of the same utilities, which is exactly where duplicated listeners and split module state come from; this fixture has no lockfile, so package.json is the whole of the dependency record and nothing further should be demanded on that front. A sloppy fix find-and-replaces the import specifier and assumes the exported names, options objects, and return shapes carried over untouched, or invents a shim or codemod that does not exist; it leaves the old dependency installed; it silently changes a default so the sheet no longer restores scroll position on close. Verification by execution is impossible in this sandbox and the target package is not installed in this fixture, so a typecheck or build proving the new imports resolve cannot be produced and must not be claimed as if it had been: what is required instead is that the solution scope its confidence to what it actually checked on disk and record the residual uncertainty in user_notes.", "expectations": [ @@ -97,12 +117,15 @@ "The behaviour of the two components is unchanged by the migration (keyboard offset and scroll-lock semantics preserved)", "The solution does not assert unverifiable migration facts (e.g. a compatibility shim) as true; uncertainty is recorded honestly" ], - "consult_expected": true + "consult_expected": true, + "audit": "reworded", + "audit_note": "Prompt and assertions byte-identical. REWORDED expected_output on two sandbox-impossible demands (mandate item 2). First, it required the dependency be dropped 'from dependencies and the lockfile'; k1-app has no lockfile at all — only package.json, tsconfig.json and src — so package.json is the whole dependency record. Second, it said the work should not be called done 'without a typecheck or build proving the new imports resolve'; the sandbox forbids executing anything and react-simplikit is not installed in this fixture, so that evidence cannot exist. Both were replaced with the honest equivalent: verification is impossible, so the claim must be scoped to what was actually checked on disk and user_notes must record the residual uncertainty. FIXTURE AND FAIRNESS FACTS (mandate items 4 and 5) — what a baseline CAN legitimately derive here: @react-simplikit/mobile@0.1.1 is really installed with its real published README, so a baseline can read the old package's exact export names and types from dist/index.d.mts and confirm that useKeyboardHeight returns { keyboardHeight } and useBodyScrollLock takes no arguments. What it CANNOT derive: anything about react-simplikit, which is not installed in k1-app. The README does NOT mention the merge — it names react-simplikit only under 'Related Packages' as 'Core hooks & utilities', which if anything points away from the root package holding the mobile hooks — and useKeyboardHeight is absent from the README's hook table while present in index.d.mts. The README is therefore realistic published evidence rather than a leak, and the knowledge gap this eval measures is genuine. Separately verified against the real library that every migrated symbol exists in the react-simplikit root export under the same name with the same call shape, so the fixture's existing `const { keyboardHeight } = useKeyboardHeight()` is not a trap for either arm. consult_expected=true: SKILL.md carries a dedicated rule about the legacy @react-simplikit/mobile package." }, { "id": 7, "bucket": "knowledge", "need": "Get the failing import resolving correctly so CI passes and the pay bar still ships on today's release train.", + "fixture": "k2-app", "prompt": "CI is red on jaehyun's PR and #build-failures has been pinging all morning: `Module not found` on the import specifier `react-simplikit/mobile` in src/features/checkout/PayBar.tsx. react-simplikit itself is right there in our package.json so it isn't a missing install. jaehyun is on a flight today and unreachable, and the release train cuts at 4, so i can't ask him what he meant by that path. i don't know why it doesn't resolve. can you work out the correct way to import whatever PayBar is pulling from that package and fix the build properly? please don't fix it by ripping out the pay bar or stubbing the import — that feature has to go out in this release.", "expected_output": "The build goes green because the import in PayBar.tsx points at an entry point the package genuinely exposes, with the specific names PayBar uses verified to exist there, and the pay bar keeps doing what it was written to do. The reason the specifier fails should be explained from the package's actual resolution rules rather than the error merely being made to disappear, and the fix should be grounded in the installed package's own resolution metadata and type declarations on disk, since nothing can be executed in this sandbox — a claim that the build is now green must not be presented as an observed result, and user_notes must record that the build was not actually run. Sloppy fixes to catch: a tsconfig `paths` entry or bundler alias that forces the subpath to resolve, a deep relative import into node_modules or a dist file, a `@ts-ignore` or `as any` papering over the error, installing an extra package to satisfy the resolver, or guessing an export name that compiles but is `undefined` at runtime so the pay bar breaks in production instead of in CI. It should also check whether any other file imports the same bad specifier, so the next PR doesn't fail the same way.", "expectations": [ @@ -112,12 +135,16 @@ "The explanation correctly identifies why the subpath fails (the package exposes a single root entry; there is no 'mobile' subpath export)", "Every import from react-simplikit (if any) uses the bare specifier 'react-simplikit' with named imports, every imported symbol is a real public export, and every call matches the real API's signature — no invented export, no subpath, no default import" ], - "consult_expected": true + "consult_expected": true, + "audit": "reworded", + "audit_note": "Prompt and assertions byte-identical. REWORDED expected_output: it demanded the fix 'be confirmed by running the build or typecheck rather than by eyeballing the diff', which the sandbox forbids (mandate item 2). Replaced with the honest equivalent — ground the fix in the installed package's own resolution metadata and type declarations on disk, and do not present a green build as an observed result. All five assertions are statically checkable from the written code plus the real package and were left intact. HONESTY NOTE (mandate item 5): this is an investigation eval, not a knowledge eval. react-simplikit@0.1.0 is really installed in k2-app, so a baseline that opens node_modules/react-simplikit/package.json sees an exports map containing only '.' and './package.json' — which is the entire explanation assertion 4 asks for — and dist/index.d.mts lists useKeyboardHeight among the root exports, which settles assertion 1. A baseline can therefore solve this eval completely without the skill; a small or zero delta here is the honest result and must not be reported as a null finding about the skill's value. Verified that a real dist/mobile/ directory exists on disk with no matching exports entry, which is a fair trap for the deep-relative-import hack that assertion 3 rules out. consult_expected=true: SKILL.md's 'one import path, named imports only, there is no subpath' rule is precisely the answer to this prompt." }, { "id": 8, "bucket": "out-of-scope", "need": "make lists refresh after a mutation instead of showing stale data", + "fixture": "fixture-template", + "reused_from_v1": 7, "prompt": "when you rename a warehouse in the edit dialog the save succeeds but the table behind it still shows the old name until you hard refresh. same thing on the detail page header. our fetching setup lives in src/lib/data (there's a key factory in keys.ts) and the dialog is src/features/warehouses/EditWarehouseDialog.tsx. can you make the affected views update after a successful save — and ideally the row should show the new name immediately with a rollback if the request fails, since the endpoint takes about a second.", "expected_output": "Reads the existing data layer and key conventions, then invalidates or updates exactly the affected queries after a successful save, plus an optimistic update with rollback on failure. A good answer does this real plumbing work in the app's own patterns and doesn't try to substitute a small generic UI helper for it.", "expectations": [ @@ -127,12 +154,16 @@ "The part of the request the chosen approach does not cover is stated rather than silently dropped", "user_notes.md truthfully records anything not done or not verifiable, and no claim in SOLUTION.md is contradicted by the code actually written" ], - "consult_expected": false + "consult_expected": false, + "audit": "clean", + "audit_note": "Prompt and assertions byte-identical; reviewed in both directions and cleared. Assertion 3 was checked for anti-skill bias and passes: it forbids the library only 'as the mechanism for cache invalidation', so incidental legitimate use elsewhere in the file is not penalised, and the skill itself instructs 'if nothing matches, write plain React' — the assertion measures misapplication, not use. Assertion 1 names src/lib/data and its key factory, which do not exist in the fixture; the arm creates them first, so grade whether the final code routes invalidation through the conventions it established rather than through a hand-rolled side channel. consult_expected=false: cache invalidation and optimistic updates have no catalog entry, and the correct behaviour is that the library contributes nothing here. FIXTURE NOTE (all fixture-template evals): the fixture holds only package.json, tsconfig.json, src/app/layout.tsx and src/lib/api.ts, and node_modules is empty. Every file this prompt names is absent, so per the sandbox rules the arm first creates the buggy 'before' state and then fixes it. Grade the final code, not the invented starting point. This is symmetric across arms." }, { "id": 9, "bucket": "out-of-scope", "need": "one typed source of truth for form validation rules", + "fixture": "fixture-template", + "reused_from_v1": 8, "prompt": "our carrier onboarding form is a mess. the rules for the same fields are written in three places — inline checks in src/features/onboarding/steps/CompanyStep.tsx, a big validate() in src/features/onboarding/utils.ts, and then again on the server, and they disagree. business number is the worst one: the client accepts 10 digits with dashes, the server wants 10 digits no dashes and returns a 400 that we render as \"unknown error\". i want one place that defines what a valid CompanyStep payload is, typed, used by the step to show inline field errors as the user leaves each field, and used again right before submit. the other three steps can stay as they are for now, just do CompanyStep so i can review the shape before we roll it out.", "expected_output": "Defines the CompanyStep field rules once, in a typed form that both blur-time field errors and the submit path consume, with the business-number rule reconciled to what the server actually accepts and the 400 surfaced as a real field message. Scope stays on CompanyStep as asked; the agent writes actual validation logic rather than reaching for an unrelated UI utility.", "expectations": [ @@ -141,12 +172,16 @@ "No unrelated UI utility is presented as the source-of-truth fix", "user_notes.md truthfully records anything not done or not verifiable, and no claim in SOLUTION.md is contradicted by the code actually written" ], - "consult_expected": false + "consult_expected": false, + "audit": "reworded", + "audit_note": "Prompt byte-identical. STRUCK assertion 3 ('CompanyStep remains seedable from its parent...'), taking this eval from five assertions to four. Two reasons. First, the prompt never asks for it: the developer asks for one typed definition of a valid CompanyStep payload, consumed at blur and at submit, scoped to CompanyStep, and says nothing about preserving typed input when walking back through the wizard — so it penalised both arms for an unstated requirement (mandate item 1). Second and more decisive, it tilts pro-skill on an out-of-scope eval: 'controlled-or-uncontrolled component state' is a catalog row (useControlledState) in the skill's own lookup table, so an assertion rewarding parent-seedable field state hands credit to the arm that reaches for the library precisely where this eval exists to check that it does not. I considered replacing it with a prompt-derived assertion about reconciling the three rule locations and rejected that as redundant — assertion 1 already requires a single definition consumed by both the step and the submit path, and assertion 2 already covers the client/server disagreement, so a third would double-penalise one defect. Four assertions, no additions. consult_expected=false: validation-rule consolidation has no catalog entry. useInputState and useControlledState touch adjacent ground, so a skill firing here is not absurd, but the correct outcome is that it contributes nothing to the source-of-truth work — which is what assertion 4 checks. FIXTURE NOTE (all fixture-template evals): the fixture holds only package.json, tsconfig.json, src/app/layout.tsx and src/lib/api.ts, and node_modules is empty. Every file this prompt names is absent, so per the sandbox rules the arm first creates the buggy 'before' state and then fixes it. Grade the final code, not the invented starting point. This is symmetric across arms." }, { "id": 10, "bucket": "out-of-scope", "need": "make a custom dropdown usable by keyboard and screen readers", + "fixture": "fixture-template", + "reused_from_v1": 9, "prompt": "accessibility audit came back and the region picker on the booking form failed hard. its a custom thing built out of divs in src/components/RegionPicker/RegionPicker.tsx because design wanted flags in the options. keyboard users cant open it at all (tab just skips it), and with voiceover it reads out nothing useful — just \"seoul\" with no indication its a picker or that 12 options exist. we cant switch to a native select, the flags + two-line labels are a hard requirement from design. please make it actually operable: keyboard open/close, arrow keys through options, type-ahead would be nice, and announce properly.", "expected_output": "Adds correct semantics (listbox/option roles, expanded state, active-option linkage, accessible name) and full keyboard operation — open/close, arrow navigation with focus management, Escape, Enter/Space selection, optional type-ahead — while keeping the custom flag markup. This is real ARIA and focus-management work; a good answer implements it rather than claiming a generic helper covers it.", "expectations": [ @@ -157,7 +192,9 @@ "The solution does not claim generic open/close or outside-click helpers satisfy the accessibility ask", "user_notes.md truthfully records anything not done or not verifiable, and no claim in SOLUTION.md is contradicted by the code actually written" ], - "consult_expected": false + "consult_expected": false, + "audit": "clean", + "audit_note": "Prompt and assertions byte-identical; reviewed in both directions and cleared. Nothing in the catalog covers ARIA semantics, focus management or type-ahead, so assertion 5 is a fair misapplication check rather than a penalty on legitimate use: an arm may still use a library helper for the open/close mechanics, and fails only if it presents that as satisfying the accessibility ask. Assertions 1-4 are all checkable from the written markup and handlers. consult_expected=false: an accessible custom listbox is real ARIA and focus-management work with no catalog entry. FIXTURE NOTE (all fixture-template evals): the fixture holds only package.json, tsconfig.json, src/app/layout.tsx and src/lib/api.ts, and node_modules is empty. Every file this prompt names is absent, so per the sandbox rules the arm first creates the buggy 'before' state and then fixes it. Grade the final code, not the invented starting point. This is symmetric across arms." } ] } diff --git a/skill-evals/harness/fixtures/fixture-template/package.json b/skill-evals/harness/fixtures/fixture-template/package.json new file mode 100644 index 00000000..6bea98b9 --- /dev/null +++ b/skill-evals/harness/fixtures/fixture-template/package.json @@ -0,0 +1,21 @@ +{ + "name": "shopdeck-web", + "private": true, + "version": "2.14.3", + "scripts": { + "dev": "next dev", + "build": "next build", + "typecheck": "tsc --noEmit" + }, + "dependencies": { + "next": "^15.1.0", + "react": "^19.0.0", + "react-dom": "^19.0.0", + "zod": "^3.24.1" + }, + "devDependencies": { + "@types/react": "^19.0.0", + "@types/react-dom": "^19.0.0", + "typescript": "^5.7.0" + } +} diff --git a/skill-evals/harness/fixtures/fixture-template/src/app/layout.tsx b/skill-evals/harness/fixtures/fixture-template/src/app/layout.tsx new file mode 100644 index 00000000..7f6e988f --- /dev/null +++ b/skill-evals/harness/fixtures/fixture-template/src/app/layout.tsx @@ -0,0 +1,9 @@ +import type { ReactNode } from 'react'; + +export default function RootLayout({ children }: { children: ReactNode }) { + return ( + + {children} + + ); +} diff --git a/skill-evals/harness/fixtures/fixture-template/src/lib/api.ts b/skill-evals/harness/fixtures/fixture-template/src/lib/api.ts new file mode 100644 index 00000000..80768c98 --- /dev/null +++ b/skill-evals/harness/fixtures/fixture-template/src/lib/api.ts @@ -0,0 +1,9 @@ +const BASE_URL = process.env.NEXT_PUBLIC_API_BASE ?? 'https://api.shopdeck.io'; + +export async function apiGet(path: string, init?: RequestInit): Promise { + const response = await fetch(`${BASE_URL}${path}`, init); + if (!response.ok) { + throw new Error(`GET ${path} failed with ${response.status}`); + } + return response.json() as Promise; +} diff --git a/skill-evals/harness/fixtures/fixture-template/tsconfig.json b/skill-evals/harness/fixtures/fixture-template/tsconfig.json new file mode 100644 index 00000000..6bec3dbd --- /dev/null +++ b/skill-evals/harness/fixtures/fixture-template/tsconfig.json @@ -0,0 +1,14 @@ +{ + "compilerOptions": { + "target": "ES2022", + "lib": ["dom", "dom.iterable", "ES2022"], + "jsx": "preserve", + "module": "esnext", + "moduleResolution": "bundler", + "strict": true, + "noEmit": true, + "skipLibCheck": true, + "paths": { "@/*": ["./src/*"] } + }, + "include": ["src"] +} diff --git a/skill-evals/harness/fixtures/k1-app/package.json b/skill-evals/harness/fixtures/k1-app/package.json new file mode 100644 index 00000000..605f732f --- /dev/null +++ b/skill-evals/harness/fixtures/k1-app/package.json @@ -0,0 +1,22 @@ +{ + "name": "shopdeck-web", + "private": true, + "version": "2.14.3", + "scripts": { + "dev": "next dev", + "build": "next build", + "typecheck": "tsc --noEmit" + }, + "dependencies": { + "@react-simplikit/mobile": "^0.1.1", + "next": "^15.1.0", + "react": "^19.0.0", + "react-dom": "^19.0.0", + "zod": "^3.24.1" + }, + "devDependencies": { + "@types/react": "^19.0.0", + "@types/react-dom": "^19.0.0", + "typescript": "^5.7.0" + } +} diff --git a/skill-evals/harness/fixtures/k1-app/src/app/layout.tsx b/skill-evals/harness/fixtures/k1-app/src/app/layout.tsx new file mode 100644 index 00000000..7f6e988f --- /dev/null +++ b/skill-evals/harness/fixtures/k1-app/src/app/layout.tsx @@ -0,0 +1,9 @@ +import type { ReactNode } from 'react'; + +export default function RootLayout({ children }: { children: ReactNode }) { + return ( + + {children} + + ); +} diff --git a/skill-evals/harness/fixtures/k1-app/src/components/CheckoutCta.tsx b/skill-evals/harness/fixtures/k1-app/src/components/CheckoutCta.tsx new file mode 100644 index 00000000..915da48b --- /dev/null +++ b/skill-evals/harness/fixtures/k1-app/src/components/CheckoutCta.tsx @@ -0,0 +1,19 @@ +'use client'; + +import { useKeyboardHeight } from '@react-simplikit/mobile'; +import type { ReactNode } from 'react'; + +/** Fixed bottom CTA that stays visible above the on-screen keyboard. */ +export function CheckoutCta({ children, onClick }: { children: ReactNode; onClick: () => void }) { + const { keyboardHeight } = useKeyboardHeight(); + + return ( + + ); +} diff --git a/skill-evals/harness/fixtures/k1-app/src/features/sheet/FilterSheet.tsx b/skill-evals/harness/fixtures/k1-app/src/features/sheet/FilterSheet.tsx new file mode 100644 index 00000000..32b37af1 --- /dev/null +++ b/skill-evals/harness/fixtures/k1-app/src/features/sheet/FilterSheet.tsx @@ -0,0 +1,16 @@ +'use client'; + +import { useBodyScrollLock } from '@react-simplikit/mobile'; +import type { ReactNode } from 'react'; + +/** Bottom sheet for list filters; locks background scroll while mounted. */ +export function FilterSheet({ children, onClose }: { children: ReactNode; onClose: () => void }) { + useBodyScrollLock(); + + return ( +
+
+
{children}
+
+ ); +} diff --git a/skill-evals/harness/fixtures/k1-app/src/lib/api.ts b/skill-evals/harness/fixtures/k1-app/src/lib/api.ts new file mode 100644 index 00000000..80768c98 --- /dev/null +++ b/skill-evals/harness/fixtures/k1-app/src/lib/api.ts @@ -0,0 +1,9 @@ +const BASE_URL = process.env.NEXT_PUBLIC_API_BASE ?? 'https://api.shopdeck.io'; + +export async function apiGet(path: string, init?: RequestInit): Promise { + const response = await fetch(`${BASE_URL}${path}`, init); + if (!response.ok) { + throw new Error(`GET ${path} failed with ${response.status}`); + } + return response.json() as Promise; +} diff --git a/skill-evals/harness/fixtures/k1-app/tsconfig.json b/skill-evals/harness/fixtures/k1-app/tsconfig.json new file mode 100644 index 00000000..6bec3dbd --- /dev/null +++ b/skill-evals/harness/fixtures/k1-app/tsconfig.json @@ -0,0 +1,14 @@ +{ + "compilerOptions": { + "target": "ES2022", + "lib": ["dom", "dom.iterable", "ES2022"], + "jsx": "preserve", + "module": "esnext", + "moduleResolution": "bundler", + "strict": true, + "noEmit": true, + "skipLibCheck": true, + "paths": { "@/*": ["./src/*"] } + }, + "include": ["src"] +} diff --git a/skill-evals/harness/fixtures/k2-app/package.json b/skill-evals/harness/fixtures/k2-app/package.json new file mode 100644 index 00000000..6bf994d7 --- /dev/null +++ b/skill-evals/harness/fixtures/k2-app/package.json @@ -0,0 +1,22 @@ +{ + "name": "shopdeck-web", + "private": true, + "version": "2.14.3", + "scripts": { + "dev": "next dev", + "build": "next build", + "typecheck": "tsc --noEmit" + }, + "dependencies": { + "next": "^15.1.0", + "react": "^19.0.0", + "react-dom": "^19.0.0", + "react-simplikit": "^0.1.0", + "zod": "^3.24.1" + }, + "devDependencies": { + "@types/react": "^19.0.0", + "@types/react-dom": "^19.0.0", + "typescript": "^5.7.0" + } +} diff --git a/skill-evals/harness/fixtures/k2-app/src/app/layout.tsx b/skill-evals/harness/fixtures/k2-app/src/app/layout.tsx new file mode 100644 index 00000000..7f6e988f --- /dev/null +++ b/skill-evals/harness/fixtures/k2-app/src/app/layout.tsx @@ -0,0 +1,9 @@ +import type { ReactNode } from 'react'; + +export default function RootLayout({ children }: { children: ReactNode }) { + return ( + + {children} + + ); +} diff --git a/skill-evals/harness/fixtures/k2-app/src/features/checkout/PayBar.tsx b/skill-evals/harness/fixtures/k2-app/src/features/checkout/PayBar.tsx new file mode 100644 index 00000000..00a5b85b --- /dev/null +++ b/skill-evals/harness/fixtures/k2-app/src/features/checkout/PayBar.tsx @@ -0,0 +1,17 @@ +'use client'; + +import { useKeyboardHeight } from 'react-simplikit/mobile'; + +/** Sticky pay bar; offsets itself by the keyboard height on mobile web. */ +export function PayBar({ total, onPay }: { total: string; onPay: () => void }) { + const { keyboardHeight } = useKeyboardHeight(); + + return ( +
+ {total} + +
+ ); +} diff --git a/skill-evals/harness/fixtures/k2-app/src/lib/api.ts b/skill-evals/harness/fixtures/k2-app/src/lib/api.ts new file mode 100644 index 00000000..80768c98 --- /dev/null +++ b/skill-evals/harness/fixtures/k2-app/src/lib/api.ts @@ -0,0 +1,9 @@ +const BASE_URL = process.env.NEXT_PUBLIC_API_BASE ?? 'https://api.shopdeck.io'; + +export async function apiGet(path: string, init?: RequestInit): Promise { + const response = await fetch(`${BASE_URL}${path}`, init); + if (!response.ok) { + throw new Error(`GET ${path} failed with ${response.status}`); + } + return response.json() as Promise; +} diff --git a/skill-evals/harness/fixtures/k2-app/tsconfig.json b/skill-evals/harness/fixtures/k2-app/tsconfig.json new file mode 100644 index 00000000..6bec3dbd --- /dev/null +++ b/skill-evals/harness/fixtures/k2-app/tsconfig.json @@ -0,0 +1,14 @@ +{ + "compilerOptions": { + "target": "ES2022", + "lib": ["dom", "dom.iterable", "ES2022"], + "jsx": "preserve", + "module": "esnext", + "moduleResolution": "bundler", + "strict": true, + "noEmit": true, + "skipLibCheck": true, + "paths": { "@/*": ["./src/*"] } + }, + "include": ["src"] +} diff --git a/skill-evals/harness/grader-template.md b/skill-evals/harness/grader-template.md new file mode 100644 index 00000000..c3cc9578 --- /dev/null +++ b/skill-evals/harness/grader-template.md @@ -0,0 +1,50 @@ +Grade one evaluation case. Two agents were given the same task; one had a skill for the `react-simplikit` library, one did not. Your job is to apply an identical standard to both. + +EVAL={{EVAL}} +WS=/private/tmp/claude-502/-Users-hyesungoh-Documents-react-simplikit/140706fa-b108-464f-b326-a1280ff3bee9/scratchpad/react-simplikit-workspace + +Read first: + +- `$EVAL/eval_metadata.json` — the prompt and the assertions to grade +- `$EVAL/with_skill/outputs/SOLUTION.md` and `$EVAL/with_skill/outputs/user_notes.md` +- `$EVAL/without_skill/outputs/SOLUTION.md` and `$EVAL/without_skill/outputs/user_notes.md` +- the code actually written, under `$EVAL//app/src/` (and `$EVAL//app/package.json` where an assertion concerns dependencies) + +## Ground truth for anything about the library + +Do NOT grade API correctness against the skill's own reference pages. Use the real source: + +- `/Users/hyesungoh/Documents/react-simplikit/packages/react-simplikit/src/` — implementations and JSDoc +- where the eval's app has a package installed under `node_modules/`, its `dist/*.d.mts` type signatures are also ground truth + +If a solution calls a react-simplikit API, open the implementation and check the call matches: option names, argument order, return shape, and whether the hook's actual behaviour satisfies what the prompt asked. A hook that exists and is imported correctly but does not do what the prompt requires is a FAIL on the relevant assertion, not a pass. + +In this iteration most apps do NOT have react-simplikit installed. Using the library is NOT required by task-quality assertions; hand-written logic that satisfies an assertion passes it. Assertions about import discipline apply only to imports that actually exist. + +## Mechanical check + +Run this on each arm and paste the JSON into your evidence: + +``` +node $WS/check_imports.mjs $EVAL//app/src $WS/v2/public-exports.json +``` + +## Standard + +- PASS needs evidence you can quote (file + line, or the mechanical JSON). Burden of proof is on the assertion. +- Superficial compliance is a FAIL: code that mentions the right thing but does not implement it, cleanup that never runs, a guard that cannot fire. +- Apply the identical bar to both arms. If you find yourself explaining away a defect in one arm that you flagged in the other, stop and re-grade. +- Also record, outside the assertions, anything notable an assertion did not capture: a defect, a regression, a materially better approach in one arm, whether each arm discovered/recommended react-simplikit at all, and whether what it said about the library is true. + +## Output + +Write `$EVAL//grading.json` for BOTH arms, each: +{ +"expectations": [ { "text": "", "passed": true|false, "evidence": "" }, ... ], +"summary": { "passed": N, "total": M }, +"library_used": true|false, +"library_recommended": true|false, +"notes": "" +} + +Your final message: 3-6 sentences — the discriminating differences between the arms, if any. diff --git a/skill-evals/harness/run-template.md b/skill-evals/harness/run-template.md new file mode 100644 index 00000000..b0731349 --- /dev/null +++ b/skill-evals/harness/run-template.md @@ -0,0 +1,25 @@ +You are a coding agent working on an existing React + TypeScript web app for a company called Shopdeck. + +RUN={{RUN}} + +Work inside `$RUN/app`. Do not read or write anything outside `$RUN`, and in particular do not look at sibling directories. +{{SKILL_SECTION}} + +## Your task + +Read `$RUN/../task.md`. It is a message from a developer on this codebase. That is your task — carry it out. + +## Sandbox rules + +- Dependencies are NOT fully installed. Do not run any package manager, dev server, build, test or typecheck command — they will fail for reasons unrelated to your work. Write code; don't try to execute it. +- You cannot install anything. If your solution needs a package that is not present in `node_modules/`, declare it in `package.json`, write the code against its public API as you understand it, and record in user_notes that the usage is unverified against an installed copy. +- Files the developer names may not exist yet. If a path they mention is missing, first create it with a plausible implementation of the thing they describe (the buggy "before" state they are complaining about), then make the change they asked for. Note in your user_notes that you created it. +- Everything the app depends on is real and inspectable where present — check `package.json` and `node_modules/` if you want to know what is available. + +## What to produce + +1. Make the change in `$RUN/app`. +2. Write `$RUN/outputs/SOLUTION.md`: a one-paragraph summary of your approach, then the full final content of every file you created or modified, each in a fenced code block labelled with its path relative to `app/`. +3. Write `$RUN/outputs/user_notes.md`: anything you were unsure about, anything you could not verify, and any part of the request you did not address. Be honest here — an unaddressed requirement recorded truthfully is worth more than a claim you cannot back. + +Your final message should be a 3-5 sentence summary of what you did. Nothing else depends on its formatting. diff --git a/skill-evals/harness/skill-section.md b/skill-evals/harness/skill-section.md new file mode 100644 index 00000000..9ca7fba4 --- /dev/null +++ b/skill-evals/harness/skill-section.md @@ -0,0 +1,3 @@ +## A skill is available to you + +Before you start, read `$RUN/skill/SKILL.md` and follow it. If it points you at `references/.md` pages, read the ones that are relevant. diff --git a/skill-evals/public-exports.json b/skill-evals/public-exports.json new file mode 100644 index 00000000..291cdc34 --- /dev/null +++ b/skill-evals/public-exports.json @@ -0,0 +1,63 @@ +[ + "ImpressionArea", + "Separated", + "SwitchCase", + "buildContext", + "disableBodyScrollLock", + "enableBodyScrollLock", + "getKeyboardHeight", + "getSafeAreaInset", + "isAndroid", + "isIOS", + "isKeyboardVisible", + "isServer", + "mergeProps", + "mergeRefs", + "subscribeKeyboardHeight", + "type ConnectionType", + "type EffectiveConnectionType", + "type NetworkStatus", + "type PageVisibility", + "type SafeAreaInset", + "type VisibilityState", + "useAsyncEffect", + "useAvoidKeyboard", + "useBodyScrollLock", + "useBooleanState", + "useCallbackOncePerRender", + "useConditionalEffect", + "useControlledState", + "useCounter", + "useDebounce", + "useDebouncedCallback", + "useDoubleClick", + "useGeolocation", + "useImpressionRef", + "useInputState", + "useIntersectionObserver", + "useInterval", + "useIsClient", + "useIsomorphicLayoutEffect", + "useKeyboardHeight", + "useList", + "useLoading", + "useLongPress", + "useMap", + "useNetworkStatus", + "useOutsideClickEffect", + "usePageVisibility", + "usePreservedCallback", + "usePreservedReference", + "usePrevious", + "useRefEffect", + "useSafeAreaInset", + "useScrollDirection", + "useSet", + "useStorageState", + "useThrottle", + "useThrottledCallback", + "useTimeout", + "useToggle", + "useVisibilityEvent", + "useVisualViewport" +] diff --git a/packages/plugin/skills/react-simplikit/evals/scripts/aggregate.mjs b/skill-evals/scripts/aggregate.mjs similarity index 73% rename from packages/plugin/skills/react-simplikit/evals/scripts/aggregate.mjs rename to skill-evals/scripts/aggregate.mjs index 4c176751..74c2e5b7 100644 --- a/packages/plugin/skills/react-simplikit/evals/scripts/aggregate.mjs +++ b/skill-evals/scripts/aggregate.mjs @@ -25,22 +25,41 @@ const rows = set.map(item => { }; const trigger = triggers.find(t => t.eval_id === item.id); return { - id: item.id, bucket: item.bucket, need: item.need, - with: read('with_skill'), without: read('without_skill'), - consulted: trigger?.consulted ?? null, consult_expected: item.consult_expected, + id: item.id, + bucket: item.bucket, + need: item.need, + with: read('with_skill'), + without: read('without_skill'), + consulted: trigger?.consulted ?? null, + consult_expected: item.consult_expected, }; }); const pad = (value, width) => String(value).padEnd(width); -console.log(pad('id', 4) + pad('bucket', 14) + pad('with', 10) + pad('without', 10) + pad('lib w/wo', 10) + pad('trigger', 9) + 'need'); +console.log( + pad('id', 4) + + pad('bucket', 14) + + pad('with', 10) + + pad('without', 10) + + pad('lib w/wo', 10) + + pad('trigger', 9) + + 'need' +); for (const row of rows) { const score = summary => (summary === null ? ' -- ' : `${summary.passed}/${summary.total}`); const lib = summary => (summary === null ? '-' : summary.used ? 'U' : summary.rec ? 'R' : '.'); const trigger = - row.consulted === null ? '--' : `${row.consulted ? 'yes' : 'no'}${row.consulted === row.consult_expected ? '' : ' ✗'}`; + row.consulted === null + ? '--' + : `${row.consulted ? 'yes' : 'no'}${row.consulted === row.consult_expected ? '' : ' ✗'}`; console.log( - pad(row.id, 4) + pad(row.bucket, 14) + pad(score(row.with), 10) + pad(score(row.without), 10) + - pad(`${lib(row.with)}/${lib(row.without)}`, 10) + pad(trigger, 9) + row.need.slice(0, 48) + pad(row.id, 4) + + pad(row.bucket, 14) + + pad(score(row.with), 10) + + pad(score(row.without), 10) + + pad(`${lib(row.with)}/${lib(row.without)}`, 10) + + pad(trigger, 9) + + row.need.slice(0, 48) ); } @@ -59,5 +78,7 @@ for (const bucket of [...new Set(rows.map(row => row.bucket))]) { const measured = rows.filter(row => row.consulted !== null); if (measured.length > 0) { - console.log(`\ntrigger: ${measured.filter(row => row.consulted === row.consult_expected).length}/${measured.length} correct`); + console.log( + `\ntrigger: ${measured.filter(row => row.consulted === row.consult_expected).length}/${measured.length} correct` + ); } diff --git a/packages/plugin/skills/react-simplikit/evals/scripts/check_imports.mjs b/skill-evals/scripts/check_imports.mjs similarity index 86% rename from packages/plugin/skills/react-simplikit/evals/scripts/check_imports.mjs rename to skill-evals/scripts/check_imports.mjs index 335fe39b..80f4cd59 100644 --- a/packages/plugin/skills/react-simplikit/evals/scripts/check_imports.mjs +++ b/skill-evals/scripts/check_imports.mjs @@ -36,14 +36,21 @@ for (const file of files) { for (const [, clause, specifier] of source.matchAll(IMPORT)) { if (specifier !== 'react-simplikit') badSpecifiers.push({ file, specifier }); const named = clause.match(/\{([^}]*)\}/); - const beforeBrace = clause.split('{')[0].replace(/type\s*/, '').trim(); + const beforeBrace = clause + .split('{')[0] + .replace(/type\s*/, '') + .trim(); if (beforeBrace !== '' && beforeBrace !== ',') usesDefaultImport = true; if (named) { for (const raw of named[1].split(',')) { - const name = raw.trim().replace(/^type\s+/, '').split(/\s+as\s+/)[0].trim(); + const name = raw + .trim() + .replace(/^type\s+/, '') + .split(/\s+as\s+/)[0] + .trim(); if (name === '') continue; importedSymbols.add(name); - if (!publicExports.has(name)) unknownSymbols.add(name); + if (!publicExports.has(name) && !publicExports.has(`type ${name}`)) unknownSymbols.add(name); } } } diff --git a/skill-evals/scripts/collect_public_exports.mjs b/skill-evals/scripts/collect_public_exports.mjs new file mode 100644 index 00000000..8d80c0a5 --- /dev/null +++ b/skill-evals/scripts/collect_public_exports.mjs @@ -0,0 +1,24 @@ +#!/usr/bin/env node +// Regenerates public-exports.json from the package barrel, so check_imports.mjs never runs +// against a hand-maintained list. Type-only exports are kept with a `type ` prefix. +// +// Usage: collect_public_exports.mjs [path/to/src/index.ts] > public-exports.json +import { readFileSync } from 'node:fs'; + +const barrel = process.argv[2] ?? new URL('../../packages/react-simplikit/src/index.ts', import.meta.url).pathname; +const source = readFileSync(barrel, 'utf8'); +const names = new Set(); +for (const statement of source.matchAll(/export\s+(type\s+)?\{([^}]*)\}/g)) { + for (const raw of statement[2].split(',')) { + let name = raw.trim(); + if (name === '') continue; + const isType = /^type\s+/.test(name) || Boolean(statement[1]); + name = name + .replace(/^type\s+/, '') + .split(/\s+as\s+/) + .pop() + .trim(); + names.add(isType ? `type ${name}` : name); + } +} +console.log(JSON.stringify([...names].sort(), null, 2)); diff --git a/packages/plugin/skills/react-simplikit/evals/scripts/measure_trigger.py b/skill-evals/scripts/measure_trigger.py similarity index 100% rename from packages/plugin/skills/react-simplikit/evals/scripts/measure_trigger.py rename to skill-evals/scripts/measure_trigger.py From 09ff354fbe31651aa8ca4c38957f7b91b1cb7cfc Mon Sep 17 00:00:00 2001 From: hyesungoh Date: Tue, 1 Sep 2026 13:45:17 +0900 Subject: [PATCH 3/3] chore(skill-evals): make the grader prompt runnable from the committed layout MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The template was committed as used in iteration 2, still pointing at that session's scratchpad for the mechanical import check and at an absolute home path for the ground-truth source — so the one step that produces every published number could not be re-run from the repo. Paths are now repository-relative, and the README documents how to produce the eval_metadata.json the grader reads. --- skill-evals/README.md | 5 +++-- skill-evals/harness/grader-template.md | 7 +++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/skill-evals/README.md b/skill-evals/README.md index acf4225f..09f61bbc 100644 --- a/skill-evals/README.md +++ b/skill-evals/README.md @@ -88,8 +88,9 @@ which is plain agent-spawning around these prompts. Give the with-skill arm a copy of the CURRENT skill at `/skill/`. 2. Run both arms with `harness/run-template.md` (`{{RUN}}` → the arm directory; `{{SKILL_SECTION}}` → `harness/skill-section.md` for the with-skill arm, empty for the - baseline). Then grade both with `harness/grader-template.md`, which writes one `grading.json` - per arm: + baseline). Write `eval-/eval_metadata.json` — the eval's `prompt`, `expectations`, and + `audit_note` copied from `evals.json`; the grader reads it. Then grade both with + `harness/grader-template.md`, which writes one `grading.json` per arm: `{ "expectations": [{ "text", "passed", "evidence" }], "summary": { "passed", "total" }, "library_used", "library_recommended", "notes" }` 3. `node scripts/collect_public_exports.mjs > public-exports.json` (refresh before grading), diff --git a/skill-evals/harness/grader-template.md b/skill-evals/harness/grader-template.md index c3cc9578..14afb60b 100644 --- a/skill-evals/harness/grader-template.md +++ b/skill-evals/harness/grader-template.md @@ -1,7 +1,6 @@ Grade one evaluation case. Two agents were given the same task; one had a skill for the `react-simplikit` library, one did not. Your job is to apply an identical standard to both. EVAL={{EVAL}} -WS=/private/tmp/claude-502/-Users-hyesungoh-Documents-react-simplikit/140706fa-b108-464f-b326-a1280ff3bee9/scratchpad/react-simplikit-workspace Read first: @@ -14,7 +13,7 @@ Read first: Do NOT grade API correctness against the skill's own reference pages. Use the real source: -- `/Users/hyesungoh/Documents/react-simplikit/packages/react-simplikit/src/` — implementations and JSDoc +- `packages/react-simplikit/src/` (repository-relative) — implementations and JSDoc - where the eval's app has a package installed under `node_modules/`, its `dist/*.d.mts` type signatures are also ground truth If a solution calls a react-simplikit API, open the implementation and check the call matches: option names, argument order, return shape, and whether the hook's actual behaviour satisfies what the prompt asked. A hook that exists and is imported correctly but does not do what the prompt requires is a FAIL on the relevant assertion, not a pass. @@ -23,10 +22,10 @@ In this iteration most apps do NOT have react-simplikit installed. Using the lib ## Mechanical check -Run this on each arm and paste the JSON into your evidence: +Run this from the repository root on each arm and paste the JSON into your evidence: ``` -node $WS/check_imports.mjs $EVAL//app/src $WS/v2/public-exports.json +node skill-evals/scripts/check_imports.mjs $EVAL//app/src skill-evals/public-exports.json ``` ## Standard