Repository navigation
docs: harness-audit specs (agents checklist, rule spec, E8 draft) - #73
Merged
Merged
Conversation
Turns the harness-audit design from the 2026 agent-incidents memo into reviewable specs. Docs only; no rule code until the spec is approved. - docs/agents-safely.md: seven-item checklist for running AI agents, each with a reason, a primary source and the planned rule or "check by hand". - docs/research/spec-harness-rules.md: inputs, match logic, severity, confidence, fixtures and false-positive risk for the ten proposed rules, checked against lib/agents. Keeps eight, merges scorer-guessable-answer into scorer-in-reach, drops scorer-trusts-agent as a rule. Includes the five-line AGENTS.md block for penthera init. - docs/research/prereg/E8-harness-guardrails.md: draft, not registered. - docs/research/agent-incidents-2026.md: the memo, with pointers from vision-2.0 (E7, E8 rows), ROADMAP (Next) and the docs index. - references.bib/.md and lib/references.json: eight verified sources, marked further reading; regenerated by scripts/build-references.mjs. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Turns the harness-audit design in the 2026 agent-incidents memo into specs you can review. Docs only. No rule code until you approve the spec. No version bump, nothing for 1.8.0.
What's in it
docs/agents-safely.mddocs/research/spec-harness-rules.mdlib/agents/discover.jsandrules.js, with where each rule would live. Also contains the five-lineAGENTS.mdblock forpenthera init.docs/research/prereg/E8-harness-guardrails.mdAGENTS.mdblock × protected judges) on benign tasks, one third of them unsolvable, run on local open-weight models offline with no credentials. Outcomes are tamper-attempt rate, safe-exit rate and gaming success. It includes a power analysis, threats to validity and a safety section.docs/research/agent-incidents-2026.mdvision-2.0.md(new E7 and E8 rows),ROADMAP.md(Next, research track) and one "By situation" row indocs/index.md.docs/references.bib/.md,lib/references.jsonscripts/build-references.mjs; nothing in it is edited by hand.Decisions for you
scorer-guessable-answeris merged intoscorer-in-reach. Proving that an answer is derived from a seed needs data-flow analysis, which can't be deterministic.scorer-trusts-agentis dropped as a rule and becomes a "check by hand" item. From the code alone you can't tell a self-report apart from a real answer.sandbox-egress-openconfidence drops fromlikelytopotential. The rule can prove that no recognized restriction exists, but it can't prove the network is open.harnessblock inpenthera.policy.json, which means extendingnormalizePolicy.docs/index.md("By situation"). Revert it if you'd rather add the page later.[Unreleased]is 1.8.0 and that release is frozen.Checks
node scripts/docs-commands.mjs: 353 commands checked, 0 failures. The new pages mention onlypenthera agentsandpenthera init.node scripts/build-references.mjs --check: in sync.npx vitest run: 1011 passed, 21 skipped.npm run lintwas not run locally because eslint isn't installed in this checkout. CI runs it.Please don't merge until you've reviewed it.
🤖 Generated with Claude Code