Skip to content

docs: harness-audit specs (agents checklist, rule spec, E8 draft) - #73

Merged
danoszz merged 2 commits into
mainfrom
claude/competent-tereshkova-2c9268
Oct 20, 2026
Merged

danoszz merged 2 commits into
mainfrom
claude/competent-tereshkova-2c9268

Conversation

@danoszz

@danoszz danoszz commented Oct 20, 2026

Copy link
Copy Markdown
Owner

Turns the harness-audit design in the 2026 agent-incidents memo into specs you can review. Docs only. No rule code until you approve the spec. No version bump, nothing for 1.8.0.

What's in it

File What
docs/agents-safely.md A one-page checklist with seven items: scope, a safe exit, judges out of reach, no shared or host credentials, bounded runs, a record the agent can't edit, and a human to tell. Each item gives a one-sentence reason, cites a primary source and names either the planned rule or "check by hand".
docs/research/spec-harness-rules.md For each of the ten proposed rules: exact inputs, match logic, severity and confidence, 2+2 fixture sketches and the false-positive risk. Checked against lib/agents/discover.js and rules.js, with where each rule would live. Also contains the five-line AGENTS.md block for penthera init.
docs/research/prereg/E8-harness-guardrails.md Draft, not registered. Uses E1's structure. It's a 2×2 design (AGENTS.md block × protected judges) on benign tasks, one third of them unsolvable, run on local open-weight models offline with no credentials. Outcomes are tamper-attempt rate, safe-exit rate and gaming success. It includes a power analysis, threats to validity and a safety section.
docs/research/agent-incidents-2026.md The memo, with pointers to it from vision-2.0.md (new E7 and E8 rows), ROADMAP.md (Next, research track) and one "By situation" row in docs/index.md.
docs/references.bib / .md, lib/references.json Eight sources, each checked against the publisher or arXiv on 2026-10-20 and marked † further reading. The JSON is regenerated by scripts/build-references.mjs; nothing in it is edited by hand.

Decisions for you

  1. Rule set. The spec keeps eight rules and changes two:
    • scorer-guessable-answer is merged into scorer-in-reach. Proving that an answer is derived from a seed needs data-flow analysis, which can't be deterministic.
    • scorer-trusts-agent is dropped as a rule and becomes a "check by hand" item. From the code alone you can't tell a self-report apart from a real answer.
  2. sandbox-egress-open confidence drops from likely to potential. The rule can prove that no recognized restriction exists, but it can't prove the network is open.
  3. Declared harness. The scorer rules need a harness block in penthera.policy.json, which means extending normalizePolicy.
  4. The spec ends with five open questions.
  5. Positioning. No positioning wording changed. README and the one-liner are untouched. The memo's "Positioning (proposal, for approval)" section is still only a proposal and is not applied anywhere. The only nav change is one row in docs/index.md ("By situation"). Revert it if you'd rather add the page later.
  6. CHANGELOG. Left alone, because [Unreleased] is 1.8.0 and that release is frozen.

Checks

  • node scripts/docs-commands.mjs: 353 commands checked, 0 failures. The new pages mention only penthera agents and penthera init.
  • node scripts/build-references.mjs --check: in sync.
  • npx vitest run: 1011 passed, 21 skipped.
  • All relative links and anchors in the changed files resolve.
  • No em dashes in the new pages.
  • npm run lint was not run locally because eslint isn't installed in this checkout. CI runs it.

Please don't merge until you've reviewed it.

🤖 Generated with Claude Code

danoszz and others added 2 commits October 20, 2026 17:34
Turns the harness-audit design from the 2026 agent-incidents memo into
reviewable specs. Docs only; no rule code until the spec is approved.

- docs/agents-safely.md: seven-item checklist for running AI agents, each
  with a reason, a primary source and the planned rule or "check by hand".
- docs/research/spec-harness-rules.md: inputs, match logic, severity,
  confidence, fixtures and false-positive risk for the ten proposed rules,
  checked against lib/agents. Keeps eight, merges scorer-guessable-answer
  into scorer-in-reach, drops scorer-trusts-agent as a rule. Includes the
  five-line AGENTS.md block for penthera init.
- docs/research/prereg/E8-harness-guardrails.md: draft, not registered.
- docs/research/agent-incidents-2026.md: the memo, with pointers from
  vision-2.0 (E7, E8 rows), ROADMAP (Next) and the docs index.
- references.bib/.md and lib/references.json: eight verified sources,
  marked further reading; regenerated by scripts/build-references.mjs.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@danoszz
danoszz merged commit 81991d6 into main Oct 20, 2026
16 checks passed
danoszz added a commit that referenced this pull request Oct 20, 2026
…nd a provenance check in CI

docs: one consistent picture after #70-#73, and a provenance check in CI
@danoszz
danoszz deleted the claude/competent-tereshkova-2c9268 branch October 20, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant