Stop telling the Tester it has HTML it never receives - #136
Open
DavertMik wants to merge 1 commit into
Open
Conversation
The Tester's prompt referenced a <page_html> section in four places, but nothing ever injected one. Its per-iteration context is the accessibility tree and the UI map; HTML reaches it only through tool results — the page diffs that come back with each action — or when it calls context() or xpathCheck() itself. The one path that did inject full page HTML unasked was Pilot's ATTACH_HTML, which dumped simplifiedHtml() straight into the Tester's conversation. Removed; ATTACH_ARIA, ATTACH_SUMMARY and ATTACH_UI_MAP stay. With the invariant real, the Tester needs somewhere to send work that the accessibility tree cannot express. That is interact(), which delegates one step to the Navigator, the agent that does read the whole page. It existed but was invisible: its description read as a generic click() alternative, and the instruction injected every turn listed only the four direct action tools. It now states its purpose and the three cases it covers, and the Tester is pointed at it each iteration. A successful interact() also now counts as browser progress, so a delegated step that leaves the page looking unchanged no longer counts toward the three-strike stall that ends a test. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The Tester's prompt referenced a
<page_html>section in four places, but nothing ever injected one. Its per-iteration context (reinjectContextIfNeeded) is the accessibility tree plus the Researcher's UI map — HTML reaches it only through tool results (the page diffs returned with each action) or when it callscontext()/xpathCheck()itself.The one path that did inject full page HTML unasked was Pilot's
ATTACH_HTML, which pushedsimplifiedHtml()straight into the Tester's conversation. Removed.ATTACH_ARIA,ATTACH_SUMMARYandATTACH_UI_MAPare unchanged.Boundary this settles
With the invariant real, the Tester needs somewhere to send work the accessibility tree cannot express. That is
interact(), which delegates to the Navigator. It already existed but was effectively invisible: its description read as a genericclick()alternative, and the instruction injected every turn listed only the four direct action tools. It now states its purpose and the three cases it covers — direct actions failed, the step needs a sequence of actions, or the element is not in the Tester's context — and the Tester is pointed at it each iteration.Changes
src/ai/tester.ts—<page_html>removed from the context block and two system rules; new rule pairingverify()andinteract()as the tools that read full HTML; per-iteration fallback line forinteract(); two deadcleanupTag('page_html')calls dropped.src/ai/tools.ts—interact()description rewritten around its role;context()and the not-found suggestion no longer reference a<page_html>section.src/ai/pilot.ts—ATTACH_HTMLbranch and its prompt mention removed;interactadded to the Tester tool list Pilot validates against, which was missing it.src/ai/rules.ts—locatorStrategyRulereferenced<aria>/<html>sections that exist under neither name; reworded, still true for Navigator.CLAUDE.md— Pilot section corrected.Behavior change worth a look
DELEGATED_ACTION_TOOLS = ['interact']now feedsshouldStopForStalledExecution. Previously a successfulinteract()that left the state hash unchanged counted as no browser progress, and three of those ended the test. Now that the prompt actively pushes the Tester towardinteract(), that was a trap worth closing — but it is the one change here that alters runtime behavior rather than prompt text.Not fixed here
A failed
interact()never lands intasknotes: the note loop keys onexecution.input?.explanationandinteract's input field isinstruction. Pre-existing, and Pilot still sees the failure throughformatActions.Testing
bun test tests/integration/80 pass / 0 fail,bun test tests/unit/1043 pass / 0 fail, format and lint clean. Prompt-level change — worth regression coverage before merge, your call on the label.🤖 Generated with Claude Code