Report what the app said and did, not just how the page moved

[DavertMik]

A click that fires a request the server rejects looks like a success today. pageDiff describes DOM and accessibility-tree movement only, so the toast carrying the server's error is either buried in raw markup — collapseHtmlParts replaces every part once the diff exceeds 8000 chars or 8 parts, which a real app re-render does routinely — or stripped outright by conversation compaction. ariaChanges cannot carry it at all: nodeToEntry drops every non-interactive role, so an alert never reaches the model.

Meanwhile the failed request reaches Pilot only as a session-wide network errors: POST /api/… → 400 in <state>, with no page context and no link to the action that caused it. A bare 400 reads as a missing value. In trace ecad7a69 that is exactly what happened: Pilot twice told the tester to fill in a Title that had been filled five minutes earlier, and the run spent 21 clicks and 8 minutes on a Save the server had already refused.

What changes

Three new fields on PageDiff, all scoped to the action that produced them:

• messages — text the app put on the page in response. Live-region content first (role=alert|alertdialog|status|log, aria-live, <output>), then any other text that appeared. The second tier is what catches notifications built with no ARIA at all, which is the common case. Built from htmlDiff.added, which already computed it and threw it away.
• requests — same-origin xhr/fetch made during the action, with status. captureMainDocumentResponse already watched the action's exact window for the main document status; it now records these too, so there is no second listener and no interaction with requestStore.clear().
• consoleErrors — what the page logged while the action ran. grabBrowserLogs() drains its store, so this was already per-action; nothing surfaced it.

A 4xx/5xx now leads the tool's suggestion and points at the message the user was shown, instead of leaving the model to read the click as successful and try again.

Pilot reads the same evidence through formatActions, which analyzeProgress already sends — so it arrives attributed to the action, and narrowed to failure signal: rejected requests only, the first two messages, one console error. The review that misfired would have read:

[SUCCESS] ACTION click: Save
   requests: POST /api/zephyr/plans → 400
   messages: Operation against a key holding the wrong kind of value
   console: Ember Data Request POST /api/zephyr/plans returned a 400

Bug found on the way

Browser console messages were never recorded. grabBrowserLogs() returns Playwright ConsoleMessage objects whose type and text are methods, and captureBrowserLogs filtered with ['info','error','warning','warn'].includes(log.type) — comparing a function against a list of strings, always false. Every entry was discarded, which is why console errors read none for a whole session while the page was logging its own failure. Both fields are normalized at capture; tests/unit/action-browser-logs.test.ts locks it down.

Deliberately out of scope

• htmlDiff runs only when the URL is unchanged, so a flash message shown after a redirect is not picked up — context re-injection already covers that case.
• Requests that never get a response (connection refused, aborted) fire requestfailed, not response, so they carry no status and do not appear.
• Cross-origin calls are filtered out, matching XhrCapture, to keep third-party analytics out of every tool result.

Testing

bun test tests/unit (1037 pass) and bun test tests/integration (81 pass), plus format:check and lint, all verified against this commit in a clean worktree. New coverage in page-diff-evidence, pilot-action-evidence and action-browser-logs.

This changes what the models see on every action, so it is worth a regression label before merging — I have not applied one.

🤖 Generated with Claude Code

[DenysKuchma]

This puts any newly appeared page text into messages, not just actual app notifications. After normal navigation, headings and content from the new page may be reported to Pilot as feedback from the action. I’d only include generic appeared text when the URL hasn’t changed and handle live-region messages separately. We should also add a test for navigation between pages

[DenysKuchma]

The limit currently applies only to successful requests, while failed requests can keep growing without a cap. Since this data is added to the model context, it would be safer to limit the total list while giving failed requests priority