Skip to content

feat: action results name the element they touched (3.1.0) - #11

Open
sblattj wants to merge 2 commits into
mainfrom
feat/describe-acted-element
Open

sblattj wants to merge 2 commits into
mainfrom
feat/describe-acted-element

Conversation

@sblattj

@sblattj sblattj commented Oct 26, 2026

Copy link
Copy Markdown
Owner

Why

A transcript of click {uid: 9} tells a reader nothing about what was clicked. Now the result says it:

{ "clicked": true, "element": "button \"Add new item\"", "x": 85.66, "y": 95.69 }

What

  • click, hover, fill, type_text, upload_file return element: a snapshot-style role "name" string.
  • fill_form returns elements (one per field, null if unreadable); drag returns element (source) and target_element when to is an element.
  • The description is read before the action (a click can navigate or unmount the node; fill reports the label, not the new value), bounded to 2s, and omitted on any failure. It never masks the action's own result or error.
  • The Firefox snapshot walker's role/accname heuristics are lifted into a shared ACCNAME_HELPERS_SOURCE; the walker's generated source is identical modulo whitespace (checked against main).
  • Additive only, so 3.1.0. CHANGELOG, toolDocs (click, fill) and the skill's "common mistakes" line updated.

Caveat

On Chrome the snapshot comes from the native AX tree while element uses the DOM heuristic. They agree for buttons, links, tabs and labelled inputs; exotic accname cases can differ.

Verification

  • bun run typecheck clean; bun test 1040 pass, 0 fail (14 new in test/describe-element.test.ts).
  • bun run smoke 13/13 against a live Chromium.
  • Live, in a throwaway tab:
    • click --uid 9 → button "Add new item" (handler fired once)
    • click --uid 8 → tab "Details"
    • click --selector #plain → div "Plain row text"
    • fill --uid 11 → textbox "Email"
    • hover --selector a → link "Next page"
    • a bad uid still errors exactly as before (uid does not resolve: 99999)
  • Not run live: Firefox/BiDi (it goes through the same callOnElement path; unit-tested only).

click, hover, fill, type_text and upload_file now return an `element`
field like `button "Save"`, read in the page before the action, so tool
output says what was clicked instead of a bare uid. fill_form returns
`elements`, drag returns `element` plus `target_element`. Best-effort:
bounded to 2s, omitted on failure, never masks the action's own error.

The snapshot walker's role/accname heuristics are lifted into a shared
ACCNAME_HELPERS_SOURCE so both paths name elements the same way.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

prxref automated review: Approved

PR: feat: action results name the element they touched (3.1.0) · files reviewed: 8

🟥 0 error · 🟧 1 warning · 🔍 0 spec · ⬜ 0 outofscope

  • 🟧 CHANGELOG.md:10 — 3.1.0 is dated before the 3.0.3 release

Reviewed by prxref · model=z-ai/glm-5.3-flash · 22434 tok · 12.6s · seed=1722819217

…lement

# Conflicts:
#	CHANGELOG.md
#	package.json
#	src/version.ts
Comment thread CHANGELOG.md

## [Unreleased]

## [3.1.0] - 2026-10-26

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 🟧 [WARNING] 3.1.0 is dated before the 3.0.3 release (CHANGELOG.md:10)

The new entry is ## [3.1.0] - 2026-10-26 while the entry below it reads ## [3.0.3] - 2026-10-29; the changelog now lists the newer version with an earlier date, which will mislead release tooling and readers. One of the two dates is wrong.


Reviewed by prxref · model=z-ai/glm-5.3-flash

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant