Skip to content

[Feat] Add verified text input to the desktop agent - #27

Open
QianCyrus wants to merge 1 commit into
ThinkFlowLab:mainfrom
QianCyrus:feat/macos-desktop-actions
Open

QianCyrus wants to merge 1 commit into
ThinkFlowLab:mainfrom
QianCyrus:feat/macos-desktop-actions

Conversation

@QianCyrus

@QianCyrus QianCyrus commented Sep 27, 2026 •

Copy link
Copy Markdown

What this enables

The desktop agent can now enter supplied text into a macOS application, click Save, and check that the file contains the expected text.

It can insert text at the current selection or replace a field's entire contents, including Chinese and multiple lines. After input, it reads the field again to confirm the result. An optional file check requires a fresh write and matching content before the task succeeds.

The implementation extends the existing desktop CLI, window environment, and Cua Driver adapter. Part of #15; RFC #26.

macOS results

Rerun on this branch with macOS 26.2 arm64 and Cua Driver 0.30.1:

Task Action selection Passed Total time Actions per trial
Enter Body and save in the native test app Local Laya 0.3.20 3/3 27.3 s 4
Replace Chinese multiline text and save Fixed action plan 1/1 4.9 s 2

Each successful trial checks the saved file independently. The Laya run uses no fixed plan, Jev, or chat-model calls. Episode time excludes model loading.

Tests

Added test_desktop_actions.py for text entry, replacement, readback, and saved-file verification.

Extended test_desktop_driver.py, test_agents_desktop.py, and test_desktop_env.py for window binding, native field identifiers, CLI configuration, dry runs, and stale or refused actions.

Core + dev suite: 515 passed, 41 skipped. Ruff, ty, lock check, package build, shell syntax, and CLI/MCP smoke passed.

How to run

Requires macOS, uv, the Swift command-line tools, and Cua Driver with Accessibility and Screen Recording permissions. These commands use the signed driver at /Applications/CuaDriver.app/Contents/MacOS/cua-driver. Run from the current project checkout. Close any older S1A document fixture before starting.

S1A_PROJECT_DIR="$(pwd -P)"
uv sync --project "$S1A_PROJECT_DIR" --extra dev --extra laya
bash "$S1A_PROJECT_DIR/evals/desktop/build_fixture.sh" /tmp/S1ADocumentFixture.app
export CUA_DRIVER_BIN=/Applications/CuaDriver.app/Contents/MacOS/cua-driver

Run a fixed input-and-save sequence to check execution:

uv run --project "$S1A_PROJECT_DIR" --no-sync s1a run desktop \
  --model rule --rethink off --episodes 1 \
  --app S1ADocumentFixture --app-path /tmp/S1ADocumentFixture.app \
  --window-title "S1A Document Fixture" --goal "Enter the text in Body and save it" --expect Saved \
  --text "Hello from the desktop agent." --text-target Body --text-mode replace \
  --verify-file /tmp/s1a-desktop-fixture.txt --clear Clear --plan 'type:Body,Save' --execute
cat /tmp/s1a-desktop-fixture.txt

To let local Laya select the actions instead, run:

S1A="$S1A_PROJECT_DIR/.venv/bin/s1a" \
  bash "$S1A_PROJECT_DIR/evals/desktop/run_local.sh" --episodes 3

@QianCyrus
QianCyrus force-pushed the feat/macos-desktop-actions branch from 4d9994e to 49ad3ac Compare September 27, 2026 12:01
@QianCyrus QianCyrus changed the title [Feat] Add window-scoped desktop text, key, and scroll actions [Feat] Add desktop input, scoped dialogs, and screenshot targets Sep 27, 2026
@QianCyrus
QianCyrus force-pushed the feat/macos-desktop-actions branch 2 times, most recently from fd089c2 to 9e1b350 Compare September 27, 2026 16:31
@QianCyrus QianCyrus changed the title [Feat] Add desktop input, scoped dialogs, and screenshot targets [Feat] Add desktop text input, shortcuts, save dialogs, and screenshot targets Sep 27, 2026
@QianCyrus
QianCyrus marked this pull request as ready for review September 27, 2026 16:35
@QianCyrus
QianCyrus force-pushed the feat/macos-desktop-actions branch from 9e1b350 to 520c0f9 Compare September 27, 2026 16:51
@QianCyrus QianCyrus changed the title [Feat] Add desktop text input, shortcuts, save dialogs, and screenshot targets [Feat] Add verified text input to the desktop agent Sep 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant