What this enables
The macOS desktop agent can enter text, use keyboard shortcuts, scroll, and complete a document's Save As flow. Saved text is checked on disk before the task succeeds.
Feature: #15. The text-input part is submitted in #27. The screenshot and local Cua-S1 4B work is covered by RFC #30.
Design
s1a/agents/desktop.py accepts task text, target fields, allowed keys and scroll directions, and allowed dialog titles.s1a/desktop/env.py builds actions from the current window observation, executes a selected action, and reads the window again. Text entry supports insertion and whole-field replacement, with field readback and an optional saved-file check.s1a/desktop/driver.py uses the existing Rust Cua Driver. Actions identify the selected process, window, and session. Editable targets use current element tokens; stale or refused input fails the action. Foreground keyboard delivery is explicit.- Save As follows explicitly named dialogs owned by the document's process. Named field values support a filename separate from the document body. Cancelling and reopening the dialog refreshes the field state.
The task supplies the text and allowed actions. The decision model chooses an offered action. Dry runs preserve the existing preview behaviour.
Implementation status
| Feature |
Current status |
| Text insertion/replacement, field readback, saved-file verification |
#27, open for review |
| Keys, shortcuts, scrolling, and the TextEdit runner |
Implemented and tested locally |
| Same-process Save As dialogs and named filename input |
Implemented and tested locally |
Verification
Added test_desktop_actions.py and test_desktop_dialogs.py, and extended the driver, environment, and CLI tests. They cover text replacement, readback, saved-file checks, keys, scrolling, dialog transitions, stale targets, refused actions, and dry runs.
Text input was tested on macOS 26.2 arm64 with Cua Driver 0.30.1:
| Task |
Action selection |
Passed |
Total time |
Actions per trial |
| Enter Body and save in the native test app |
Local Laya 0.3.20 |
3/3 |
27.3 s |
4 |
| Replace Chinese multiline text and save |
Fixed action plan |
1/1 |
4.9 s |
2 |
Each trial checks the saved file independently. Episode time excludes model loading. The Laya run uses no fixed plan or API key.
The core + dev suite passes with 515 passed, 41 skipped on #27 and 524 passed, 41 skipped for the local implementation including save dialogs. Ruff and ty pass on both; #27 also passes the lock check, package build, shell syntax, and CLI/MCP smoke checks.
How to run
Use the project checkout for #27. Requires macOS, uv, Swift command-line tools, and Cua Driver with Accessibility and Screen Recording permissions. Close any older S1A document fixture before starting. From the project directory:
S1A_PROJECT_DIR="$(pwd -P)"
uv sync --project "$S1A_PROJECT_DIR" --extra dev --extra laya
export CUA_DRIVER_BIN=/Applications/CuaDriver.app/Contents/MacOS/cua-driver
S1A="$S1A_PROJECT_DIR/.venv/bin/s1a" \
bash "$S1A_PROJECT_DIR/evals/desktop/run_local.sh" --episodes 3
The script builds the native app, runs local Laya, and checks the saved file. #27 also includes a fixed-plan command for checking execution.
What this enables
The macOS desktop agent can enter text, use keyboard shortcuts, scroll, and complete a document's Save As flow. Saved text is checked on disk before the task succeeds.
Feature: #15. The text-input part is submitted in #27. The screenshot and local Cua-S1 4B work is covered by RFC #30.
Design
s1a/agents/desktop.pyaccepts task text, target fields, allowed keys and scroll directions, and allowed dialog titles.s1a/desktop/env.pybuilds actions from the current window observation, executes a selected action, and reads the window again. Text entry supports insertion and whole-field replacement, with field readback and an optional saved-file check.s1a/desktop/driver.pyuses the existing Rust Cua Driver. Actions identify the selected process, window, and session. Editable targets use current element tokens; stale or refused input fails the action. Foreground keyboard delivery is explicit.The task supplies the text and allowed actions. The decision model chooses an offered action. Dry runs preserve the existing preview behaviour.
Implementation status
Verification
Added
test_desktop_actions.pyandtest_desktop_dialogs.py, and extended the driver, environment, and CLI tests. They cover text replacement, readback, saved-file checks, keys, scrolling, dialog transitions, stale targets, refused actions, and dry runs.Text input was tested on macOS 26.2 arm64 with Cua Driver 0.30.1:
Each trial checks the saved file independently. Episode time excludes model loading. The Laya run uses no fixed plan or API key.
The core + dev suite passes with 515 passed, 41 skipped on #27 and 524 passed, 41 skipped for the local implementation including save dialogs. Ruff and ty pass on both; #27 also passes the lock check, package build, shell syntax, and CLI/MCP smoke checks.
How to run
Use the project checkout for #27. Requires macOS,
uv, Swift command-line tools, and Cua Driver with Accessibility and Screen Recording permissions. Close any older S1A document fixture before starting. From the project directory:The script builds the native app, runs local Laya, and checks the saved file. #27 also includes a fixed-plan command for checking execution.