[Feat] Add local Cua-S1 4B and screenshot target selection

.env.example

@@ -39,7 +39,17 @@ MODEL_NAME=google/gemini-2.5-flash
 # CUA_S1_SUBFOLDER=text                     # the text-only checkpoint; the window is 256 bytes of state
 # CUA_S1_DEVICE=auto                        # cpu, cuda, mps
 
-# ---- Cua Driver (the hands of the desktop agents, e.g. calculator; install: https://cua.ai/docs/cua-driver) ----
+# ---- Optional Cua-S1 4B ----
+# uv sync --extra cua-four-b (separate from the Laya demo).
+# CUA_S1_VARIANT=4b
+# CUA_S1_MODALITY=multimodal
+# CUA_S1_BASE_MODEL=Qwen/Qwen3.5-4B  # or a local model directory
+# CUA_S1_CHECKPOINT=cua-ai/cua-s1-4b-0.2  # or a directory with text/ and multimodal/
+# CUA_S1_DEVICE=mps
+# CUA_S1_DTYPE=bfloat16
+# HF_DEACTIVATE_ASYNC_LOAD=1  # synchronous Transformers loading on Apple MPS
+
+# ---- Cua Driver (the hands of the desktop agents; install: https://cua.ai/docs/cua-driver) ----
 # CUA_DRIVER_BIN=/usr/local/bin/cua-driver  # when cua-driver is not on PATH
 # CUA_DRIVER_BIN=C:/path/to/cua-driver.exe  # Windows: full path to the installed executable
 # CUA_DRIVER_PERMISSION_MODE=standard       # the driver's mode; bounded needs a capability manifest

CONTRIBUTING.md

@@ -72,8 +72,9 @@ Everything outside `openjiuwen` is an extra. An agent whose extra is missing say
 | `laya` | laya (torch, transformers) | `--model laya` on every agent and on `decide` and `probe`: Laya in process, no Jev key; the checkpoint downloads into the Hugging Face cache (`HF_HOME`) on first use |
 | `cua` | cua-s1 (torch), huggingface-hub | `--model cua` on tool and browser agents and on `decide` and `probe`: Cua-S1 Nano in process; the 3 MB checkpoint downloads into the Hugging Face cache (`HF_HOME`) on first use |
 | `dev` | pytest, pytest-asyncio, ruff, ty | the test suite, `scripts/smoke.sh` and the lint and type checks |
+| `cua-four-b` | the `cua` extra, transformers 5, peft, torchvision, pillow | optional local Cua-S1 4B; set `CUA_S1_VARIANT=4b`, and `CUA_S1_MODALITY=text` or `multimodal` |
 
-`uv sync --all-extras` installs all seven. The CLI runs from a checkout; a wheel install (`uv tool install`,
+`uv sync --all-extras` installs all eight. The CLI runs from a checkout; a wheel install (`uv tool install`,
 `pip install`) is unsupported, because the data folders (`evals/2048`, `evals/millionaire`, `evals/labelled`) sit
 next to the package; the command refuses to start outside a checkout with one line on stderr. Runs, logs and
 results go under the checkout (`runs/`, `evals/results`), or under `S1A_HOME` when that variable names another

docs/agents.md

@@ -33,6 +33,11 @@ models behind them: [architecture.md](architecture.md#models).
 `--app`, `--goal`, `--expect`, `--execute`, `--plan` and `--clear`; `ticket_router` takes `--dataset` and
 `--batch-size`; `injection_guard` takes `--labelled-set`. The four games take no flag of their own.
 
+`--pixel-target KEY=X,Y` offers named points in screenshot fractions. Local Cua-S1 4B multimodal can select
+among them when controls have no accessibility elements. Each click carries its screenshot capture ID.
+Install the `cua-four-b` extra and set `CUA_S1_VARIANT=4b`, `CUA_S1_MODALITY=multimodal`.
+See [the macOS visual task](../evals/desktop/visual.md) for setup and verification commands.
+
 ## Allrecipes
 
 `allrecipes` is the first Allrecipes task of the [WebVoyager](https://github.com/MinorJerry/WebVoyager) task set

evals/desktop/build_visual_fixture.sh

@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+app="${1:-/tmp/S1AVisualFixture.app}"
+root="$(cd "$(dirname "$0")/../.." && pwd)"
+mkdir -p "$app/Contents/MacOS"
+swiftc -O "$root/evals/desktop/visual_fixture.swift" -framework AppKit -o "$app/Contents/MacOS/S1AVisualFixture"
+cat > "$app/Contents/Info.plist" <<'PLIST'
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0"><dict>
+  <key>CFBundleExecutable</key><string>S1AVisualFixture</string>
+  <key>CFBundleIdentifier</key><string>org.thinkflowlab.s1a.visual-fixture</string>
+  <key>CFBundleName</key><string>S1AVisualFixture</string>
+  <key>CFBundlePackageType</key><string>APPL</string>
+  <key>LSMinimumSystemVersion</key><string>14.0</string>
+</dict></plist>
+PLIST
+printf '%s\n' "$app"

evals/desktop/visual.md

@@ -0,0 +1,45 @@
+# Screenshot target selection on macOS
+
+The native test app draws Save and Cancel tiles without accessibility children. Reset swaps their positions.
+The agent receives the current screenshot and chooses between two configured points. A successful click
+shows Saved; the app also appends the selected tile to a temporary file for an independent check.
+
+The optional `cua-four-b` extra loads the official Cua-S1 4B scorer locally. Set `CUA_S1_VARIANT=4b` and
+`CUA_S1_MODALITY=multimodal` for screenshots, or `CUA_S1_MODALITY=text` for text observations. Existing
+`--model cua` runs keep using Nano unless the variant is set. The 4B scorer supports up to 26 choices.
+
+## Run the visual task
+
+Requires macOS, `uv`, Swift command-line tools, and Cua Driver with Accessibility and Screen Recording
+permissions. Keep the Mac unlocked. Close an older S1A visual fixture before rebuilding it.
+Run these commands from the current project directory:
+
+```bash
+S1A_PROJECT_DIR="$(pwd -P)"
+uv sync --project "$S1A_PROJECT_DIR" --extra cua-four-b
+bash "$S1A_PROJECT_DIR/evals/desktop/build_visual_fixture.sh" /tmp/S1AVisualFixture.app
+export CUA_DRIVER_BIN=/Applications/CuaDriver.app/Contents/MacOS/cua-driver
+export CUA_S1_VARIANT=4b CUA_S1_MODALITY=multimodal
+export CUA_S1_DEVICE=mps CUA_S1_DTYPE=bfloat16 HF_DEACTIVATE_ASYNC_LOAD=1
+export PYTORCH_ENABLE_MPS_FALLBACK=1
+: > /tmp/s1a-visual-target.txt
+uv run --project "$S1A_PROJECT_DIR" --no-sync s1a run desktop \
+  --model cua --rethink off --episodes 4 --max-steps 1 \
+  --app S1AVisualFixture --app-path /tmp/S1AVisualFixture.app --expect Saved \
+  --window-title "S1A Visual Fixture" \
+  --goal "Click the tile labelled Save in the screenshot" \
+  --pixel-target left=0.27,0.51 --pixel-target right=0.73,0.51 \
+  --clear Reset --execute
+uv run --project "$S1A_PROJECT_DIR" --no-sync python -c \
+  'from pathlib import Path; rows = Path("/tmp/s1a-visual-target.txt").read_text().splitlines(); assert rows == ["Save selected"] * 4, rows; print("Verified all four clicks")'
+```
+
+The base model and adapter download on first use. `CUA_S1_BASE_MODEL` and `CUA_S1_CHECKPOINT` can point
+to downloaded directories. Omit `--execute` to preview a decision without clicking or resetting the app.
+
+For a fixed-action check, use `--model rule --episodes 1 --plan pixel:right` on a newly launched fixture.
+Its first Reset moves Save to the right; later resets alternate sides. `--pixel-target` coordinates are fractions
+of the captured window. Every click carries its capture ID so the driver can reject a stale target.
+
+The temporary selection file and ignored run outputs are generated locally. Screenshot bytes are passed
+to the model separately from the serialized observation state.

evals/desktop/visual_fixture.swift

@@ -0,0 +1,86 @@
+import AppKit
+import Foundation
+
+private let output = URL(fileURLWithPath: "/tmp/s1a-visual-target.txt")
+
+// The tiles are drawn pixels, deliberately absent from the accessibility tree.
+final class TileCanvas: NSView {
+    var saveOnLeft = true
+    var selected: ((Bool) -> Void)?
+    override var isFlipped: Bool { true }
+    private let tiles = [NSRect(x: 60, y: 140, width: 200, height: 100),
+                         NSRect(x: 340, y: 140, width: 200, height: 100)]
+
+    override func draw(_ dirtyRect: NSRect) {
+        NSColor.windowBackgroundColor.setFill()
+        bounds.fill()
+        let heading: NSString = "Choose the tile labelled Save"
+        heading.draw(at: NSPoint(x: 60, y: 55), withAttributes: [.font: NSFont.systemFont(ofSize: 24),
+                                                               .foregroundColor: NSColor.labelColor])
+        for (index, tile) in tiles.enumerated() {
+            NSColor.controlBackgroundColor.setFill()
+            NSBezierPath(roundedRect: tile, xRadius: 12, yRadius: 12).fill()
+            let label: NSString = (index == 0) == saveOnLeft ? "Save" : "Cancel"
+            let attributes: [NSAttributedString.Key: Any] = [.font: NSFont.systemFont(ofSize: 28),
+                                                            .foregroundColor: NSColor.labelColor]
+            let size = label.size(withAttributes: attributes)
+            label.draw(at: NSPoint(x: tile.midX - size.width / 2, y: tile.midY - size.height / 2),
+                       withAttributes: attributes)
+        }
+    }
+
+    override func mouseDown(with event: NSEvent) {
+        let point = convert(event.locationInWindow, from: nil)
+        if let index = tiles.firstIndex(where: { $0.contains(point) }) {
+            selected?((index == 0) == saveOnLeft)
+        }
+    }
+}
+
+final class VisualFixture: NSObject, NSApplicationDelegate {
+    private let window = NSWindow(contentRect: NSRect(x: 300, y: 300, width: 600, height: 400),
+                                  styleMask: [.titled, .closable, .miniaturizable], backing: .buffered, defer: false)
+    private let canvas = TileCanvas(frame: NSRect(x: 0, y: 0, width: 600, height: 400))
+    private let status = NSTextView(frame: .zero)
+
+    func applicationDidFinishLaunching(_ notification: Notification) {
+        window.title = "S1A Visual Fixture"
+        canvas.setAccessibilityElement(false)
+        status.isEditable = false
+        status.isSelectable = false
+        status.drawsBackground = false
+        status.string = "Waiting"
+        status.setAccessibilityLabel("Status")
+        status.frame = NSRect(x: 300, y: 315, width: 230, height: 35)
+        let reset = NSButton(title: "Reset", target: self, action: #selector(resetTask))
+        reset.frame = NSRect(x: 60, y: 315, width: 120, height: 35)
+        canvas.addSubview(reset)
+        canvas.addSubview(status)
+        canvas.selected = { correct in
+            do {
+                let previous = (try? String(contentsOf: output, encoding: .utf8)) ?? ""
+                let result = correct ? "Save selected\n" : "Cancel selected\n"
+                try (previous + result).write(to: output, atomically: true, encoding: .utf8)
+                self.status.string = correct ? "Saved" : "Wrong tile"
+            } catch {
+                self.status.string = "Write failed"
+            }
+        }
+        window.contentView = canvas
+        window.makeKeyAndOrderFront(nil)
+        NSApp.activate(ignoringOtherApps: true)
+    }
+
+    @objc private func resetTask() {
+        status.string = "Waiting"
+        canvas.saveOnLeft.toggle()
+        canvas.needsDisplay = true
+    }
+
+    func applicationShouldTerminateAfterLastWindowClosed(_ sender: NSApplication) -> Bool { true }
+}
+
+let app = NSApplication.shared
+let delegate = VisualFixture()
+app.delegate = delegate
+app.run()

pyproject.toml

@@ -45,6 +45,7 @@ cua = [  # Cua-S1 Nano behind --model cua; pinned to the Cua PR that ships the c
     "cua-s1 @ git+https://github.com/trycua/cua.git@aea61b6eb97e2d8c0f6f71eb804e5769fe910af4#subdirectory=libs/cua-s1/python",
     "huggingface-hub>=0.24",
 ]
+cua-four-b = ["system1-agents[cua]", "transformers>=5.10.1,<6", "peft>=0.21,<1", "torchvision>=0.15", "pillow>=10"]
 dev = ["pytest>=8", "pytest-asyncio>=0.24", "ruff>=0.6", "ty>=0.0.83"]
 
 [build-system]
@@ -75,6 +76,7 @@ include = [
     "s1a/agents/alfworld.py",
     "s1a/agents/blackjack.py",
     "s1a/decision_models/cua.py",
+    "s1a/decision_models/cua_four_b.py",
     "s1a/decision_models/laya.py",
 ]
 

s1a/agents/desktop.py

@@ -18,12 +18,15 @@
 
 import argparse
 import asyncio
+import math
+import os
 import sys
+import uuid
 from contextlib import asynccontextmanager
 from typing import Any, AsyncIterator
 
 from s1a.desktop.driver import CuaDriver, Snapshot, driver_from_env, opened
-from s1a.desktop.env import ABSTAIN, DONE, WindowEnv, clickable
+from s1a.desktop.env import ABSTAIN, DONE, WindowEnv, clickable, observable
 from s1a.spec import Budget, Series, ToolAgentSpec
 
 RULES = (
@@ -38,22 +41,42 @@ def shows(snapshot: Snapshot, text: str) -> bool:
     wanted = text.strip()
     if not wanted:
         return False
-    return any((e.value.strip() == wanted or e.label.strip() == wanted) for e in snapshot.elements if not clickable(e))
+    return any(
+        (e.value.strip() == wanted or e.label.strip() == wanted)
+        for e in snapshot.elements
+        if observable(e) and not clickable(e)
+    )
 
 
 def parse_plan(text: str) -> tuple[tuple[str, ...], ...]:
     """``"1,2,Multiply|×"`` to ``(("1",), ("2",), ("Multiply", "×"))``."""
     return tuple(tuple(v.strip() for v in step.split("|")) for step in text.split(",") if step.strip())
 
 
+def parse_pixel_targets(entries: list[str]) -> dict[str, tuple[float, float]]:
+    """Task-defined points in screenshot fractions; the model chooses among these bounded targets."""
+    points: dict[str, tuple[float, float]] = {}
+    for entry in entries:
+        key, sep, value = entry.partition("=")
+        try:
+            x, y = map(float, value.split(","))
+        except ValueError as exc:
+            raise ValueError("--pixel-target requires KEY=X,Y with screenshot fractions") from exc
+        if not sep or not key.strip() or key in points or not all(math.isfinite(p) and 0 <= p < 1 for p in (x, y)):
+            raise ValueError("--pixel-target requires unique keys and finite coordinates in [0, 1)")
+        points[key] = (x, y)
+    return points
+
+
 def plan_rule(plan: tuple[tuple[str, ...], ...]) -> Any:
     """The baseline: the next button of the plan by how many presses were made, then done."""
 
     def rule(state: dict[str, Any], candidates: dict[str, str]) -> str:
         step = len(state["presses"])
         if step >= len(plan):
             return DONE
-        return next((f"click:{label}" for label in plan[step] if f"click:{label}" in candidates), ABSTAIN)
+        keys = (label if label.startswith(("click:", "pixel:")) else f"click:{label}" for label in plan[step])
+        return next((key for key in keys if key in candidates), ABSTAIN)
 
     return rule
 
@@ -78,7 +101,20 @@ async def _session(driver: CuaDriver, app: str) -> AsyncIterator[None]:
 
 
 def make_series(flags: argparse.Namespace) -> Series:
-    driver = driver_from_env("s1a-desktop")  # raises before the series starts when the driver is missing
+    if flags.app_path and sys.platform != "darwin":
+        raise ValueError("--app-path is supported only on macOS")
+    pixel_targets = parse_pixel_targets(flags.pixel_target)
+    visual_model = (
+        flags.model == "cua"
+        and os.getenv("CUA_S1_VARIANT", "nano") == "4b"
+        and os.getenv("CUA_S1_MODALITY", "multimodal") == "multimodal"
+    )
+    if pixel_targets and flags.model not in {"rule", "random"} and not visual_model:
+        raise ValueError(
+            "--pixel-target requires a screenshot model: use --model cua with CUA_S1_VARIANT=4b and "
+            "CUA_S1_MODALITY=multimodal, or an explicit rule/random baseline"
+        )
+    driver = driver_from_env(f"s1a-desktop-{uuid.uuid4().hex[:8]}")
     plan = parse_plan(flags.plan) if flags.plan else ()
     return Series(
         seeds=range(flags.seed, flags.seed + flags.episodes),
@@ -89,20 +125,32 @@ def make_series(flags: argparse.Namespace) -> Series:
             done_when=lambda snapshot: shows(snapshot, flags.expect),
             execute=flags.execute,
             clear_labels=tuple(v.strip() for v in flags.clear.split(",") if v.strip()),
+            window_title=flags.window_title,
+            pixel_targets=pixel_targets,
+            screenshot=visual_model,
         ),
-        session=_session(driver, flags.app),
+        session=_session(driver, flags.app_path or flags.app),
         baseline=("plan", plan_rule(plan)) if plan else None,
         annotate=lambda env, episode: None,
     )
 
 
 def flags(parser: argparse.ArgumentParser) -> None:
     parser.add_argument("--app", required=True, help="app name (Windows Calculator / Calculator), or a Windows AUMID")
+    parser.add_argument("--app-path", default="", help="macOS app bundle path to launch; --app identifies its window")
+    parser.add_argument("--window-title", default="", help="exact title of the app window to use")
     parser.add_argument("--goal", required=True, help="what to do in the window, read by the model on every turn")
     parser.add_argument("--expect", required=True, help="the text a display or label shows when the goal is met")
     parser.add_argument("--execute", action="store_true", help="click for real; without it one decision is planned")
     parser.add_argument("--plan", default="", help="the rule baseline: button labels in order, | between variants")
     parser.add_argument("--clear", default="", help="button labels pressed on reset when the window has one")
+    parser.add_argument(
+        "--pixel-target",
+        action="append",
+        default=[],
+        metavar="KEY=X,Y",
+        help="task-defined screenshot point (fractions in [0,1)); repeat for closed visual choices",
+    )
 
 
 SPEC = ToolAgentSpec(