Add macOS Electron companion with scoped Messages and computer access

README.md

@@ -109,6 +109,7 @@ Roost is a pnpm workspace with a native Xcode app:
 | Path | Responsibility |
 | --- | --- |
 | `apps/web` | TanStack Start web/PWA, Node server, mobile API, CLI, and server tests |
+| `apps/desktop` | Electron Mac companion, selected Messages reads, and local computer sessions |
 | `apps/ios` | Native SwiftUI iPhone app, Xcode project, and native tests |
 | `apps/docs` | Static documentation website built from `docs/` |
 | `apps/marketing` | Public marketing website |
@@ -117,6 +118,8 @@ Roost is a pnpm workspace with a native Xcode app:
 
 Install once at the root with `corepack pnpm install --frozen-lockfile`.
 `pnpm dev` runs the web app; `pnpm dev:docs` and `pnpm dev:marketing` run the sites.
+`pnpm desktop:dev` opens the Electron companion; see the
+[desktop guide](docs/desktop.md) for setup, local permissions, and packaging.
 `pnpm check` verifies all JavaScript apps and the production web/mobile API.
 On macOS, `pnpm ios:build` builds the native app; see the
 [native app guide](apps/ios/README.md) for connection setup and simulator tests.

apps/desktop/README.md

@@ -0,0 +1,155 @@
+# Roost
+
+An Electron companion for an existing Roost server. The first release targets
+macOS 14 or later and packages for the architecture of the build machine.
+Agents, schedules, files, and conversation history remain on the server.
+
+## Build and run
+
+From the repository root, install the workspace dependencies, then run:
+
+```sh
+corepack pnpm desktop:dev
+```
+
+Building the native capture/input helper requires Xcode command line tools and Python 3.
+`corepack pnpm desktop:pack` creates an ad-hoc signed app in
+`apps/desktop/output/desktop/Roost.app` and a ZIP archive alongside it. Set
+`ROOST_DESKTOP_OUTPUT_DIR` to choose another output directory. Packaging refuses
+to overwrite an existing app. Developer ID signing, notarization, automatic
+updates, and Windows/Linux local adapters are not included in this first build.
+
+## Connect
+
+For exe.dev, click **Connect to exe.dev**, choose a running VM, and click
+**Connect to Roost**. The app uses your existing SSH login to discover VMs and
+provision a dedicated Roost-Mac device credential and a VM-scoped HTTPS token.
+Roost must be installed through its packaged installer on that VM, and the server
+must include the companion API and JSON pairing commands. The app does not
+install or upgrade the server. SSH setup is noninteractive and requires the
+VM's host key to be trusted already.
+
+Requests connect directly to the VM's HTTPS address. There is no persistent SSH
+tunnel. Native requests send the VM token in `X-Exedev-Authorization` and the
+Roost device token in `Authorization`. The remote workspace receives proxy
+authentication only for that exact origin and its secure WebSocket connections;
+it receives no Roost device credential or local IPC bridge.
+
+Both credentials are encrypted with Electron safeStorage. The app renews them
+through SSH one day before expiry. Explicit reconnect refreshes the proxy key;
+renewal preserves the Roost device ID and fails if that device was revoked.
+Disconnect clears local access immediately and attempts to revoke both managed
+credentials. If SSH is unavailable, the app reports that server revocation
+could not complete. A failed setup also cleans up its generated credentials.
+Replacing an exe.dev pairing retires the previous managed credentials.
+
+For another host, expand **Connect another server** and enter its HTTPS origin
+and a dedicated device token created on that server:
+
+```sh
+roost mobile create --name Mac --output /private/path/mac-token.txt
+```
+
+Loopback HTTP is accepted for development. Manual device tokens expire after
+90 days and must reach the device API without an interactive browser-login
+proxy. Disconnect removes the manual connection but does not revoke its token;
+use `roost mobile revoke <device-id>` on the server. Web cookies remain until
+browser sign-out or removal of app data.
+
+The Mac window controls share the app's draggable header. Buttons remain
+clickable and computer access can still be stopped from the toolbar, menu bar,
+or keyboard shortcut.
+
+## Local access
+
+Choose one agent in Companion settings. Access starts off, resets on restart or
+reconnection, and is enforced on the Mac as well as the server.
+
+### Messages
+
+Install [imsg](https://github.com/openclaw/imsg) separately, then allow Roost Full Disk Access in System Settings. The app finds imsg in
+`~/.local/bin`, `/opt/homebrew/bin`, or `/usr/local/bin`.
+
+Choose conversations from a local list of up to 100 recent chats. Only the
+selected chat IDs and labels are shared with the server. The agent can list
+those shared chats and read up to 100 recent messages from one selected chat;
+an optional filter searches that bounded page. It cannot read other chats,
+open attachments, mark messages as read, or send through this integration.
+Messages content is returned on demand rather than continuously mirrored.
+
+Tool: `roost_desktop_messages`, with `action=chats` or
+`action=history, chatId, limit, query?`.
+
+### Computer
+
+Enable a 10-minute session for your selected agent. Allow screen recording and
+Accessibility in System Settings. The agent can capture your main display,
+click, move, scroll, send shortcuts, and type Unicode text. Screenshots are
+bounded JPEGs; pixel coordinates are mapped to the main display's logical
+coordinates, including Retina scaling. A fresh screenshot is required before
+input and each action returns another screenshot.
+
+This operates your real desktop and can interfere with your input. It does not
+create a virtual desktop or restrict UI interaction to the selected Messages
+chats. Messages reading restrictions apply to the structured integration above;
+enabling computer control is a separate, broader grant. User instructions and
+Roost approvals still govern consequential actions, including sending messages;
+the capture/input helper does not recognize or block particular app buttons.
+
+Stop with the visible toolbar button, menu-bar item, or **Command–Shift–Escape**.
+While a session is active, local permission edits and session renewal are locked.
+Expiry, disconnect, revocation, and connection loss abort in-flight helpers.
+Stopped runs are checked on the server and propagated on the next poll. Local
+tools are unavailable while the Mac sleeps. Input requests are never replayed
+after a timeout or reconnect.
+
+Tool: `roost_desktop_computer`, with actions `screenshot`, `click`, `move`,
+`scroll`, `key`, or `type`. It is distinct from the Linux server's
+`roost_computer` tool.
+
+## Security and server contract
+
+The Roost workspace runs in a sandboxed WebContentsView with no preload, Node
+access, or local IPC bridge. Only the bundled companion panel can change local
+grants. Device API requests originate in the desktop main process, refuse
+redirects, and never send credentials to another origin. Remote navigation and
+popups open in the system browser; cross-origin authentication redirects need
+an appropriate directly reachable Roost origin.
+
+The authenticated API uses `/api/mobile/v1/companion/{connect,poll,result,disconnect}`.
+Polling keeps an ephemeral lease alive and advertises local grants. Each tool
+request is bound to its agent, active run, request UUID, and 15-second deadline.
+The server rechecks persisted run capabilities at dispatch, delivery, and result
+acceptance. Feed, handoff, and reflection runs cannot use these tools. Desktop
+access is serialized per companion, and conflicting eligible desktops are
+rejected rather than selected implicitly. The server must route a companion's
+requests and agent runs to the same process; the current single-server Roost
+deployment satisfies this requirement.
+
+Screenshots and message excerpts become tool results in the agent's active
+model session. Normal provider/session retention applies. They are not an
+offline-only data source. The local activity panel records action labels, not
+message bodies, screenshots, or tokens.
+
+## Verify
+
+```sh
+corepack pnpm desktop:test
+corepack pnpm --filter @roost/desktop test:e2e
+corepack pnpm --filter @roost/web exec node --import tsx --test tests/companion.test.ts tests/run-capabilities.test.ts
+corepack pnpm lint
+corepack pnpm typecheck
+```
+
+The Electron test uses a disposable local server, fake device token, and isolated
+user-data directory. It tests connection, agent selection, sandbox isolation,
+the companion panel, and disconnect. It does not read personal Messages, change macOS permissions, or execute
+desktop input. OS consent and credential encryption are simulated in the disposable process;
+this test does not exercise the system Keychain. Local-tool tests exercise scope checks, coordinate
+mapping, cancellation, and command construction using fixtures. The server
+integration test uses real stores, device authentication, and run policies.
+
+Live Messages reads, macOS permission/Keychain prompts, and real input injection require
+explicitly enabling access in the installed app; they are not verified by these
+fixture tests. The native helper is compiled and its shortcut self-check runs
+during every Mac build.

apps/desktop/native/MacHelper.swift

@@ -0,0 +1,143 @@
+import AppKit
+import ApplicationServices
+import ScreenCaptureKit
+
+enum HelperError: Error { case invalidInput, permissionRequired, captureFailed }
+
+let keyCodes: [String: CGKeyCode] = [
+    "a": 0, "s": 1, "d": 2, "f": 3, "h": 4, "g": 5, "z": 6, "x": 7, "c": 8, "v": 9,
+    "b": 11, "q": 12, "w": 13, "e": 14, "r": 15, "y": 16, "t": 17, "1": 18, "2": 19,
+    "3": 20, "4": 21, "6": 22, "5": 23, "9": 25, "7": 26, "8": 28, "0": 29, "o": 31,
+    "u": 32, "i": 34, "p": 35, "return": 36, "enter": 36, "l": 37, "j": 38, "k": 40,
+    "n": 45, "m": 46, "tab": 48, "space": 49, "backspace": 51, "delete": 51, "escape": 53,
+    "esc": 53, "home": 115, "pageup": 116, "end": 119, "pagedown": 121,
+    "left": 123, "right": 124, "down": 125, "up": 126,
+]
+
+func typingEvents(_ text: String) throws -> [CGEvent] {
+    var batches: [[UniChar]] = []
+    var batch: [UniChar] = []
+    for scalar in text.unicodeScalars {
+        let units = Array(String(scalar).utf16)
+        if batch.count + units.count > 20 {
+            batches.append(batch)
+            batch = []
+        }
+        batch.append(contentsOf: units)
+    }
+    if !batch.isEmpty { batches.append(batch) }
+    return try batches.map { units in
+        guard let event = CGEvent(keyboardEventSource: nil, virtualKey: 0, keyDown: true) else { throw HelperError.invalidInput }
+        units.withUnsafeBufferPointer { event.keyboardSetUnicodeString(stringLength: units.count, unicodeString: $0.baseAddress!) }
+        return event
+    }
+}
+
+func shortcut(_ key: String) throws -> (CGKeyCode, CGEventFlags) {
+    let parts = key.lowercased().split(separator: "+").map(String.init)
+    guard let last = parts.last, let code = keyCodes[last], parts.count <= 5 else { throw HelperError.invalidInput }
+    var flags: CGEventFlags = []
+    for modifier in parts.dropLast() {
+        switch modifier {
+        case "cmd", "command", "super", "meta": flags.insert(.maskCommand)
+        case "ctrl", "control": flags.insert(.maskControl)
+        case "alt", "option": flags.insert(.maskAlternate)
+        case "shift": flags.insert(.maskShift)
+        default: throw HelperError.invalidInput
+        }
+    }
+    return (code, flags)
+}
+
+func output(_ value: [String: Any]) throws {
+    FileHandle.standardOutput.write(try JSONSerialization.data(withJSONObject: value))
+}
+
+func capture() async throws {
+    let content = try await SCShareableContent.excludingDesktopWindows(false, onScreenWindowsOnly: true)
+    guard let display = content.displays.first(where: { $0.displayID == CGMainDisplayID() }) else { throw HelperError.captureFailed }
+    let bounds = CGDisplayBounds(display.displayID)
+    let scale = min(1.0, 1600.0 / bounds.width, 1200.0 / bounds.height)
+    let configuration = SCStreamConfiguration()
+    configuration.width = Int(bounds.width * scale)
+    configuration.height = Int(bounds.height * scale)
+    configuration.showsCursor = true
+    let filter = SCContentFilter(display: display, excludingWindows: [])
+    let image = try await SCScreenshotManager.captureImage(contentFilter: filter, configuration: configuration)
+    let bitmap = NSBitmapImageRep(cgImage: image)
+    guard let jpeg = bitmap.representation(using: .jpeg, properties: [.compressionFactor: 0.7]) else { throw HelperError.captureFailed }
+    try output([
+        "dataURL": "data:image/jpeg;base64," + jpeg.base64EncodedString(),
+        "width": image.width, "height": image.height,
+        "bounds": ["x": bounds.origin.x, "y": bounds.origin.y, "width": bounds.width, "height": bounds.height],
+    ])
+}
+
+func input(_ value: [String: Any], action: String) throws {
+    guard AXIsProcessTrusted() else { throw HelperError.permissionRequired }
+    if action == "click" || action == "move" {
+        guard let x = value["x"] as? Double, let y = value["y"] as? Double, x.isFinite, y.isFinite else { throw HelperError.invalidInput }
+        let point = CGPoint(x: x, y: y)
+        guard CGDisplayBounds(CGMainDisplayID()).contains(point) else { throw HelperError.invalidInput }
+        guard let move = CGEvent(mouseEventSource: nil, mouseType: .mouseMoved, mouseCursorPosition: point, mouseButton: .left) else { throw HelperError.invalidInput }
+        move.post(tap: .cghidEventTap)
+        if action == "click" {
+            let right = value["button"] as? String == "right"
+            let count = value["clicks"] as? Int ?? 1
+            guard (1...2).contains(count) else { throw HelperError.invalidInput }
+            for number in 1...count {
+                guard let down = CGEvent(mouseEventSource: nil, mouseType: right ? .rightMouseDown : .leftMouseDown, mouseCursorPosition: point, mouseButton: right ? .right : .left),
+                      let up = CGEvent(mouseEventSource: nil, mouseType: right ? .rightMouseUp : .leftMouseUp, mouseCursorPosition: point, mouseButton: right ? .right : .left) else { throw HelperError.invalidInput }
+                down.setIntegerValueField(.mouseEventClickState, value: Int64(number))
+                up.setIntegerValueField(.mouseEventClickState, value: Int64(number))
+                down.post(tap: .cghidEventTap); up.post(tap: .cghidEventTap)
+                usleep(60000)
+            }
+        }
+    } else if action == "scroll" {
+        guard let amount = value["amount"] as? Int, (1...10).contains(amount), let direction = value["direction"] as? String, ["up", "down"].contains(direction) else { throw HelperError.invalidInput }
+        guard let event = CGEvent(scrollWheelEvent2Source: nil, units: .line, wheelCount: 1, wheel1: Int32(amount * (direction == "up" ? 1 : -1)), wheel2: 0, wheel3: 0) else { throw HelperError.invalidInput }
+        event.post(tap: .cghidEventTap)
+    } else if action == "key" {
+        guard let key = value["key"] as? String else { throw HelperError.invalidInput }
+        let (code, flags) = try shortcut(key)
+        for pressed in [true, false] {
+            guard let event = CGEvent(keyboardEventSource: nil, virtualKey: code, keyDown: pressed) else { throw HelperError.invalidInput }
+            event.flags = flags
+            event.post(tap: .cghidEventTap)
+        }
+    } else if action == "type" {
+        guard let text = value["text"] as? String, text.utf16.count <= 10000 else { throw HelperError.invalidInput }
+        // Small Unicode batches preserve non-ASCII text without changing the
+        // user's clipboard or relying on their keyboard layout.
+        for event in try typingEvents(text) {
+            event.post(tap: .cghidEventTap)
+            event.type = .keyUp
+            event.post(tap: .cghidEventTap)
+        }
+    } else { throw HelperError.invalidInput }
+}
+
+@main struct MacHelper {
+    static func main() async {
+        do {
+            if CommandLine.arguments.contains("--self-test") {
+                guard try shortcut("cmd+shift+l").0 == 37, try shortcut("Return").0 == 36 else { throw HelperError.invalidInput }
+                for text in ["", "hello", String(repeating: "a", count: 19) + "😀z", String(repeating: "😀", count: 21), "中文 e\u{301} 👨‍👩‍👧‍👦"] {
+                    let events = try typingEvents(text)
+                    let payload = events.compactMap { NSEvent(cgEvent: $0)?.characters }.joined()
+                    guard payload == text else { throw HelperError.invalidInput }
+                }
+                try output(["ok": true]); return
+            }
+            let data = FileHandle.standardInput.readDataToEndOfFile()
+            guard data.count <= 40000, let value = try JSONSerialization.jsonObject(with: data) as? [String: Any], let action = value["action"] as? String else { throw HelperError.invalidInput }
+            if action == "screenshot" { try await capture() }
+            else { try input(value, action: action); try await Task.sleep(for: .milliseconds(150)); try output(["ok": true]) }
+        } catch {
+            // Avoid logging private text or raw screen contents on failure.
+            FileHandle.standardError.write(Data("Mac access failed. Check permissions and action arguments.\n".utf8))
+            exit(1)
+        }
+    }
+}

apps/desktop/package.json

@@ -0,0 +1,30 @@
+{
+  "name": "@roost/desktop",
+  "productName": "Roost",
+  "version": "0.1.3",
+  "private": true,
+  "type": "module",
+  "main": "dist/main.cjs",
+  "scripts": {
+    "build": "node scripts/build.mjs",
+    "dev": "pnpm build && electron .",
+    "start": "electron .",
+    "package": "pnpm build && node scripts/pack.mjs",
+    "typecheck": "tsc --noEmit",
+    "test": "node --import tsx --test tests/*.test.*",
+    "test:e2e": "pnpm build && node --import tsx tests/e2e.ts"
+  },
+  "dependencies": {
+    "@roost/desktop-protocol": "workspace:*",
+    "zod": "^4.0.0"
+  },
+  "devDependencies": {
+    "@roost/typescript-config": "workspace:*",
+    "@types/node": "^22.0.0",
+    "electron": "44.7.0",
+    "esbuild": "0.28.2",
+    "playwright": "1.58.2",
+    "tsx": "4.23.13",
+    "typescript": "7.0.2"
+  }
+}

apps/desktop/scripts/build.mjs

@@ -0,0 +1,48 @@
+import { execFileSync } from "node:child_process";
+import { copyFileSync, mkdirSync } from "node:fs";
+import { build } from "esbuild";
+
+mkdirSync("dist", { recursive: true });
+await build({
+  entryPoints: ["src/main.ts", "src/preload.ts"],
+  outdir: "dist",
+  outExtension: { ".js": ".cjs" },
+  bundle: true,
+  platform: "node",
+  format: "cjs",
+  target: "node22",
+  external: ["electron"],
+  sourcemap: true,
+});
+for (const file of ["panel.html", "panel.css", "panel.js"])
+  copyFileSync(`src/${file}`, `dist/${file}`);
+copyFileSync("../web/src/assets/roost.svg", "dist/roost.svg");
+execFileSync(
+  "python3",
+  ["scripts/icons.py", "../web/src/assets/roost.svg", "dist"],
+  { stdio: "inherit" },
+);
+if (process.platform === "darwin") {
+  execFileSync(
+    "iconutil",
+    ["-c", "icns", "dist/Roost.iconset", "-o", "dist/Roost.icns"],
+    { stdio: "inherit" },
+  );
+  execFileSync(
+    "xcrun",
+    [
+      "swiftc",
+      "-parse-as-library",
+      "-O",
+      "-module-cache-path",
+      "dist/swift-cache",
+      "-target",
+      `${process.arch === "arm64" ? "arm64" : "x86_64"}-apple-macosx14.0`,
+      "native/MacHelper.swift",
+      "-o",
+      "dist/roost-mac-helper",
+    ],
+    { stdio: "inherit" },
+  );
+  execFileSync("dist/roost-mac-helper", ["--self-test"], { stdio: "inherit" });
+}