Skip to content

controlkit.md

Latest commit

 

History

History
262 lines (206 loc) · 10.2 KB

controlkit.md

File metadata and controls

262 lines (206 loc) · 10.2 KB
Raw

ControlKit runners

ControlKit is the native XCTest runner used by the CLI's Apple-platform MCP tools. The public runner project is xcrs-controlkit. It uses one Xcode project generated by XcodeGen, with platform-specific targets and schemes:

Platform Scheme App product
iOS / iPadOS ControlKit-iOS ControlKit.app
tvOS ControlKit-tvOS ControlKit.app
visionOS ControlKit-visionOS ControlKit.app
watchOS ControlKit-watchOS ControlKit.app
macOS ControlKit-macOS ControlKit.app

The scheme and target names keep their platform suffixes so Xcode can select the correct SDK, while each built app is named ControlKit.

Build a runner

Install XcodeGen once, clone the runner project, and generate the Xcode project:

brew install xcodegen
cd <path-to-xcrs-controlkit>/Runners
xcodegen generate

Build the simulator runner for the platform you need:

xcodebuild -project XCRSControlKitRunner.xcodeproj \
  -scheme ControlKit-<platform> \
  -destination 'generic/platform=<platform> Simulator' \
  build-for-testing

Replace <platform> with iOS, tvOS, watchOS, or visionOS. macOS is built with -destination 'platform=macOS' and runs on the local Mac rather than a simulator.

Start the runner

Boot a compatible simulator and run the UI-test host:

xcodebuild -project XCRSControlKitRunner.xcodeproj \
  -scheme ControlKit-<platform> \
  -destination 'id=<simulator-udid>' \
  test-without-building

The shared XCTest runner starts a local ControlKit JSON-RPC server on 127.0.0.1:12004. Keep that test process running while calling the MCP tools. The macOS UI-test target includes the network-server entitlement required to bind the local port.

Run on a physical device

The same runner also works installed on a real iOS, tvOS, watchOS, or visionOS device, not just a simulator. Build and start it against the device's UDID (find it with xcrun devicectl list devices) instead of a simulator destination:

xcodebuild -project XCRSControlKitRunner.xcodeproj \
  -scheme ControlKit-<platform> \
  -destination 'id=<device-udid>' \
  build-for-testing

TEST_RUNNER_CONTROLKIT_LISTEN_HOST=0.0.0.0 xcodebuild \
  -project XCRSControlKitRunner.xcodeproj \
  -scheme ControlKit-<platform> \
  -destination 'id=<device-udid>' \
  test-without-building

The CONTROLKIT_LISTEN_HOST environment variable controls what the in-process RPC server binds to (Shared/ControlKitRPCServer.swift) — it defaults to 127.0.0.1, which is loopback-only on the device itself and unreachable from your Mac. xcodebuild forwards any TEST_RUNNER_-prefixed environment variable into the test process with the prefix stripped, so setting TEST_RUNNER_CONTROLKIT_LISTEN_HOST=0.0.0.0 makes the server bind on every interface, including the one reachable over your LAN. Find the device's address with its Bonjour hostname (<device-name>.local, discoverable via dns-sd -B _airplay._tcp local for an Apple TV) or its CoreDevice tunnel address (xcrun devicectl device info details --device <udid>, under Tunnel IP Address).

Connect through the automation MCP profile

Start either automation server:

xcrs --mcp
# or
smb --mcp --scope automation

The runner tools accept either simulator_name or simulator_udid for a simulator, or host (plus controlkit_port, defaulting to 12004) for a physical device or any other remote runner. Call device_select once to remember the target so later tools need no target arguments. Both commands expose the same unprefixed tool names.

Tool Runner Purpose
device_select All platforms Remember the active simulator or device for later tools.
device_capabilities All platforms Read platform and capability information.
input_click macOS Click at screen coordinates.
input_spatial_tap visionOS Perform a spatial tap.
input_tap iOS / tvOS / watchOS, simulator or physical Perform a touch tap.
input_text iOS / tvOS, simulator or physical Type into the focused field.
input_swipe iOS / tvOS, simulator or physical Swipe between coordinates.
input_button iOS / tvOS, simulator or physical Press a Home or tvOS remote button.
app_launch/app_terminate simulator or physical Launch/terminate an app by bundle ID. On a physical device this calls the runner's device.apps.launch/device.apps.terminate RPC methods instead of simctl.
screen_capture Simulator or physical Capture a PNG screenshot. Uses simctl for a simulator, or devicectl device capture screenshot when a device identifier is given.
ui_describe All UI-test runners Read the accessibility hierarchy for the supplied bundle_id.
ui_element_list All UI-test runners Read actionable elements with tap coordinates for the supplied bundle_id.
ui_tap All UI-test runners Resolve and activate an accessibility element by label, identifier, or value.

UI introspection requires a recent ControlKit runner that implements device.dump.ui. The ui_tap action requires a runner that implements device.ui.tap; it resolves and activates the requested element inside the target app, so an MCP client does not need to turn accessibility coordinates into a second action. The label, identifier, or value must match exactly one element. On tvOS, that element must already have focus; use input_button to navigate focus before calling ui_tap. All three UI tools require the target app's bundle_id; this lets XCTest attach to an already-running app instead of accidentally reading the ControlKit host app.

For a local macOS runner, omit simulator fields and pass host and controlkit_port when needed:

{
  "host": "127.0.0.1",
  "controlkit_port": 12004,
  "x": 400,
  "y": 300
}

For a physical device, pass its Bonjour hostname or tunnel address as host instead:

{
  "host": "<device-name>.local",
  "controlkit_port": 12004,
  "button": "select"
}

Platform-specific operations intentionally reject unsupported input. For example, visionOS rejects raw touch, watchOS rejects pointer and spatial input, and macOS rejects touch and Home-button requests.

Inspect the macOS host

The host subcommands describe what is actually on the local Mac's screen. They are for the cases where a UI test's own output is not enough: which windows are up, what text a screenshot contains, or which colors run along one line of pixels. Unlike the runner tools above they talk to no device and start no XCTest — they run a small Swift helper, xcrs-host-inspect, in process on the host. They need macOS 13 or later for the AppKit window server and Vision's text recognizer, are CLI-only, and are not part of the --mcp tool set.

Build the helper

Build it once with SwiftPM from the xcrs-controlkit checkout:

cd <path-to-xcrs-controlkit>
swift build -c release

The CLI uses the first xcrs-host-inspect it finds, in this order:

  1. the path in the XCRS_HOST_INSPECT environment variable,
  2. next to the xcrs binary itself, for a release that ships the helper,
  3. .build/release, then .build/debug, relative to the current directory,
  4. ~/bin, /usr/local/bin, and /opt/homebrew/bin.

Because steps 3 are relative, running xcrs from inside the checkout finds a swift build without any further setup. Set XCRS_HOST_INSPECT when you have several builds and want to choose between them.

Read the screen

List the host's on-screen windows, ordered by layer and then window id:

xcrs host windows
xcrs host windows --owner Mail --json

Each window reports its id, the owning application name (owner), its title, its layer, and its bounds. --owner keeps only windows owned by one application name.

Read the text in an image with Vision and print each string with its confidence:

xcrs host ocr --image shot.png
xcrs host ocr --image shot.png --language en-US --language sv-SE --json

--language is repeatable; omit it to use the platform's own default.

Crop a rect out of an image and write it to a new PNG:

xcrs host crop --image shot.png --rect 20,40,320,180 --out header.png

--rect takes x,y,width,height. The result reports the written path and its pixel dimensions, and the input image is never modified.

Probe one row or column and describe it as a run of colors, so a solid surface collapses to a single line instead of thousands of pixels:

xcrs host probe --image shot.png --row --position 300
xcrs host probe --image shot.png --column --position 120 --from 40 --to 260

--position is the row or column to read, and exactly one of --row or --column is required. --from and --to clip the line to part of its length. Runs report start, end, width, and a quantized #rrggbb color, so low-order noise does not split one surface into several runs.

Coordinates and scale

Coordinates you pass are in caller units, and --scale says how many pixels one unit covers: the default 1 reads them as pixels, and 2 reads them as Retina points on a 2x display. crop and probe honor it for their rect, position, and bounds. Output geometry follows the same convention — an image reports width/height in caller units alongside pixel_width/ pixel_height — while window bounds are screen points regardless of --scale.

Exit codes and errors

The helper separates a malformed command from failed work: it exits with status 2 when it rejects the arguments, and 1 when the operation itself fails, writing the reason to stderr. The CLI validates rects, the scale, and probe bounds before it spawns the helper, so bad input usually never reaches it. When the helper does fail, the CLI reports it with its status — for example xcrs-host-inspect crop exited with status 1: … — and exits non-zero. A helper that cannot be started at all, such as a missing binary, is reported with a hint to run swift build in xcrs-controlkit.

See also

For the product-level overview of end-to-end testing with xcrs and ControlKit, see smbcloud.xyz/xcrs-end-to-end-testing.