Repository navigation
Feat/benchmark templates swe tbench - #256
Merged
Merged
Conversation
…wers Orchestration layer that turns Orcabot into an English-driven harness for slop-code-bench. Pairs with the fork branch feat/host-tmux-executor (no-Docker executor that mirrors each run into an attachable tmux window). A Claude Code "orchestrator" terminal — driven in plain English via Remote Control — runs `slop-code run` with the host-tmux environment, then spawns a read-only viewer pane per run via its create_terminal MCP tool (boot_command: `tmux attach -r -t scb:<window>`) and surfaces results in a browser block pointed at the Dash dashboard. No infra change needed: tmux is already in sandbox/docker/Dockerfile, and the desktop rootfs is docker-exported from that same image. - orchestrator-runbook.md: system prompt + runbook for the orchestrator agent - config-wizard.md: Orcabot-chat config phase (CLI / auth / target / problems) - template.json: importable DashboardTemplateWithData (orchestrator + note + browser) - bin/scb-runs, bin/scb-attach, bin/scb-dashboard: in-VM helpers (validated vs real tmux) - README.md: architecture, quick start, validated-vs-TODO status Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
… templates
Add Orcabot runner templates for three more coding benchmarks, plus an honest
"not runnable" note for FrontierCode and a top-level index.
Unlike slop-code-bench (host-executable), these are Docker-per-task harnesses
where container isolation is core to correctness — so the value is orchestrator
+ pluggable backend, not removing Docker:
- swebench/ SWE-bench (Lite/Verified/Full). Inference on host -> predictions
JSONL; eval is Docker-per-task via Modal (--modal true) or DinD.
- swebench-pro/ Scale's public 731-task set via scaleapi/SWE-bench_Pro-os.
Inference -> patches JSON; eval via Modal or --use_local_docker.
- terminal-bench/ Terminal-Bench 2.1 via the **Harbor** harness (not legacy tb).
Docker-per-task; --env daytona/Modal remote backend avoids DinD.
- frontiercode/ NOT-RUNNABLE.md — Cognition's FrontierCode is closed/
submission-only (held-out tasks, no public harness).
- README.md index + the two architecture families (host vs Docker-per-task).
Each runnable dir ships README + orchestrator-runbook + config-wizard +
template.json. Grounded in each harness's real commands; templates validated to
parse. Not validated end-to-end (need Docker/Modal/Daytona + provider keys).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Light end-to-end smoke test for the benchmark read-only viewer pattern (branch 2), run against the live desktop stack via the orcabot CLI + the sandbox /debug/exec endpoint. Asserts: tmux present in VM; a read-only viewer terminal attaches to a tmux-mirrored run and renders its output; and injected keystrokes from the viewer never reach the run pane (read-only holds). Uses a shared, world-accessible tmux socket in /workspace so it's uid-agnostic (Orcabot PTYs may run under different uids when the egress UID pool is active) — the pattern the real host-tmux executor should adopt for cross-PTY viewing. Result on desktop VM (sandbox main-v39, tmux 3.3a): 4 passed, 0 failed. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…ed on VM) Revise the viewer-pattern smoke test away from the rejected shared tmux control socket to the secure mechanism: a read-only viewer terminal tails the run's per-run logfile (`tail -F`). A world-accessible tmux control socket would let any in-VM process attach = read every pane AND inject commands into other sessions, bypassing output redaction (CLAUDE.md:96) and the cross-session isolation invariants (CLAUDE.md:94, sandbox/CLAUDE.md:248) via a local channel the egress proxy never sees. A logfile tail has no inject capability and no reach into other runs, staying within the already-shared /workspace trust boundary. Result on desktop VM: 3 passed, 0 failed (viewer shows the run; injected keystrokes never reach the run logfile). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
… socket Correct the viewer mechanism to fit the security model. A separate viewer PTY may run under a different uid (egress UID pool); bridging it to the run's tmux *control* socket would require world-accessibility = read+inject across sessions, bypassing output redaction (CLAUDE.md:96) and cross-session isolation (CLAUDE.md:94) via a local channel the egress proxy never sees. Cross-PTY Orcabot viewer panes now use `tail -n +1 -F <run-logfile>` (read-only, reaches only that run, stays within shared /workspace). Same-uid interactive attach (`scb-attach` / `tmux attach -r` in the executor's own shell) is still fine and clarified as such. Updates orchestrator-runbook, both READMEs, scb-runs. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Uncomment the OpenCode CLI install in the sandbox Dockerfile so benchmark harnesses (slop-code-bench) can drive it for OpenRouter models (e.g. Kimi). OpenCode/Droid/OpenClaw were stripped to save image size; re-enable OpenCode only. The desktop VM image is bumped to IMAGE_SIZE_MB=4096 at build time because the extra binary doesn't fit the old 3GB rootfs (had ~188MB free). Verified: opencode 1.17.9 present at /usr/local/bin/opencode in the rebuilt VM. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Self-contained handoff: what works, the committed fixes (auth solved via the LocalStreamingRuntime env_vars fix + synthetic openai-compatible provider), the remaining streaming hang, the VM/venv/broker setup, the exact run launcher, and a diagnostic-first plan (broker already streams SSE with flush, so localize the hang before assuming the broker buffers; add a stream timeout to fail fast). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The 3GB rootfs overflows once OpenCode is re-enabled alongside the other agent CLIs (commit 5bd9360 re-enabled it but only bumped the size via an IMAGE_SIZE_MB env override at build time, not in source — a fresh build would still produce 3GB). Commit the default so the image matches the shipped Dockerfile. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
The secrets broker listens on 127.0.0.1 only (IPv4). Node/undici-based agent harnesses (OpenCode, Droid) resolve 'localhost' to IPv6 ::1 first, get ECONNREFUSED against the IPv4-only broker, and retry with exponential backoff -- which presents as a multi-minute 'hang' with no output. curl, Go, and Python prefer IPv4 so they connected fine and masked the bug. Confirmed live: an OpenCode -> OpenRouter (Kimi K2.6) run hung indefinitely with baseURL http://localhost:8082/...; switching to http://127.0.0.1:8082/... made the agent loop complete immediately. Affects every brokered provider env var (OPENROUTER/OPENAI/ANTHROPIC_BASE_URL, *_BROKER). Takes effect in the deployed sandbox after a VM image rebuild. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
The opencode->Kimi 'streaming hang' was an IPv4/IPv6 loopback mismatch (broker is 127.0.0.1-only; Node/undici tries ::1 for 'localhost'). Updates the resume doc with the diagnosis, the env.go fix (17e7cd4), and bakes the 127.0.0.1 swap into the run launcher so it works on a not-yet-rebuilt image. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
OpenCode was hidden in the dashboard 'add terminal' picker pending the broker connection issue (now fixed in 17e7cd4 -- the localhost->127.0.0.1 IPv6 loopback bug). Re-enable the menu entry; the harness is installed in the sandbox image (5bd9360) and OpenRouter routing is validated end-to-end. Voice/TTS is intentionally left out of the OpenCode menu (the TTS gate already excludes opencode); model selection, MCP, and skills follow the existing per-agent gating. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…ard template Templates are DB-only (no code seed), so the template.json was never visible in the New Dashboard grid. Add bin/scb-insert-template to upsert it into the dashboard_templates table (status=approved), and rework the template so the orchestrator is actually useful: - orchestrator bootCommand appends an operating briefing to CLAUDE.md, so the Claude Code orchestrator boots already knowing the validated slop-code commands (opencode + Kimi K2.6, 127.0.0.1 broker URL, venv path) -- fixing the 'agent has no idea how to run a benchmark' experience. - note carries the one-time VM setup + how to drive it in English. - README documents the insert path and that 'set up a benchmark' in chat is NOT a feature. Manual one-time VM setup (fork + venv) still required, per the chosen scope. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
Templates can now carry a 'setupGuide' that the Orcabot chat runs as a guided, tool-driven setup. When a dashboard is created from such a template, the guide is copied onto the dashboard; the chat injects it (keyed by dashboardId) so it walks the user through setup using its existing tools (terminal_input, secrets_create, create_terminal, etc.). - schema: setup_guide column on dashboard_templates + dashboards (CREATE TABLE for fresh DBs + idempotent ALTER migrations for existing). - templates/handler: ensureTemplateColumns adds the column; formatTemplateWithData surfaces setupGuide; populateFromTemplate copies template.setup_guide onto the new dashboard. - chat/handler: streamMessage injects an ACTIVE SETUP WALKTHROUGH block when the active dashboard has a setup_guide. - SlopCodeBench Runner template gains a setupGuide: choose harness source (our fork / custom URL / already-present), clone, uv sync 3.12, ensure OPENROUTER broker key, patch broker URL to 127.0.0.1, run + confirm the first arm. - scb-insert-template writes setup_guide. Slice 1 (core). Activating in the running desktop app needs a controlplane workerd rebuild; the data layer is already migrated + populated. Slice 2 (frontend auto-kickoff + mark-complete) still pending. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…lkthroughs Auto-kickoff: createDashboard now returns hasSetupGuide (from populateFromTemplate); when true, the dashboards page seeds the existing orcabot_initial_prompt localStorage hook before navigating, so ChatPanel auto-expands and sends 'Walk me through setting up this dashboard.' on the new dashboard -- and the control plane injects that dashboard's setup_guide so the chat knows the steps. Mark-complete: new complete_setup_walkthrough chat tool nulls the dashboard's setup_guide so it stops being injected. The injected walkthrough block now tells the chat to call it (with the dashboard id) once the user confirms setup is done. Activating in the running desktop app still needs a controlplane + frontend rebuild. Typechecks clean on both sides. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…ude Code
Two template bugs:
- the note item's content was JSON-wrapped ({"text":...}); NoteBlock uses
data.content directly, so it rendered the raw JSON. Store plain markdown.
- the orchestrator terminal was named 'Benchmark Orchestrator', which is not in
the frontend NAME_MAP and whose bootCommand isn't an exact BOOT_COMMAND_MAP
key, so getAgentType fell through to 'shell' -- losing the Claude Code icon/
branding. Name it exactly 'Claude Code' (don't rebrand the harness).
Re-applied to the shared dev/prod D1 row too.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
… no /etc/hosts The browser block's noVNC stream failed with a WebSocket 1011 close, 'Failed to connect to downstream server'. Root cause: the VM had no /etc/hosts, so 'localhost' did not resolve. websockify is started with target 'localhost:<vncPort>', so it could not reach x11vnc (which listens fine on 127.0.0.1) and closed every VNC client. Xvfb/x11vnc/Chromium were all healthy — only name resolution was broken. Why /etc/hosts was missing (not a security measure): Docker provides /etc/hosts as a runtime bind-mount at 'docker run', so it is NOT in the image, and this VM rootfs is built via 'docker export' (build-images.sh), which drops those runtime files — the same reason /etc/hostname and /etc/network/interfaces are recreated here by hand. /etc/hosts is the same family of file and was simply missed. The egress/broker security model keys on the loopback IP range (iptables ! -d 127.0.0.0/8) and numeric 127.0.0.1, never on 'localhost' failing to resolve, so restoring it changes nothing about those controls. Two fixes: - browser.go: target websockify at 127.0.0.1:<vncPort> instead of localhost, so it works regardless of /etc/hosts (matches how the rest of the code addresses loopback numerically). - build-images.sh: write /etc/hosts into the rootfs. localhost is mapped IPv4-only (not ::1) to avoid the IPv6-preference stall that IPv4-only-bound services hit. Both need a sandbox + VM image rebuild to reach the deployed VM. The running desktop VM was hotfixed (IPv4-only /etc/hosts), restoring the noVNC stream now (verified: WS relays RFB 003.008). Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
#254 added download-on-demand of a published VM image (vm-image.json -> GitHub releases), which overrides a locally-built sandbox.img in dev — so branch work (opencode, baked /etc/hosts, etc.) gets bypassed by the slim published image. Add an ORCABOT_VM_IMAGE env override, checked first in ensure_vm_image: if set to an existing raw .img or .gz, it's staged directly, bypassing the version check + release download. Marks the staged image 'dev-override' so unsetting the var re-triggers the normal published download. Inherited by the headless orcabot-desktop the CLI spawns, so 'ORCABOT_VM_IMAGE=... orcabot up' works. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
The merged surface-token enforcement (#250) added X-Orcabot-Surface to Remote::apply_auth but not to cp_call, the helper used by ls/new-terminal and the other dashboard/session commands. On desktop (SURFACE_TOKEN provisioned), those commands 401. Send the token from cp_call too, mirroring apply_auth. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
open_pty_ws set X-User-ID on the handshake but not X-Orcabot-Surface, so tail/connect (the PTY WS path) 401'd on desktop once SURFACE_TOKEN enforcement landed. Send the token header on the WS upgrade too (this client can set headers), matching cp_call/apply_auth. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
Docker Desktop for macOS file-sharing doesn't reliably flush thousands of incremental loop-mount writes back to a host-shared (-v) bind-mounted image, so building sandbox.img directly on /output silently truncated/dropped files (the 167MB opencode binary vanished, /usr/bin/head corrupted to 'Exec format error') with no error and space to spare. The Docker image + rootfs tarball were always correct — only the ext4 populated on the host mount got mangled. Build the ext4 on the container's own fs (/build), then copy the finished image to /output ONCE. A single sequential copy flushes reliably through file-sharing, unlike the many small loop writes. (CI on real Linux was never affected.) Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…tes-swe-tbench # Conflicts: # desktop/app/src-tauri/src/bin/orcabot.rs
…ache
macOS/Virtualization.framework caches a disk image's SIZE keyed on its file
PATH. Staging every image to a fixed 'sandbox.img' meant that after a 3GB image
(the published v1) sat there, a later 4GB image at the same path was presented
to the guest as a 3GB disk. Everything ext4 placed above 3GB (opencode, head,
curl) fell off the end of the truncated disk and read back as garbage
('Exec format error') — while the on-disk image was byte-perfect. The image
source (local build vs GitHub download) was irrelevant; only the reused path
mattered.
Stage under a content-derived name instead — sandbox-<token>.img (token = hash of
the source's nanosecond-mtime+size for local/override images, or the manifest
version for downloads). A different size => different name => a fresh path VZ has
never cached => the guest sees the real size. Adds cleanup of stale staged images
and a migration that renames a legacy 'sandbox.img' onto the new scheme.
Verified: a 4GB override image now presents 8388608 sectors (4GiB) to the guest
and opencode/head/curl all run; previously the same image showed 6291456 (3GiB)
and those binaries were 'Exec format error'.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…ie check to PTYs Two fidelity fixes so regression-smoke passes against a current stack: 1. Surface-token auth. The control plane now only trusts dev-auth when the X-Orcabot-Surface header matches the per-boot SURFACE_TOKEN (devAuthSurfaceTrusted, #250/#255). The smoke's raw control-plane curls sent only X-User-ID, so every one 401'd ('Authentication required') and all dashboard/item setup collapsed. Read the token from the same file the CLI/app use and attach it to all $CP calls via an AUTH header array. Empty token (pre-enforcement stacks) is harmless. 2. Zombie check scoped to PTYs. #1/#7 asserted no leaked PTY after delete via a global 'ps -eo stat | grep -c Z', which also counts the browser block's chromium <defunct> zygote/helper processes (PPID of the browser supervisor, present even at fresh-boot baseline). Those are not a PTY-reap leak. Count only bash zombies ('^Z.*bash') so the check reflects what it claims to test. Verified: bash PTYs reap cleanly (0 bash zombies) while chromium helpers idle in the background. Result: 12 passed, 0 failed on a fresh VM. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
Same fix as regression-smoke: the benchmark viewer smoke's raw control-plane curls sent only X-User-ID, so they 401'd against a stack that enforces the per-boot SURFACE_TOKEN (devAuthSurfaceTrusted). Read the token from the CLI/app token file and attach X-Orcabot-Surface to all $CP calls via an AUTH header array. Empty on pre-enforcement stacks is harmless. Verified: 3 passed, 0 failed on the desktop VM. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…is honored On desktop the control plane only trusts dev-auth when a request carries the per-boot surface token (devAuthSurfaceTrusted). HTTP works because bootstrap creates an orcabot_session cookie — but the cross-origin GUI WebSockets (:8788 webview -> :8787 control plane) carry NO cookie (hasSessionCookie=false), so their only accepted proof is the surface token as a query param. Every WS was being rejected (isAuthenticated=false), breaking terminal streaming AND the browser-block VNC in the GUI once SURFACE_TOKEN enforcement is on. Two root causes: 1. Poisoned null token. ensureSurfaceToken() cached the fetch *promise* even when it resolved null (the Tauri get_surface_token IPC can be unreachable during an early/headless pre-window load). It then returned that null forever and never retried, so getCachedSurfaceToken() stayed null for the whole session. HTTP hid this via the cookie; WS had no fallback. Now a null result clears the promise so the next caller retries. 2. Frozen tokenless WS URL. The WS URL was built once at construct with a *synchronous* getCachedSurfaceToken() and reused on every reconnect — so a token that resolved later (or after a retry) was never applied. Added an overridable resolveUrl() hook to BaseWebSocketManager that builds the URL at connect time; TerminalWSManager and DashboardWSManager override it to await ensureSurfaceToken() and append &surface= on every (re)connect. BrowserBlock's VNC WS (noVNC RFB, not a manager) resolves the token into state and includes it in the URL — it appended no surface at all before. Also adds a get_surface_token delivery-proof log (token length only) that pinned this down: the IPC returns len=64 to the webview, so delivery was fine — the bug was purely caching/usage. Verified in the running GUI: pty-ws goes from auth_rejected=N/proxy_start=0 to proxy_start=11/auth_rejected=0, with hasSessionCookie=false (i.e. authed by the surface token, as designed). Frontend typecheck clean. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
… mid-persist Resizing a component reverted to its previous size ~1s later, ~half the time — while moving did not. Root cause was an asymmetry in the canvas's optimistic reconciliation: a drag holds its dropped position in localPositionsRef and applyLocalPositionOverrides forces it during rebuilds for 2500ms, masking the window where the debounced (~500ms) persist is still round-tripping and a collab echo / unrelated refetch transiently reverts the item cache. Resize persisted the same way but had NO equivalent size override, so a rebuild in that window snapped the node back to the old size. (The collaboration WS echoes only started flowing again once the surface-token WS fix re-enabled that socket, which is why this surfaced now.) Mirror the position mechanism for size: add applyLocalSizeOverrides (pure, unit-tested) and a localSizesRef; apply it in rebuildNodes; on resize-end record the new size AND position (a top/left resize also moves the node). 6 new tests. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…spam) The workspace sidebar showed 'Request failed with status 404'. workspaceSessionId fell back to sessions[0] when no session was active — and sessions[0] could be a STOPPED session whose sandbox is gone. Proxying files to it returns the sandbox's 'E79709: session not found' (404), which the sidebar rendered as a raw error and polled on repeat. - workspaceSessionId now selects only an active/creating session; if none is live it returns undefined, and loadFiles already no-ops on an undefined session, so the sidebar renders its empty state instead of polling a dead session. - loadFiles also degrades gracefully on a 404 (session-not-found / stale sandbox id after a VM restart): render an empty tree, not a scary HTTP error string. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
… browser smoke The browser block showed 'Browser failed to start': chromium in the VM image was 150.0.7871.46-1~deb12u1, a BROKEN Debian bookworm-security build that IMMEDIATE_CRASHes (SIGTRAP) on launch — start() succeeds and Xvfb/websockify come up, but chromium dies so status.ready never flips true and the frontend times out. Root cause was NOT chromium (150.0.7871.100 works fine) and NOT the custom kernel — it was a stale, broken build baked into the rootfs because Docker cached the base apt layer from when .46 was current. Isolated by running the SAME chromium in a clean debian:bookworm container (.100 → works) and in the sandbox rootfs under Docker's kernel (.46 → crashes), then upgrading in-place (.46 → .100 → works). Fix: a late, cheap chromium-refresh layer that re-reads the mirror and upgrades to the current build, gated by a CHROMIUM_CACHEBUST build-arg (timestamp from build-images.sh) so it re-runs every image build instead of caching stale chromium. Placed after the heavy npm/agent layers so the cache-bust only re-runs this layer. No kernel rebuild needed (SKIP_KERNEL-safe). Also adds desktop/tests/browser-smoke.sh — the browser block had zero coverage, which let the broken build ship. It asserts start -> chromium up -> status.ready=true within a bound, and flags the known-broken .46 build. Verified on a rebuilt image: chromium 150.0.7871.100, browser/status ready:true in ~4s, browser-smoke 2/2. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…type) The browser block showed chromium's yellow 'You are using an unsupported command-line flag: --no-sandbox. Stability and security will suffer.' banner. --no-sandbox is required (chromium runs as root in the VM) and --disable-infobars does not hide this specific warning in current chromium. --test-type suppresses it. Cosmetic-only; the block already worked. Verified: browser-smoke still 2/2. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…block
Two fixes.
1) Claude Code 'exec format error'. The image installed claude via npm, but npm's
--omit=optional (set in this image) skips claude-code's platform-specific native
binary (shipped as an OPTIONAL dep), leaving a shebang-less 'native binary not
installed' stub at /usr/bin/claude → 'fork/exec /usr/bin/claude: exec format
error' when a Claude Code PTY starts. Switch to the official native installer.
Its per-user launcher resolves the ~250MB binary from $HOME, which our no-home
pty-NNN users lack ('Failed to install'), so we point /usr/local/bin/claude
straight at the self-contained binary in a shared /opt/claude (HOME-independent).
Removes the dead claude-wrapper.sh. Verified: claude 2.1.205 runs as a no-home
pty user and a Claude Code terminal renders its UI.
2) Browser block white border. VncViewer used scaleViewport (aspect-fit) with
resizeSession=false and background=#ffffff, so a non-16:9 block showed a big white
border around the fixed 1280x720 framebuffer. Set resizeSession=true (ask
x11vnc/Xvfb to resize the framebuffer to fill the block) with a neutral dark
background as the fallback instead of jarring white.
Verified on a rebuilt image (fsck clean): claude PTY works, browser-smoke 2/2,
chromium 150.0.7871.100.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…ship
The VM installs chromium/claude/agents at "latest" and can't safely pin them (a
Debian chromium version pin vanishes on the next security bump), so a rebuild can
silently inherit an upstream regression and replace a working VM with a broken one
-- which is exactly what happened (Debian chromium 150.0.7871.46 SIGTRAPs on launch;
claude-code repackaged so npm left an exec-format stub).
Add sandbox/docker/validate-image.sh, run by build-images.sh via "docker run"
immediately after "docker build" and BEFORE the rootfs is exported. Because
build-images.sh uses "set -e", a non-zero exit aborts the whole build. It catches
what a --version check cannot:
- chromium is actually LAUNCHED headless and its DevTools must answer (catches the
installs-fine-but-crashes-on-launch class).
- claude + agents are run as a no-own-home, sandbox-group user with a fresh HOME
(mirrors pty-NNN users) -- catches per-user/root-only installs (claude's
"exec format error" / "Failed to install"). HOME is under /home, not /tmp,
because codex refuses to create helpers in a temp dir.
Proven: real chromium passes / a crash-on-launch chromium aborts the build; all
four agents (claude 2.1.205, opencode 1.17.18, gemini 0.50.0, codex 0.144.0) pass
against the current good image with no false-fails.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
Clicks in the browser block landed off-target unless the canvas was at 100% zoom. noVNC's Display maps a pointer to a framebuffer coord via absX(x) = x / _scale, where x comes from getBoundingClientRect (which DOES include React Flow's ancestor transform: scale(zoom)) but _scale does NOT — noVNC's ResizeObserver only fires on layout-box changes, and an ancestor CSS transform doesn't change the box, so _scale stays at its un-zoomed value. Net error is exactly the zoom factor (fine at 1.0, progressively worse zoomed out). Pass the live React Flow zoom (transform scale) into VncViewer and pre-divide the pointer coordinate by it before noVNC's absX/absY, cancelling the ancestor scale. A no-op at zoom=1; guarded/best-effort so a noVNC-internals change can't break input. Frontend-only; no VM rebuild. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…alhost
Three independent fixes from review:
1. VALID_ITEM_TYPES (controlplane POST /dashboards/:id/items) rejected "decision"
and "outlook_calendar" with E79306, though both have blocks registered in the
canvas and are in the DashboardItemType union — so creating them 400'd. Added
both. (recipe stays out: intentionally not createable yet; cursor is the presence
overlay, not an item.)
2. SWE-bench / SWE-bench Pro / Terminal-Bench note items stored content as a JSON
string {"text":"..."}, but NoteBlock renders data.content verbatim, so they showed
raw JSON. Unwrapped to plain markdown (matches the slopcodebench template).
Targeted value replacement so the rest of each template.json is untouched.
3. applyCustomEndpointEnv built the broker URL with "localhost", reintroducing the
Node/undici IPv6-first hang against the IPv4-only broker for custom-endpoint
OpenAI-native agents (opencode/droid). Use 127.0.0.1, matching env.go.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…al-Bench templates Only slopcodebench had a setupGuide, so the other three benchmark templates never triggered the in-chat setup walkthrough (hasSetupGuide) when a user created a dashboard from them. Add a tailored, concise walkthrough to each, distilled from its orchestrator-runbook: pick variant/backend, clone the harness, add the broker-protected model key (never echoed), set up the eval backend (Modal / local Docker), run the two phases (inference then evaluation), and wire a viewer terminal to watch a run. The insert path already consumes this: scb-insert-template maps template.json's `setupGuide` to the dashboard_templates.setup_guide column, which propagates to dashboards.setup_guide and is injected by the control-plane chat handler. Field name matches slopcodebench. Appended as the last key so the rest of each template.json is untouched. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
Ported from the feat/desktop-workspace-and-pty-pan working tree. 1. Open workspace folder. New reveal_workspace Tauri command opens the app's own host workspace dir (WorkspaceState.workspace_path) in Finder/Explorer/xdg-open; it takes no path from the frontend. Exposed via tauri-bridge.revealWorkspace and a FolderOpen button in the WorkspaceSidebar header (desktop only). 2. Pan out of a big PTY. A terminal that fills the viewport was a pan trap — xterm captures left-drags. Right-button drag on the terminal body now drives React Flow's viewport directly (getViewport/setViewport) and suppresses the follow-up context menu only when a pan actually happened, so left/middle stay with xterm and right-click still works when you don't drag. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…n works The PTY right-drag pan was defeated by macOS's native WebView text menu (from xterm's hidden input textarea): it fires on right-mousedown, BEFORE any drag is detected, so the earlier 'preventDefault only if we panned' never fired and the menu came up and captured the mouse. Worse, the contextmenu handler also nulled the pan state (termPan) on that same early event, so even a drag couldn't pan. Now: always preventDefault the terminal's contextmenu (the menu is useless over a canvas — Cut/Copy are dead, paste is Cmd+V), and move pan-state cleanup to mouseup so the early contextmenu event can't wipe it. Right-drag now pans reliably. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
…crash loop) opencode's newer file picker (fff) refuses to run when the project dir IS the home dir — "Can not run certain FFF features in a file system root or home directories" — and the sandbox set HOME = workspace root = the PTY's cwd. That failed fff init on every boot and drove opencode into a re-init/reopen loop. Point HOME at a dedicated <workspace>/.home subdir so cwd (the workspace) != HOME. Keeps ~ inside the workspace tree (dot-hidden, so it stays out of the file sidebar) and self-contained. Agent MCP/config is unaffected: it's enforced via HOME-independent overrides (GEMINI_CLI_SYSTEM_SETTINGS_PATH explicit path, /etc/codex/config.toml system config, project-scope .mcp.json read from cwd) — the ~-fallbacks moving here are harmless. Created 02775 + setgid so pty-NNN users (gid=sandbox) can write. Confirmed: opencode logs the fff error and re-inits when HOME==cwd, and boots clean with 0 fff errors when HOME!=cwd. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
… renders opencode's newer "opentui" TUI probes and uses modern terminal features (synchronized output, DEC mode 2026, etc.). The terminal component imported the DEPRECATED xterm 5.3.0, which lacks them — so opencode's screen garbled/broke (read as a "crash", though the opencode process is actually fine). Worse, the app was in a mismatched state: old xterm 5.3.0 core paired with the newer @xterm/addon-fit 0.10 (which targets @xterm/xterm 5.5). @xterm/xterm 5.5.0 was already installed (transitive). Migrate the imports (xterm -> @xterm/xterm, incl. the css path), make it a direct dep, and drop the old xterm package. 5.5.0 supports mode 2026, so opencode's TUI renders correctly, and the core/addon versions now match. Typecheck clean (API-compatible). Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
OpenCode's newer opentui TUI still crashes/garbles in the xterm.js terminal and
exits back to the shell ("Connected to Sandbox"). Comment out its add-terminal
entry so users can't select it, matching the existing Droid hidden-until-stable
pattern. Re-enable in a later PR once the TUI issue is fixed.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
This is the sandbox/docker/Dockerfile half of the earlier "install Claude Code via native installer" fix (6bf8125) — it was never actually committed there: that commit's `git add` silently dropped the Dockerfile (a trailing bad pathspec aborted the add), so only the claude-wrapper.sh deletion landed and the VM was rebuilt from the uncommitted working tree. Replaces the npm pre-warm + wrapper approach with the native installer, pointing /usr/local/bin/claude at the self-contained binary in a shared /opt/claude so it resolves for the no-home pty-NNN users. Matches the running VM image. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
applyOpenRouterEnv (OPENAI_BASE_URL for opencode/droid, ANTHROPIC_BASE_URL for Claude Code) and buildCodexOpenRouterCommand still emitted http://localhost:.../broker/..., while the custom-endpoint and gemini-shim paths already used 127.0.0.1. Node/undici resolves "localhost" to IPv6 ::1 first, which the IPv4-only broker never answers -> the request hangs. Make every broker URL 127.0.0.1 for consistency with the product-wide fix. Update the Codex-config test expectation to match (the reviewer's P2 finding, extended to the two sibling call sites that had the same latent bug). Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
The 2.1 walkthrough ran `harbor run --dataset [email protected] --agent oracle` to validate the environment before running the 2-1 dataset -- a version mismatch that validates the wrong task set. Point the oracle pre-check at terminal-bench/terminal-bench-2-1 in the runbook and template setupGuide. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
The SlopCodeBench helper (scb-insert-template) only seeded its own template and its docstring said templates are DB-only, so the new SWE-bench / SWE-bench Pro / Terminal-Bench template.json files had no import path and never appeared in the New Dashboard grid. Add benchmarks/bin/seed-benchmark-templates, which discovers every benchmarks/*/template.json and upserts each (delete-by-name then insert, status='approved') against the desktop d1-shim. Accepts optional dir-name args to scope to a subset. Repoint scb-insert-template at it (delegates for just slopcodebench) so the upsert logic has one source of truth, and document the seeder in benchmarks/README.md. Verified: seeds all 4 templates (item_count matches items, status/featured correct); the wrapper seeds only SlopCodeBench. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01XBxVYcEidYfRm5Gf4j7nPL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.