Feat/benchmark templates swe tbench

benchmarks/README.md

@@ -0,0 +1,80 @@
+# Orcabot benchmark runners
+
+Templates that turn Orcabot into an English-driven harness for running coding
+benchmarks: a Claude Code **orchestrator** terminal you direct in plain language
+sets up and runs the benchmark, runs show up as live panes on the canvas, and
+results surface in a browser/note block.
+
+## Benchmarks
+
+| Benchmark | Dir | Runnable? | Execution model |
+|-----------|-----|-----------|-----------------|
+| slop-code-bench | [`slopcodebench/`](slopcodebench/) | ✅ yes | **Host** (no Docker) via the fork's tmux executor |
+| SWE-bench (Lite/Verified/Full) | [`swebench/`](swebench/) | ✅ yes | Inference on host + **Docker-per-task** eval (Modal or DinD) |
+| SWE-bench Pro (public 731) | [`swebench-pro/`](swebench-pro/) | ✅ yes | Inference on host + **Docker-per-task** eval (`--use_local_docker` or Modal) |
+| Terminal-Bench 2.1 | [`terminal-bench/`](terminal-bench/) | ✅ yes | **Harbor** harness; Docker-per-task (or `--env daytona`/Modal remote) |
+| FrontierCode (Diamond) | [`frontiercode/`](frontiercode/) | ❌ no | **Closed / submission-only** — no public harness, see the note |
+
+## Two architecture families
+
+**Family A — host-executable.** The agent-under-test runs directly in the
+Orcabot sandbox VM; no inner container. The sandbox VM *is* the isolation
+boundary. This is slop-code-bench (via the fork's host-tmux executor), where
+each run is mirrored into an attachable read-only tmux pane.
+
+**Family B — Docker-per-task.** SWE-bench, SWE-bench Pro, and Terminal-Bench run
+each task in its own container — that isolation is core to correctness, so it
+can't be dropped. Two backends:
+
+- **Cloud sandbox backend (recommended, Orcabot-friendly):** the harness
+  offloads container execution to a remote sandbox — **Modal** for
+  SWE-bench/Pro, **Daytona/Modal** for Harbor. The orchestrator stays thin; the
+  Orcabot VM never needs a Docker daemon. Needs a provider token as a secret.
+- **Docker-in-VM (DinD):** run a Docker daemon inside a **Docker-capable
+  benchmark sandbox** (a privileged x86 Fly microVM). Heavier; the desktop VM
+  can't do this. Use when you want everything self-contained.
+
+For SWE-bench and SWE-bench Pro the **inference phase** (agent → predictions
+JSONL) is host-native and needs neither backend — only the **evaluation phase**
+does. The predictions file is the clean handoff boundary.
+
+## Shared pattern (all runnable benchmarks)
+
+1. **Config phase** — the Orcabot chat wizard (`config-wizard.md` in each dir)
+   asks agent / model / target / backend, stores keys in the secrets broker,
+   and provisions the orchestrator terminal + a results block.
+2. **Orchestrator** — a Claude Code terminal you drive in English
+   (`orchestrator-runbook.md`), running the benchmark's native CLI.
+3. **Viewers** — the orchestrator spawns a read-only viewer pane per run via its
+   `create_terminal` MCP tool, using a **read-only** boot command (never a tmux
+   *control* socket across PTYs — that would grant read+inject across sessions
+   and bypass output redaction). The command is benchmark-specific:
+   `tail -n +1 -F <run-logfile>` (slop-code), `docker logs -f <id>` / `tail -f logs/…`
+   (SWE-bench family), `harbor`/Modal log streaming (Terminal-Bench).
+
+## Seeding the templates into D1
+
+These templates are curated, DB-only rows in `dashboard_templates` (there is no
+code seed). To make them selectable in the **New Dashboard** grid on a local
+desktop stack, seed them directly:
+
+```
+benchmarks/bin/seed-benchmark-templates            # all benchmark templates
+benchmarks/bin/seed-benchmark-templates swebench   # just one, by dir name
+```
+
+It upserts each `benchmarks/*/template.json` (delete-by-name then insert,
+`status='approved'`) against the desktop d1-shim (`D1_URL`, default
+`http://127.0.0.1:9001`). `slopcodebench/bin/scb-insert-template` is a thin
+wrapper that seeds only the SlopCodeBench template. For a **remote/cloud** control
+plane, export from an existing dashboard via the templates API (`POST /templates`)
+instead — direct INSERT is the local/curated path.
+
+## Status
+
+All templates here are **prototypes**. slop-code-bench is validated end-to-end
+(see its README). The Family-B templates are grounded in each harness's real
+commands but are **not** validated end-to-end from here, because they need a
+Docker-capable backend (Modal token or DinD sandbox) + provider keys that aren't
+available in this environment. Each dir's README states exactly what's verified
+vs pending.

benchmarks/bin/seed-benchmark-templates

@@ -0,0 +1,94 @@
+#!/usr/bin/env python3
+"""Seed (or refresh) every benchmark dashboard template into D1.
+
+Templates are DB-only -- there is no code seed -- so a curated template has to be
+written to the `dashboard_templates` table directly. This discovers every
+`benchmarks/*/template.json` and upserts each one (delete-by-name then insert),
+status='approved' so they show in the New Dashboard grid immediately.
+
+This is the generic path; `slopcodebench/bin/scb-insert-template` delegates here
+for the single SlopCodeBench template.
+
+Usage:
+  bin/seed-benchmark-templates                  # all templates, desktop d1-shim
+  bin/seed-benchmark-templates swebench         # only benchmarks/swebench/template.json
+  D1_URL=http://127.0.0.1:9001 AUTHOR_ID=dev-desktop bin/seed-benchmark-templates
+
+For a remote/cloud control plane, export a template instead via the templates API
+from an existing dashboard (POST /templates) -- direct INSERT is the local/curated
+path.
+"""
+import datetime
+import json
+import os
+import pathlib
+import sys
+import urllib.request
+import uuid
+
+D1_URL = os.environ.get("D1_URL", "http://127.0.0.1:9001")
+AUTHOR_ID = os.environ.get("AUTHOR_ID", "dev-desktop")
+AUTHOR_NAME = os.environ.get("AUTHOR_NAME", "OrcaBot")
+
+# benchmarks/ root (this file lives in benchmarks/bin/).
+ROOT = pathlib.Path(__file__).resolve().parent.parent
+
+
+def q(sql, params):
+    req = urllib.request.Request(
+        D1_URL.rstrip("/") + "/query",
+        data=json.dumps({"sql": sql, "params": params}).encode(),
+        headers={"Content-Type": "application/json"},
+    )
+    return json.load(urllib.request.urlopen(req))
+
+
+def discover(filter_dirs):
+    """Return the template.json paths to seed, optionally limited to named dirs."""
+    if filter_dirs:
+        paths = []
+        for name in filter_dirs:
+            p = ROOT / name / "template.json"
+            if not p.exists():
+                sys.exit(f"no template.json in benchmarks/{name}/")
+            paths.append(p)
+        return paths
+    return sorted(ROOT.glob("*/template.json"))
+
+
+def upsert(tpl):
+    tid = str(uuid.uuid4())
+    now = datetime.datetime.utcnow().isoformat()
+    q("DELETE FROM dashboard_templates WHERE name = ?", [tpl["name"]])
+    q(
+        """INSERT INTO dashboard_templates
+           (id,name,description,category,author_id,author_name,items_json,edges_json,
+            viewport_json,setup_guide,item_count,is_featured,use_count,status,created_at,updated_at)
+           VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)""",
+        [
+            tid, tpl["name"], tpl["description"], tpl.get("category", "coding"),
+            AUTHOR_ID, AUTHOR_NAME,
+            json.dumps(tpl["items"]), json.dumps(tpl["edges"]),
+            json.dumps(tpl.get("viewport")), tpl.get("setupGuide"), len(tpl["items"]),
+            1, 0, "approved", now, now,
+        ],
+    )
+    return tid
+
+
+def main():
+    paths = discover(sys.argv[1:])
+    # Ensure the setup_guide column exists (idempotent) so older DBs accept the insert.
+    try:
+        q("ALTER TABLE dashboard_templates ADD COLUMN setup_guide TEXT", [])
+    except Exception:
+        pass
+    for path in paths:
+        tpl = json.loads(path.read_text())
+        tid = upsert(tpl)
+        print(f"seeded '{tpl['name']}' (id={tid}) from {path.relative_to(ROOT.parent)}")
+    print(f"done: {len(paths)} template(s) -> {D1_URL}")
+
+
+if __name__ == "__main__":
+    main()

benchmarks/frontiercode/NOT-RUNNABLE.md

@@ -0,0 +1,50 @@
+# FrontierCode (Diamond) — not runnable as an Orcabot template
+
+**Verdict: no runner template is possible.** FrontierCode is real and
+well-identified, but it is **closed / submission-only**. There is nothing to
+`pip install`, no dataset to download, and no public harness to drive — so an
+Orcabot "runner" would be a fiction. This note exists so the gap is documented
+rather than papered over with a fake template.
+
+## What it is
+
+- **FrontierCode** — a coding benchmark from **Cognition AI** (the Devin team),
+  announced June 2026: <https://cognition.ai/blog/frontier-code>
+- Premise: goes *beyond correctness*. Instead of "do the tests pass" (SWE-bench
+  style) it asks **"would an open-source maintainer actually merge this PR?"** —
+  judging regression safety, test quality, scope discipline, and repo
+  conventions. Built with 20+ maintainers across 36 OSS repos.
+- **Diamond** is the hardest nested subset (your GPQA-Diamond analogy is right):
+  Diamond = 50 hardest tasks ⊂ Main = 100 ⊂ Extended = 150. Scores are brutal
+  (Claude Opus 4.8 ≈ 13.4% on Diamond).
+
+## Why it can't be run locally
+
+- **Tasks are deliberately held out** — Cognition states they don't plan to
+  release tasks publicly, to avoid contamination.
+- **No public repo, no HuggingFace dataset, no harness, no download.**
+- **Access is submission-only:** model creators request that Cognition runs the
+  eval. Grading (unit tests + reverse-classical tests + adaptive LLM grading +
+  rubric review + scope checks) happens entirely on Cognition's side.
+- Scores are reported as **model + agent-harness** combinations (Claude Code,
+  Codex, Gemini CLI, mini-swe-agent, Devin) — so it's agentic, but you still
+  can't reproduce it without the held-out tasks and private graders.
+
+## The only partial path (and why it's not a real score)
+
+You *could* point an Orcabot agent harness (Claude Code, mini-swe-agent, …) at
+public OSS repos and grade "mergeability" with your own rubric — but that
+produces a *different* benchmark, not a FrontierCode score, since neither the
+tasks nor the graders match. Not worth a template.
+
+## If/when this changes
+
+If Cognition later publishes tasks or a submission API, the natural Orcabot fit
+would be an **inference-only** template (run the agent harness against the
+provided repos, emit submissions in their format) — mirroring the SWE-bench
+inference phase. Until then: **not runnable.**
+
+_Sources: [Cognition blog](https://cognition.ai/blog/frontier-code) ·
+[HN discussion](https://news.ycombinator.com/item?id=48451723). Not to be
+confused with Epoch AI's **FrontierMath** (a math benchmark with a similar
+hardest-tier structure)._

benchmarks/slopcodebench/README.md

@@ -0,0 +1,145 @@
+# SlopCodeBench Runner for Orcabot
+
+Turn Orcabot into a **1-click, English-driven harness for running
+[slop-code-bench](https://github.com/SprocketLab/slop-code-bench)** — a coding
+agent you direct in plain language sets up and runs the benchmark, and every
+agent-under-test run shows up as a live, read-only pane on the canvas.
+
+This directory is the **orchestration layer**. It pairs with a branch on the
+slop-code-bench fork that adds the no-Docker, watchable executor:
+
+| Piece | Where | What |
+|-------|-------|------|
+| Host (no-Docker) executor + tmux mirror | `robdmac/slop-code-bench` branch `feat/host-tmux-executor` | Runs the agent-under-test directly in the VM and mirrors each run into an attachable tmux window |
+| Orchestration layer (this dir) | `orcabot3` branch `feat/benchmark-tmux-viewers` | Orchestrator runbook, read-only viewer convention, results browser, config wizard, importable template |
+
+## Why this shape
+
+slop-code-bench normally spawns a Docker container per problem. Inside Orcabot
+the sandbox is *already* an isolated VM, so we drop the inner container and run
+on the host — which also sidesteps Docker-in-Docker and works on the desktop
+VM. The cost is weaker isolation/reproducibility than the pinned image (fine for
+relative comparisons; use the Docker env for leaderboard-exact numbers). See
+the fork's `docs/HOST_TMUX_EXECUTOR.md`.
+
+Keeping a Claude Code agent as the **orchestrator** (not a deterministic script)
+is deliberate: you can drive stop/start/chaining/interim-analysis/debugging in
+English via Remote Control, which is the actual product value here.
+
+## Architecture
+
+```
+┌─ Terminal: ORCHESTRATOR ───────────┐        ┌─ Browser block ───────────┐
+│  Claude Code, driven in English     │        │  slop-code Dash dashboard │
+│  runs: slop-code run (local-tmux)   │        │  http://localhost:8050    │
+│  has Orcabot MCP tools              │        └───────────────────────────┘
+│  (create_terminal, connect_nodes)   │
+└───┬───────────┬───────────┬─────────┘
+    │ spawns viewer panes + edges per run
+    ▼           ▼           ▼
+┌─ ▶ run ──┐ ┌─ ▶ run ──┐ ┌─ ▶ run ──┐   boot_command:
+│ tail -F  │ │ tail -F  │ │ tail -F  │   tail -n +1 -F <run-logfile>
+└──────────┘ └──────────┘ └──────────┘   (read-only viewers; no inject)
+        ▲ per-run logfiles (the executor tees each run to one)
+        │
+   host-tmux executor in the slop-code-bench fork
+```
+
+Viewer panes **tail the per-run logfile**, not the executor's tmux socket — a
+separate viewer PTY may run under a different uid, and bridging a tmux *control*
+socket across uids would need world-accessibility (read+inject across sessions,
+bypassing output redaction). Logfile tail is read-only and reaches only that
+run. The executor's tmux session stays private to its own uid (its multiplexing
+convenience); `scb-attach` is for same-uid interactive use only.
+
+## Prerequisites
+
+- **tmux** — already in the sandbox image (`sandbox/docker/Dockerfile`), and the
+  desktop VM rootfs is `docker export`ed from that same image, so it's present
+  on both prod and desktop. No infra change needed.
+- The **fork branch** `feat/host-tmux-executor` checked out in the workspace as
+  `slop-code-bench`, with `uv sync` run.
+- Problems repo `scb-problems` checked out.
+- Keys in the Orcabot **secrets broker** for whatever provider you benchmark.
+
+## Quick start
+
+1. **Config phase** — pick the "SlopCodeBench Runner" template (or ask the
+   Orcabot chat to "set up a benchmark"). The chat wizard
+   (`config-wizard.md`) asks CLI / auth / target / problems, stores keys in the
+   broker, and provisions the orchestrator terminal + results browser.
+2. **Drive the orchestrator** in English (`orchestrator-runbook.md`):
+   - "Run `file_backup` and `execution_server` with claude_code on opus-4.5."
+   - "Show me the runs." → it spawns a read-only viewer pane per run.
+   - "execution_server looks stuck — stop it."
+   - "Score the finished runs and open the dashboard."
+3. **Watch** runs in the viewer panes; **read results** in the browser block.
+
+## Files
+
+| File | Purpose |
+|------|---------|
+| `orchestrator-runbook.md` | System prompt + runbook for the orchestrator agent |
+| `config-wizard.md` | Orcabot-chat config phase (Part 1 of the plan) |
+| `template.json` | Importable `DashboardTemplateWithData` (orchestrator + note + results browser) |
+| `bin/scb-runs` | List active runs + read-only attach commands |
+| `bin/scb-attach` | Attach read-only to a run |
+| `bin/scb-dashboard` | Serve the Dash results dashboard on :8050 |
+
+## Importing the template
+
+`template.json` matches `DashboardTemplateWithData`
+(`frontend/src/types/dashboard.ts`). Templates are **DB-only** — there is no code
+seed and nothing imports this file automatically. To make it appear in the
+**New Dashboard** grid, insert it into the `dashboard_templates` table:
+
+```bash
+benchmarks/slopcodebench/bin/scb-insert-template      # desktop d1-shim :9001
+```
+
+(For a cloud control plane, export a template from an existing dashboard via the
+`POST /templates` API instead — direct INSERT is the local/curated path.)
+
+The orchestrator terminal's `bootCommand` appends the operating briefing to a
+`CLAUDE.md` in its working dir, so Claude Code boots **already knowing** the
+validated `slop-code` commands (this is what was missing when a stock agent had
+"no idea" how to run a benchmark). Viewer panes are intentionally **not** in the
+template; the orchestrator can create them per run at runtime.
+
+> **"Set up a benchmark" in the in-app chat is not a feature.** There is no
+> benchmark chat intent — that phrase just spins up a generic agent with no
+> slop-code context. Use this template (above) instead.
+
+### One-time VM setup (manual, required before driving the orchestrator)
+1. Fork at `/workspace/slop-code-bench` with venv `/workspace/scb-venv`
+   (`uv sync --python 3.12`, managed python on `/workspace/.uv-python`). On
+   desktop this is the host-shared workspace, so it persists across reboots.
+2. Add your provider key in the **secrets panel**, broker-protected
+   (e.g. `OPENROUTER_API_KEY`).
+
+Then drive the orchestrator in English: *"Run `file_backup` with opencode on
+kimi-k2.6."* The validated arm + the `localhost`→`127.0.0.1` broker note live in
+`RESUME-kimi-opencode.md`.
+
+## Status — prototype
+
+**Validated**
+- Host-tmux executor: unit + real-tmux lifecycle tests pass; end-to-end smoke
+  run confirms harness output is unchanged while a tmux window mirrors it
+  (`feat/host-tmux-executor`).
+- `scb-runs` / `scb-attach` against a real tmux session; read-only resolution.
+- `template.json` parses; every embedded `content` string is valid.
+
+**Not yet validated end-to-end (needs a live Orcabot VM + keys)**
+- A full `slop-code run` of a real problem under `local-tmux-py.yaml` with a
+  real agent CLI.
+- The orchestrator calling `create_terminal`/`connect_nodes` to spawn viewer
+  panes from inside an Orcabot terminal.
+- The Dash dashboard rendering in a browser block at :8050.
+- Skills-variant copy-in path for OMC / SuperPowers / Karpathy (only GSD has a
+  ready-made local env + prompt; others follow the documented pattern).
+
+**Deliberately deferred**
+- Docker-env parity path (for leaderboard-exact runs) — orthogonal; the Docker
+  configs already exist upstream.
+- Per-run cgroup memory caps for safe parallelism — documented, not wired.