Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions .claude/skills/build-s1a-agent/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: build-s1a-agent
description: Build a new System 1 agent (an openJiuwen agent with a System 1 decision model in its model slot) for a task the user names, in this repository. Runs a fit probe first, then scaffolds one agent module from the template of the right front (tool, browser or rail), its test and its README row, and verifies it slot by slot. Use when the user asks to build, add or scaffold a System 1 agent, an S1A agent, a Jev agent, a new game, task, site or rail for s1a.
description: Build a new System 1 agent (an openJiuwen agent with a System 1 decision model in its model slot) for a task the user names, in this repository. Runs a fit probe first, then scaffolds one agent module from the template of the right front (tool, browser or rail), its test and its README row, and verifies it model by model. Use when the user asks to build, add or scaffold a System 1 agent, an S1A agent, a Jev agent, a new game, task, site or rail for s1a.
---

# Build a System 1 agent
Expand Down Expand Up @@ -47,7 +47,7 @@ Copy the front's template from `s1a/agents/_templates/` to `s1a/agents/<name>.py
part; the template's comments say what each part is. `SPEC.name` is the module name. Write
`tests/test_agents_<name>.py` with the two tool-front classes of `tests/test_templates.py`, `TestNimEnv` and
`TestNimThroughTheLoop`, rewritten for the new module: the env's reset, candidates and winning line, then the rule
and random slots through `series.play` with `loop.WORKSPACE` and `series.optional_chat_model` patched as there.
and random models through `series.play` with `loop.WORKSPACE` and `series.optional_chat_model` patched as there.
For a browser agent, copy `TestBrowserTemplateOffline`: the spec reaches the faked subagent through
`support.browse_offline`. For a rail, copy `TestRailTemplateOffline`: precision and recall on five hand-labelled
records through a `ScriptedModel(noul=[...])` from `s1a.decision_models`. Add one row to the agents table in `README.md`. Follow `references/state-design.md` for the
Expand All @@ -60,20 +60,20 @@ Run each command, read its output, fix the agent before the next rung. Stop at t

```bash
uv run pytest tests/test_agents_<name>.py -q # the adapter contract, no keys
uv run s1a run <name> --slot random --rethink off --episodes 3 # mechanics through the loop, no keys
uv run s1a run <name> --slot rule --rethink off --episodes 3 # when a baseline exists
uv run s1a run <name> --slot jev --rethink off --episodes 3 --log # keys: latency, invalid keys must be 0
uv run s1a run <name> --slot llm --rethink off --episodes 3 # the same seeds with the chat model
uv run python -m evals.table evals/results # one row per slot
uv run s1a run <name> --model random --rethink off --episodes 3 # mechanics through the loop, no keys
uv run s1a run <name> --model rule --rethink off --episodes 3 # when a baseline exists
uv run s1a run <name> --model jev --rethink off --episodes 3 --log # keys: latency, invalid keys must be 0
uv run s1a run <name> --model llm --rethink off --episodes 3 # the same seeds with the chat model
uv run python -m evals.table evals/results # one row per model
uv run pytest tests -q # the whole suite stays green
```

Every `run` prints one JSON object: the series summary with `scored`, the episodes that got a score, `errors`, the
count the model in the slot could not play, and `job_dir`. The `random` and `rule` slots exist for the tool front only; `laya`
(Laya in process, after `uv sync --extra laya`) fills any slot `jev` does, and so does `cua` (Cua-S1 Nano, after
count the model could not play, and `job_dir`. `random` and `rule` exist for the tool front only; `laya`
(Laya in process, after `uv sync --extra laya`) runs wherever `jev` does, and so does `cua` (Cua-S1 Nano, after
`uv sync --extra cua`) except on a rail. For a browser or rail agent the
key-free rung is the offline test from step 4. The paid rung follows. A browser agent runs one task per call and
needs the chat-model key and a Jev key: `uv run s1a run <name> --slot jev --goal "..."`. A rail runs its
needs the chat-model key and a Jev key: `uv run s1a run <name> --model jev --goal "..."`. A rail runs its
labelled set and needs a Jev key: `uv run s1a run <name> --labelled-set records.jsonl`.

Report the table and stop. Series of a hundred episodes cost money; ask the user before starting one.
6 changes: 3 additions & 3 deletions .claude/skills/build-s1a-agent/references/fronts.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@ Every agent states every field. The spec classes in `s1a/spec.py` define no defa
## Tool front: `ToolAgentSpec`, template `_templates/tool_agent.py`

A DeepAgent plays through two tools, `observe` and `act`, with a System 1 decision model in the model slot
(`--slot jev|llm|random|rule|laya|cua`).
(`--model jev|llm|random|rule|laya|cua`).

| field | meaning |
|---|---|
| `name` | the module name: lowercase letters, digits and `_` after a letter; also the job folder name |
| `description` | one sentence, shown by `list_agents` and the caller skill |
| `rules` | the text the model in the slot reads on every decision |
| `rules` | the text the model reads on every decision |
| `budget` | `Budget(max_steps, timeout_s, stall_after)`; the defaults of `--max-steps` and `--timeout` |
| `flags` | `(ArgumentParser) -> None`: the agent's own switches, after the shared ones |
| `series` | `(Namespace) -> Series`: the seeds, `env_for(seed)`, the page `session`, the `baseline`, `annotate` |
Expand All @@ -24,7 +24,7 @@ window's clickable elements as candidates, plus `done` and `abstain`, and `s1a/d

## Browser front: `BrowserAgentSpec`, template `_templates/browser_agent.py`

openJiuwen's browser subagent with `BrowserDecisionModel` in the slot (`--slot jev` for TypeSafe Jev, `laya` for
openJiuwen's browser subagent with `BrowserDecisionModel` in the slot (`--model jev` for TypeSafe Jev, `laya` for
Laya in process, `cua` for Cua-S1 Nano in process, `llm` for the chat model alone); the chat model types values and writes the answer.

| field | meaning |
Expand Down
6 changes: 3 additions & 3 deletions .claude/skills/build-s1a-agent/references/state-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,16 +22,16 @@ What the loop sends Jev on every decision: the observation, the candidates, the

- Under about 120 words. Facts to recognise, phrased as what a good move looks like. Leave out chains of reasoning.
- Name the winning shapes and the traps in the state's own words so that Jev can match them by recognition.
- Every slot reads the same text, including the chat-model arm.
- Every model reads the same text, including the chat-model arm.

## The budget and the baseline

- `max_steps`: the acts one episode may spend; the chat slot gets twice as many iterations for malformed calls.
- `max_steps`: the acts one episode may spend; the chat model gets twice as many iterations for malformed calls.
- `timeout_s`: the wall clock per episode; a timed-out episode keeps its score so far.
- `stall_after`: acts without a score change before the rethink rail asks the chat model for a plan; 0 for games
where every act changes the score or the question.
- Give a `baseline` whenever a rule or an expert plan is known; it is the upper or lower bound the table shows.
- The `random` slot draws from its own stream, derived from the episode seed. An env opponent seeded from the plain
- The `random` model draws from its own stream, derived from the episode seed. An env opponent seeded from the plain
integer seed, as the template does, draws a different stream.
- In the summary a score above 0 counts as a win and below 0 as a loss. A draw scored 0.5 counts as a win there.

Expand Down
6 changes: 3 additions & 3 deletions .env.example
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Jev: a direct TypeSafe key. Leave TYPESAFE_API_URL unset with it; setting that URL selects the OpenRouter proxy below.
TYPESAFE_API_KEY=

# The chat model (typed values, final answers, the llm slot, the rethink planner); every slot that calls it needs MODEL_NAME.
# The chat model (typed values, final answers, `--model llm`, the rethink planner); every model that calls it needs MODEL_NAME.
# OPENROUTER_* stands in for LLM_API_KEY and LLM_BASE_URL when those are unset.
OPENROUTER_API_KEY=
OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
Expand All @@ -27,14 +27,14 @@ MODEL_NAME=google/gemini-2.5-flash
# HF_HOME=~/.cache/huggingface # where the laya and cua checkpoints download on first use
# HF_HUB_OFFLINE=1 # after the first run, no network for the checkpoints

# ---- Laya (the in-process decision model behind --slot laya; needs `uv sync --extra laya`) ----
# ---- Laya (the in-process decision model behind --model laya; needs `uv sync --extra laya`) ----
# LAYA_MODEL=convaiinnovations/laya
# LAYA_SUBFOLDER=multilingual # or typed-decisions
# LAYA_DEVICE=cpu # cuda when available
# LAYA_MAX_LEN=1024
# LAYA_HEAD_MAX_LEN=512 # raise for choice questions with many options

# ---- Cua-S1 Nano (the in-process option scorer behind --slot cua; needs `uv sync --extra cua`) ----
# ---- Cua-S1 Nano (the in-process option scorer behind --model cua; needs `uv sync --extra cua`) ----
# CUA_S1_CHECKPOINT=cua-ai/cua-s1-nano-0.1 # a Hugging Face id, or a local directory holding <subfolder>/
# CUA_S1_SUBFOLDER=text # the text-only checkpoint; the window is 256 bytes of state
# CUA_S1_DEVICE=auto # cpu, cuda, mps
Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,5 +22,5 @@ uv run s1a ...

- OS and Python version:
- `uv sync` extras installed (`blackjack`, `alfworld`, `laya`, `cua`, ...):
- slot (`jev`, `laya`, `cua`, `llm`) and, for `jev`, direct key or OpenRouter:
- `--model` (`jev`, `laya`, `cua`, `llm`) and, for `jev`, direct key or OpenRouter:
- commit (`git rev-parse --short HEAD`):
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: Feature request
about: A new agent, slot, rail or eval
about: A new agent, decision model, rail or eval
labels: enhancement
---

Expand Down
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@

The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versions follow semver.

## Unreleased

### Changed

- `--model` picks the model on every agent, on `decide` and on `probe`: `jev`, `laya`, `cua`, `llm`, `random` or
`rule`. The results table's column, the replay page's badge data and a browser run's `answer.json` name it
`model` as well; the replay still reads the `slot` key of records written by 0.1.0.

## 0.1.0 - 2026-09-23

### Added
Expand Down
6 changes: 3 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,8 @@ Everything outside `openjiuwen` is an extra. An agent whose extra is missing say
| `alfworld` | alfworld, textworld | `s1a run alfworld`; also `ALFWORLD_DATA` and Python 3.11, see `evals/README.md` |
| `alfworld-visual` | the `alfworld` extra, ai2thor 2.1.0, torch | `evals/replay/thor_replay.py`, the AI2-THOR scene behind an ALFWorld trial in the replay page; the 400 MB Unity build downloads on first use |
| `report` | pillow, playwright | `python -m evals.replay`, the showcase pages and GIFs; `--gif` also needs `uv run playwright install chromium` |
| `laya` | laya (torch, transformers) | `--slot laya` on every agent and on `decide` and `probe`: Laya in process, no Jev key; the checkpoint downloads into the Hugging Face cache (`HF_HOME`) on first use |
| `cua` | cua-s1 (torch), huggingface-hub | `--slot cua` on tool and browser agents and on `decide` and `probe`: Cua-S1 Nano in process; the 3 MB checkpoint downloads into the Hugging Face cache (`HF_HOME`) on first use |
| `laya` | laya (torch, transformers) | `--model laya` on every agent and on `decide` and `probe`: Laya in process, no Jev key; the checkpoint downloads into the Hugging Face cache (`HF_HOME`) on first use |
| `cua` | cua-s1 (torch), huggingface-hub | `--model cua` on tool and browser agents and on `decide` and `probe`: Cua-S1 Nano in process; the 3 MB checkpoint downloads into the Hugging Face cache (`HF_HOME`) on first use |
| `dev` | pytest, pytest-asyncio, ruff, ty | the test suite, `scripts/smoke.sh` and the lint and type checks |

`uv sync --all-extras` installs all seven. The CLI runs from a checkout; a wheel install (`uv tool install`,
Expand All @@ -87,7 +87,7 @@ The `desktop` agent runs on Windows or macOS and has no extra. It needs Cua Driv

`.env` needs a key for Jev (`TYPESAFE_API_KEY` direct, or `OPENROUTER_API_KEY` for the proxy) and for the chat
model (`OPENAI_API_KEY` or `LLM_API_KEY`, `MODEL_NAME`; `OPENROUTER_API_KEY` and `OPENROUTER_BASE_URL` stand in
for `LLM_*` when those are unset). Exported variables win over the file. `--slot laya` and `--slot cua` need no
for `LLM_*` when those are unset). Exported variables win over the file. `--model laya` and `--model cua` need no
decision key. A host agent (Claude Code, Codex, Cursor, Hermes) reaches the keys through its own environment or
the checkout's `.env`.

Expand Down
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
† The first Allrecipes task of the [WebVoyager](https://github.com/MinorJerry/WebVoyager) task set
([He et al., 2024](https://arxiv.org/abs/2401.13919), Apache-2.0, attribution in [NOTICE](NOTICE)): a vegetarian
lasagna with over 100 reviews, 4.5 stars or more, for 6. The chat model of that row is Claude Fable 5.1 through
OpenRouter; both slots pay it for the typed search text and the answer. \* Estimated; the chat-model run recorded no
OpenRouter; both models pay it for the typed search text and the answer. \* Estimated; the chat-model run recorded no
cost. Each replay below is the episode behind its row, Jev on the left and the chat model on the right, both on the
wall clock. The other Allrecipes runs, longer games and the Google Flights driver comparison:
[docs/benchmarks.md](docs/benchmarks.md).
Expand Down Expand Up @@ -82,18 +82,18 @@ uv sync && cp .env.example .env # the first sync resolves the openjiuwen pin
```

Put a Jev key in `.env` (`TYPESAFE_API_KEY` from the [TypeSafe console](https://console.typesafe.ai), or
`OPENROUTER_API_KEY`), then ask for one decision and run one agent on both slots:
`OPENROUTER_API_KEY`), then ask for one decision and run one agent with each model:

```bash
uv run s1a decide --state '{"player_total": 18, "dealer_upcard": 9}' \
--option hit="take a card" --option stand="keep the hand" --rules "stand on 17 or more"
uv sync --extra blackjack
uv run s1a run blackjack --slot jev --rethink off --episodes 20
uv run s1a run blackjack --slot llm --rethink off --episodes 20 # the chat model in the same slot
uv run s1a run blackjack --model jev --rethink off --episodes 20
uv run s1a run blackjack --model llm --rethink off --episodes 20 # the chat model in the same agent
```

`decide` prints one JSON object with `choice`, a probability per option, `confidence` and `ms`; `run` writes a job
folder with the score. Without a key, `--slot cua` answers in process after `uv sync --extra cua`.
folder with the score. Without a key, `--model cua` answers in process after `uv sync --extra cua`.

### As an MCP server

Expand Down Expand Up @@ -138,7 +138,7 @@ interface fits: [docs/architecture.md](docs/architecture.md), [docs/decision-mod
- [docs/benchmarks.md](docs/benchmarks.md): the six runs above, the Google Flights driver comparison, a longer game, the guard rail.
- [docs/skills.md](docs/skills.md): the caller skill, the builder skill, what to delegate.
- [docs/agents.md](docs/agents.md): every agent with its flags, run command and extra.
- [docs/architecture.md](docs/architecture.md) and [docs/decision-models.md](docs/decision-models.md): the fronts, the slot, the model interface, adding a backend.
- [docs/architecture.md](docs/architecture.md) and [docs/decision-models.md](docs/decision-models.md): the fronts, the model slot, the model interface, adding a backend.
- [docs/browser-front.md](docs/browser-front.md): the browser policy, decision by decision.
- [docs/why.md](docs/why.md): the problem, the philosophy, the precedents.
- [docs/roadmap.md](docs/roadmap.md) and [CHANGELOG.md](CHANGELOG.md).
Expand Down
2 changes: 1 addition & 1 deletion agents/s1a-browser.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ You run one page task through the s1a command and report the result.

1. Turn the request into one goal sentence: the site URL first, the values to enter, and the stop condition
("Stop when the matching results are visible").
2. Run `uv run --project ${CLAUDE_PLUGIN_ROOT} s1a run flights --slot jev --goal "<goal>"`.
2. Run `uv run --project ${CLAUDE_PLUGIN_ROOT} s1a run flights --model jev --goal "<goal>"`.
3. Read the JSON object on stdout. When `ok` is true, answer with `final` and name `terminal.url` and
`terminal.title`. When `ok` is false, report `error`, then `status` and the last actions in `report.history`
when present (a timeout leaves `error` only). A BLOCKED or timed-out run exits 0 with `ok` false. Exit 1 with
Expand Down
Loading
Loading