diff --git a/CHANGELOG.md b/CHANGELOG.md index c258aa4..437765b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,34 @@ All notable changes to lcode are documented here. The format follows ## [Unreleased] +### Fixed + +Found in a real session that spent an hour on an Android build whose downloads this machine +couldn't reach, then ended without a word: + +- **A site this machine can't reach** now stops the guessing. lcode tracks, per site, the calls that + failed to reach it (HTTP errors, names that don't resolve, refused or timed-out connections) and + the ones that got an answer. After three different failed calls and none that worked, the model is + told to check one address that must exist and, if that fails too, to tell you what fails and what + you could do. Two more and lcode stops the request (status `blocked`). Before, every new address + looked like a new attempt, so nothing stepped in. +- **Thinking in circles** is stopped as it streams. A model that repeats the same few paragraphs + used to think until the context was full (here 70,000 characters, about ten minutes behind the + spinner). Now the reply is dropped once its sentences keep repeating, and the model is asked for + its next step or what's blocking it. +- **A reply with no answer** no longer ends the request as if it were done. An empty reply, or one + that filled the context before it said anything, gets a second chance; the second time lcode stops + (status `loop`) and offers the stronger model first, when one is set. An answer the context cut off + midway now says so. OpenAI-compatible servers report a cut-off reply too. +- **A `cd` out of the project** no longer moves lcode there. A model that ran `cd /tmp` took the + session, the project checks and the memory notes with it, and the session was saved under `/tmp`, + so `lcode -c` in the project couldn't find it. Now a `cd` within the project is kept, and one out + of it isn't: the next command starts in the project again. Sessions are saved under the folder + lcode works in. +- **Scripts it never ran** are sent back once before the answer counts: a model that wrote + `build_android.sh` and called the job done never ran it, and its first line failed. It now runs it + and fixes what fails, or says plainly that it's untested and why. + ## [0.35.0] - 2026-10-14 ### Added diff --git a/docs/automation.md b/docs/automation.md index 837ab14..8784d4b 100644 --- a/docs/automation.md +++ b/docs/automation.md @@ -50,7 +50,7 @@ went: `0` success, `1` an error, `3` stopped at `--max-steps`, `4` stopped becau | Field | | |---|---| -| `status` | `success`, `error`, `max_steps`, `loop` (the model kept repeating the same call) or `interrupted` | +| `status` | `success`, `error`, `max_steps`, `loop` (the model kept repeating the same call, or its replies went nowhere), `blocked` (a site it needed couldn't be reached) or `interrupted` | | `text` | The model's final answer | | `tool_calls` | Every tool call, with its arguments and output (the first 4,000 characters). `error` is true when the call failed or wasn't allowed | | `files_changed` | Files the request added (`A`), modified (`M`) or deleted (`D`), shell commands included | diff --git a/docs/configuration.md b/docs/configuration.md index fee070d..a9711a3 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -48,7 +48,7 @@ lcode config path # print the file location | `escalate_model` | — | A [stronger model](usage.md#a-stronger-model-when-stuck) to ask, with your OK, when the model is stuck: `anthropic/…`, `openai/…`, `xai/…` (keys from `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `XAI_API_KEY`), `ollama/`, or `custom/` with `escalate_url` | | `escalate_url` | — | Its server, when it isn't the provider's usual one: another Ollama, or any OpenAI-compatible API | | `escalate_price` | — | Its price in dollars per million tokens, input/output (e.g. `3/15`), for the cost shown before each use | -| `repeat_limit` | `3` | Identical tool calls with identical results before lcode [steps in](how-it-works.md#when-the-model-repeats-itself); `0` turns it off | +| `repeat_limit` | `3` | Identical tool calls with identical results before lcode [steps in](how-it-works.md#when-the-model-repeats-itself); `0` turns it off, along with the checks for [unreachable sites](how-it-works.md#when-a-site-cant-be-reached) and [thinking in circles](how-it-works.md#when-a-reply-goes-nowhere) | | `secret_check` | `true` | Before `git commit` and `git push`, look for passwords, keys and tokens they'd add ([details](usage.md#secrets-in-commits)) | | `checkpoints` | `true` | Save a checkpoint before the model changes files, so [`/undo`](usage.md#undo-and-checkpoints) can restore them | diff --git a/docs/how-it-works.md b/docs/how-it-works.md index be2fb26..5471a09 100644 --- a/docs/how-it-works.md +++ b/docs/how-it-works.md @@ -59,7 +59,37 @@ A result that changes isn't a repeat, so a test that's run again after a fix, or that's checked while it runs, never adds up. A call that may have changed something, like an edit or a shell command that isn't read-only, starts the count again for the other calls: checking the build after each edit is fine. Change the threshold with `lcode config set repeat_limit 5`, or turn it off -with `0`. +with `0` (that also turns off the checks for unreachable sites and for thinking in circles, below). + +### When a site can't be reached + +A download that fails however its address is written usually means this machine can't reach the +site: a firewall, a proxy, a regional block, or no internet. No address the model tries will fix +that, and every attempt differs, so the repeat count above never fires. lcode keeps track, per site, +of the calls that failed to reach it (an HTTP error, a name that doesn't resolve, a refused or +timed-out connection) and of the ones that got an answer. + +- **When three different calls failed for a site and none worked,** the result gets a note: check + one address on that site that must exist, and if that fails too, stop and tell you what fails + and what you could do (another network or a VPN, a proxy, a mirror, or running that step + somewhere else, such as CI). +- **Two more,** and lcode ends the request with `Stopped: every request to dl.google.com failed (5 + calls: HTTP 404)…`. The JSON output's status is `blocked`. + +A site that answered once in the request is never counted as out of reach, so a wrong address on a +working site doesn't add up. Local addresses (localhost, private networks, names without a dot) +aren't tracked. + +### When a reply goes nowhere + +Small models sometimes think in circles: a few paragraphs, over and over, until the context is full. +That takes minutes behind the spinner and ends without an answer. lcode watches the thinking as it +streams, and when its sentences keep repeating, it stops the model, drops that reply and asks for a +short one: the next tool call, or what's blocking it. An empty reply, or one that filled the context +before it said anything, gets the same second chance. The second time in a request, lcode stops it +(status `loop`), offering the [stronger model](usage.md#a-stronger-model-when-stuck) first when one +is set. An answer the context cut off midway is kept, with a warning to `/compact` or use a larger +`/context`. ## Subagents diff --git a/docs/usage.md b/docs/usage.md index 38c817f..f524bc7 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -101,7 +101,7 @@ reasoning is on. | `list_dir` | Directory tree, skipping `.git`, `node_modules`, virtualenvs and caches | | `glob` | Find files by pattern, newest first | | `grep` | Regex search with ripgrep (falls back to Python if ripgrep is missing) | -| `bash` | Run a shell command with live output, a timeout and a persistent working directory; with `background`, keep it running ([details](#background-commands)) | +| `bash` | Run a shell command with live output, a timeout and a persistent working directory (a `cd` within the project is kept, one out of it isn't); with `background`, keep it running ([details](#background-commands)) | | `bash_output`, `bash_stop` | Read what a background command printed since last time; stop it | | `todo_write` | Keep a visible task list for multi-step work, with notes (decisions, findings) that survive when old messages are removed; `/status` shows them | | `web_search` | Search the web for current information (needs a [search provider](#web-search)) | @@ -183,6 +183,13 @@ command = "uv run pytest -q && uv run ruff check ." Or set it for yourself with `lcode config set checks "make check"`; `lcode config set checks off` turns checks off. +**Scripts it never ran.** A model that writes a build or install script tends to call it done +without running it once, and the first thing you see is its first error. When a request wrote a +shell script (a `.sh` file, or a file with a shell shebang) and no command in it ran the script, +lcode sends it back once before the answer counts: run it and fix what fails, or say plainly that +it's untested and why, when running it would install, deploy or delete things. Making it executable +or reading it doesn't count as running it. `checks = "off"` turns this off too. + ## A review before the answer Models claim more than they did: half of a two-part request done, "it works now" without having @@ -224,8 +231,10 @@ lcode config set escalate_price 3/15 # dollars per million tokens ``` **When.** lcode offers it when the [loop breaker](how-it-works.md#when-the-model-repeats-itself) -would stop the request, and when the project's [checks](#checks-before-done) fail twice in a row -after the model's fixes, at most once per request. You can also ask at any time: `/escalate`, or +would stop the request, when the model's replies [go nowhere](how-it-works.md#when-a-reply-goes-nowhere) +twice, and when the project's [checks](#checks-before-done) fail twice in a row after the model's +fixes, at most once per request. Not for a [site this machine can't reach](how-it-works.md#when-a-site-cant-be-reached): +a stronger model can't reach it either. You can also ask at any time: `/escalate`, or `/escalate the window still grows` to say what's wrong. **Nothing is sent without asking**, in any permission mode: diff --git a/src/lcode/agent.py b/src/lcode/agent.py index f8914cf..f1877a6 100644 --- a/src/lcode/agent.py +++ b/src/lcode/agent.py @@ -51,7 +51,7 @@ from lcode.ollama import Ollama, OllamaError from lcode.permissions import LEADING_CD, Permissions from lcode.render import MarkdownStreamer -from lcode.repeats import Repeats +from lcode.repeats import Circles, Repeats from lcode.sandbox import Sandbox, SandboxError, project_root from lcode.steer import NOTE_INTRO, Steering from lcode.tools import ( @@ -71,6 +71,7 @@ tree, truncate, ) +from lcode.unreachable import NOTE_AT, Sites GPU_MEMORY_ERRORS = ( "out of memory", @@ -80,6 +81,27 @@ ) SAFE_NUM_BATCH = 512 # Ollama's default prompt batch MAX_STEPS_PER_TURN = 150 +MAX_STALLS = 2 # replies that go round in circles or say nothing, per request, before lcode stops it +STALLED = { + "circles": "the reply went round in circles, so lcode stopped it", + "cut_off": "the reply filled the context before it said anything", + "": "the reply was empty", +} + + +def stall_note(trouble: str) -> str: + """What the model is told after a reply that went round in circles or ended without a word.""" + first = { + "circles": "Your last reply went round in circles (the same thoughts over and over), so lcode stopped it.", + "cut_off": "Your last reply filled the rest of the context before you said anything.", + "": "Your last reply was empty.", + }[trouble] + return ( + f"[lcode] {first} Don't deliberate at length now. Either make your next tool call, or, if you're stuck, " + "tell the user in a few lines what you did, what fails, and what they could do." + ) + + EVENT_OUTPUT = 4000 # characters of a tool's output in an event # How a tool result starts when the call failed or wasn't allowed (an "error" in events and results). REFUSED = ("Error:", "A deny rule", "The user denied", "This needs the user's permission") @@ -319,6 +341,9 @@ def __init__(self, ollama: Ollama, settings: Settings, cwd: Path, console: Conso self.ollama = ollama self.settings = settings self.cwd = cwd.resolve() + # The folder lcode works in: where it started, or where /cd or a resumed session took it. A shell + # command may move the working directory around its project (self.cwd), but not out of it. + self.workspace = self.cwd self.perms = Permissions(self.console, settings.permission_mode) self.tools = Toolbox(self) self.session_id = self.new_session_id() @@ -377,8 +402,10 @@ def __init__(self, ollama: Ollama, settings: Settings, cwd: Path, console: Conso self.escalate_approved: bool | None = None # decided in advance (lcode -p --escalate, the benchmark) self.sent_chars = 0 # characters of the conversation the model had already read self.counted_chars: int | None = None # characters that ctx_used covers (None: all of them) - self.turn_status = "" # how the last request ended: success, max_steps, loop + self.turn_status = "" # how the last request ended: success, max_steps, loop, blocked self.repeats = Repeats(settings.repeat_limit) # notices when the model repeats itself + self.sites = Sites(NOTE_AT if settings.repeat_limit else 0) # notices a site this machine can't reach + self.reply_trouble = "" # what went wrong with the last reply: "circles", "cut_off" or "" self.response = None # the streaming response, so another thread can abort it self.ctx_used = 0 self.last_speed = 0.0 @@ -648,7 +675,7 @@ def save(self) -> None: f = self.session_file() f.parent.mkdir(parents=True, exist_ok=True) data = { - "cwd": str(self.cwd), + "cwd": str(self.workspace), # where `lcode -c` there finds it, wherever the shell moved "model": self.settings.model, "name": self.session_name, "title": self.session_title, @@ -687,7 +714,7 @@ def load(self, info: sessions.SessionInfo) -> str: saved_cwd = Path(info.cwd) if info.cwd else self.cwd if saved_cwd != self.cwd: if saved_cwd.is_dir(): - self.cwd = saved_cwd.resolve() + self.cwd = self.workspace = saved_cwd.resolve() note = f"Working directory is now {self.cwd}" else: note = f"The session's directory {saved_cwd} no longer exists; staying in {self.cwd}" @@ -707,7 +734,7 @@ def load(self, info: sessions.SessionInfo) -> str: return note def load_latest(self) -> bool: - latest = sessions.list_sessions(self.cwd, limit=1) + latest = sessions.list_sessions(self.workspace, limit=1) if latest: self.load(latest[0]) return bool(latest) @@ -805,8 +832,11 @@ def stop_spinner() -> None: status = None start_spinner() + circles = Circles() + self.reply_trouble = "" + stream = self.chat(self.messages, self.tool_schemas(), self.settings.think) try: - for chunk in self.chat(self.messages, self.tool_schemas(), self.settings.think): + for chunk in stream: if self.cancel is not None and self.cancel.is_set(): raise KeyboardInterrupt msg = chunk.get("message", {}) @@ -822,6 +852,9 @@ def stop_spinner() -> None: stop_spinner() # raw text is printed as it streams self.console.print(Text(msg["thinking"], style="dim italic"), end="") printed_thinking = True + if self.settings.repeat_limit and circles.feed(msg["thinking"]): + self.reply_trouble = "circles" + break # closing the stream below stops the model if msg.get("content"): if printed_thinking: self.console.print("\n") @@ -846,9 +879,12 @@ def stop_spinner() -> None: self.messages.append({"role": "assistant", "content": content + "\n[interrupted by user]"}) raise finally: + stream.close() stop_spinner() if watch is not None: watch.stop() + if final.get("done_reason") == "length" and not self.reply_trouble: + self.reply_trouble = "cut_off" # it filled the context (or hit the output limit) mid-reply if printed_thinking: self.console.print() md.flush() @@ -918,6 +954,12 @@ def checks_prompt(self) -> str: "is final, and gives you the failures to fix. Run them yourself while you work, too." ) + def unrun_scripts(self, commands: list[str]) -> list[str]: + """Shell scripts the model wrote in this request and never ran (only where the checks would run).""" + if self.allowed_tools is not None or self.planning() or self.no_changes or self.settings.checks == "off": + return [] + return checks.unrun_scripts(project_root_or_cwd(self.cwd), list(self.tools.before), commands) + def project_check(self) -> checks.Check | None: """The project's check command, for the main agent outside plan mode and reviews (None otherwise).""" if self.allowed_tools is not None or self.planning() or self.no_changes or self.settings.checks == "off": @@ -1106,6 +1148,7 @@ def _run_turn(self, user_text: str) -> None: self.messages.append({"role": "user", "content": self.expand_mentions(user_text) + note}) self.turn_requests = [user_text.strip()] self.repeats.reset() + self.sites.reset() self.check_results = [] self.review_results = None self.escalations = [] @@ -1116,6 +1159,10 @@ def _run_turn(self, user_text: str) -> None: check_rounds = 0 malformed = 0 looping = "" # the call the model kept repeating, when lcode stopped the request for it + blocked = None # the site that couldn't be reached, when lcode stopped the request for it + stalls = 0 # replies that went round in circles or ended without a word + commands: list[str] = [] # shell commands run in this request + scripts_noted = False # the model was told about scripts it wrote and never ran for step in range(self.max_steps): if self.cancel is not None and self.cancel.is_set(): raise KeyboardInterrupt @@ -1165,9 +1212,46 @@ def _run_turn(self, user_text: str) -> None: } ) continue + if not calls and (self.reply_trouble == "circles" or not (reply.get("content") or "").strip()): + stalls += 1 + self.messages.pop() # nothing to keep: no answer, and circles would only fill the context + if stalls < MAX_STALLS: + self.console.print( + Text(f" ⎿ {STALLED[self.reply_trouble]}: asking for a short answer", style="yellow") + ) + self.messages.append({"role": "user", "content": stall_note(self.reply_trouble)}) + continue + why = "its replies went round in circles or ended without an answer, twice in this request" + if not escalated and self.can_escalate(): + escalated = True + advice = escalate.run(self, why) + if advice: + self.messages.append({"role": "user", "content": advice}) + stalls = 0 + continue + self.turn_status = "loop" + self.console.print( + Text( + "Stopped: the model's replies went round in circles or ended without an answer, twice in " + "this request. Try again with a smaller step, or tell it what to do differently.", + style="yellow", + ) + ) + return if not calls: + if self.reply_trouble == "cut_off": + self.say( + "The answer was cut off: the context window is full. /compact or a larger /context helps.", + warning=True, + ) if self.steering is not None and self.steering.waiting(): continue # the user added a note while the model wrote its answer: it reads the note first + unrun = [] if scripts_noted else self.unrun_scripts(commands) + if unrun: + scripts_noted = True + self.console.print(Text(f" ⎿ never ran {', '.join(unrun)}: asking the model to", style="yellow")) + self.messages.append({"role": "user", "content": checks.unrun_note(unrun)}) + continue check = self.project_check() if changed and not verified else None if check is not None and check_rounds < checks.MAX_ROUNDS: check_rounds += 1 @@ -1247,6 +1331,21 @@ def _run_turn(self, user_text: str) -> None: ) if verdict.stop: looping = self.describe_call(name, args) + out_of_reach = self.sites.check(name, args, result) + if out_of_reach.note and not looping: + result += out_of_reach.note + what = "stopping the request" if out_of_reach.stop else "told the model" + self.console.print( + Text( + f" ⎿ {out_of_reach.calls} calls to {out_of_reach.site} failed " + f"({out_of_reach.what}), none worked: {what}", + style="yellow", + ) + ) + if out_of_reach.stop: + blocked = out_of_reach + if name == "bash": + commands.append(str(args.get("command", ""))) if name in ("edit_file", "write_file", "apply_patch") and not result.startswith(REFUSED): changed, verified = True, False elif name == "bash" and changed and self.model_ran_checks(str(args.get("command", "")), result): @@ -1264,7 +1363,7 @@ def _run_turn(self, user_text: str) -> None: "output": result[:EVENT_OUTPUT] + ("…" if len(result) > EVENT_OUTPUT else ""), } ) - if looping: + if looping or blocked: for rest in calls[i + 1 :]: skipped = { "role": "tool", @@ -1283,6 +1382,18 @@ def _run_turn(self, user_text: str) -> None: self.repeats.reset() looping = "" continue + if blocked: + self.turn_status = "blocked" + self.console.print( + Text( + f"Stopped: every request to {blocked.site} failed ({blocked.calls} calls: {blocked.what}) and " + "none worked, so this machine probably can't reach it: a firewall, a proxy, a regional block " + "or no internet. Check the connection (another network or a VPN, a proxy setting), or tell " + "the model another way to get what it needs.", + style="yellow", + ) + ) + return if looping: self.turn_status = "loop" self.console.print( diff --git a/src/lcode/api.py b/src/lcode/api.py index 92bd138..1e55493 100644 --- a/src/lcode/api.py +++ b/src/lcode/api.py @@ -285,7 +285,7 @@ def collect(event: dict) -> None: try: agent.run_turn(prompt) - status = agent.turn_status if agent.turn_status in ("max_steps", "loop") else "success" + status = agent.turn_status if agent.turn_status in ("max_steps", "loop", "blocked") else "success" except KeyboardInterrupt: status, error = "interrupted", "interrupted before the request was done" except ModelError as e: diff --git a/src/lcode/backends.py b/src/lcode/backends.py index f20e600..e3a4378 100644 --- a/src/lcode/backends.py +++ b/src/lcode/backends.py @@ -302,6 +302,7 @@ def chat_stream(self, payload: dict, on_open: Callable | None = None) -> Iterato calls: dict[int, dict] = {} usage: dict = {} timings: dict = {} # llama-server measures prompt and generation time itself + finished = "" # why the reply ended, as Ollama's done_reason: "length" when it was cut off splitter = ThinkSplitter() first_token = None with r: @@ -323,6 +324,7 @@ def chat_stream(self, payload: dict, on_open: Callable | None = None) -> Iterato usage = chunk.get("usage") or usage timings = chunk.get("timings") or timings for choice in chunk.get("choices") or []: + finished = choice.get("finish_reason") or finished delta = choice.get("delta") or {} thinking = delta.get("reasoning_content") or delta.get("reasoning") if thinking: @@ -356,6 +358,7 @@ def chat_stream(self, payload: dict, on_open: Callable | None = None) -> Iterato "prompt_eval_duration": first_token - started, "eval_count": int(usage.get("completion_tokens") or 0), "eval_duration": max(1, ended - first_token), + "done_reason": "length" if finished == "length" else "stop", } if timings.get("prompt_ms") is not None and timings.get("predicted_ms"): # The whole prompt counts (it's what fills the context), cached tokens included. diff --git a/src/lcode/bench.py b/src/lcode/bench.py index 8f7b4f7..7fffe58 100644 --- a/src/lcode/bench.py +++ b/src/lcode/bench.py @@ -904,7 +904,7 @@ def run_task( if agent is None: agent = Agent(ollama, settings, folder, console=log) else: # the same conversation moves on to this task's folder, as with /cd - agent.cwd = folder.resolve() + agent.cwd = agent.workspace = folder.resolve() agent.messages[0]["content"] = agent.system_prompt() if settings.embed_model != "off": # --code-search: index the task's folder first, outside the timer from lcode import codesearch diff --git a/src/lcode/checks.py b/src/lcode/checks.py index 8c0f442..54a5207 100644 --- a/src/lcode/checks.py +++ b/src/lcode/checks.py @@ -158,3 +158,74 @@ def missing(check: Check, result: str) -> bool: return True head = result.splitlines()[:5] # a missing program says so first; test failures look different return code not in (0, None) and any(MISSING.search(line) for line in head) + + +# ----------------------------------------------------------------------------- scripts written, never run + +SCRIPT_SUFFIXES = (".sh", ".bash", ".zsh") +SHELL_SHEBANG = re.compile(r"^#!\S*?(?:/|env\s+)(?:ba|z|da|k)?sh\b") +RUNNERS = frozenset({"bash", "sh", "zsh", "dash", "ksh", "source", ".", "exec"}) +PREFIXES = frozenset({"sudo", "time", "nohup", "env", "command", "nice"}) +SEGMENTS = re.compile(r"&&|\|\||[;|\n&()]") + + +def unrun_scripts(root: Path, changed: list[str], commands: list[str]) -> list[str]: + """Shell scripts written in this request that no command in it ran. + + Small models write a build or install script and call it done without running it once; the first thing + the user then sees is its first error. Only shell scripts count (a .sh file, or a shell shebang): a Python + file with a shebang is as likely an app or a server that never ends. + """ + unrun = [] + for rel in changed: + path = root / rel + if not is_shell_script(path): + continue + if not any(runs(command, path.name) for command in commands): + unrun.append(rel) + return unrun + + +def is_shell_script(path: Path) -> bool: + if not path.is_file(): + return False + if path.suffix in SCRIPT_SUFFIXES: + return True + if path.suffix: + return False + try: + with path.open("rb") as f: + first = f.readline(200).decode("utf-8", "replace") + except OSError: + return False + return bool(SHELL_SHEBANG.match(first)) + + +def runs(command: str, name: str) -> bool: + """Whether a shell command runs the script called `name` (not just chmods, cats or commits it).""" + for segment in SEGMENTS.split(command): + words = segment.split() + while words and (("=" in words[0] and not words[0].startswith("=")) or words[0] in PREFIXES): + words.pop(0) + if words and words[0] == "timeout": + words = words[2:] if len(words) > 2 else [] + if not words: + continue + program = words[0].strip("'\"") + if program in RUNNERS: + rest = [w.strip("'\"") for w in words[1:] if not w.startswith("-")] + program = rest[0] if rest else "" + if Path(program).name == name: + return True + return False + + +def unrun_note(scripts: list[str]) -> str: + names = ", ".join(scripts) + it = "it" if len(scripts) == 1 else "them" + return ( + f"[lcode] You wrote {names} in this request but never ran {it}. Run {it} now and fix what fails, so you know " + f"{it} works before you say so. If running {it} would change things outside the project (install packages, " + f"deploy, delete data) or needs something you don't have, don't: tell the user plainly that {it} is untested " + "and why." + ) diff --git a/src/lcode/repeats.py b/src/lcode/repeats.py index 4b7d748..fb95a17 100644 --- a/src/lcode/repeats.py +++ b/src/lcode/repeats.py @@ -16,6 +16,8 @@ import hashlib import json +import re +from collections import deque from dataclasses import dataclass from lcode.permissions import is_read_only @@ -102,3 +104,38 @@ def changes_things(name: str, args: dict, result: str) -> bool: if name == "bash": return not is_read_only(str(args.get("command", ""))) return True # other tools (MCP, subagents…) might + + +class Circles: + """Notice a reply that goes round in circles while it streams: the same sentences over and over. + + Small models sometimes fall into this while they think: a few paragraphs, repeated until the context is + full. It takes minutes behind the spinner and ends without an answer. Sentences shorter than MIN_SENTENCE + don't count (they repeat legitimately). The text is going round in circles when, of the last WINDOW + sentences, LOOPING or more were each seen REPEATS times or more. + """ + + MIN_SENTENCE = 20 + WINDOW = 20 + REPEATS = 4 + LOOPING = 16 + SPLIT = re.compile(r"\n|(?<=[.!?])\s+") + + def __init__(self) -> None: + self.partial = "" + self.seen: dict[str, int] = {} + self.recent: deque[str] = deque(maxlen=self.WINDOW) + + def feed(self, text: str) -> bool: + """Add streamed text; whether it's now going round in circles.""" + parts = self.SPLIT.split(self.partial + text) + self.partial = parts.pop() + for part in parts: + sentence = " ".join(part.split()).lower() + if len(sentence) < self.MIN_SENTENCE: + continue + self.seen[sentence] = self.seen.get(sentence, 0) + 1 + self.recent.append(sentence) + if len(self.recent) < self.WINDOW: + return False + return sum(self.seen[s] >= self.REPEATS for s in self.recent) >= self.LOOPING diff --git a/src/lcode/repl.py b/src/lcode/repl.py index 5d8c62e..ae19c61 100644 --- a/src/lcode/repl.py +++ b/src/lcode/repl.py @@ -1059,7 +1059,7 @@ def handle_command(agent: Agent, line: str, hardware: Hardware) -> bool: p = Path(os.path.expanduser(arg or "~")) p = (p if p.is_absolute() else agent.cwd / p).resolve() if p.is_dir(): - agent.cwd = p + agent.cwd = agent.workspace = p agent.messages[0]["content"] = agent.system_prompt() c.print(f"Working directory: {p}") else: diff --git a/src/lcode/subagents.py b/src/lcode/subagents.py index b561cf0..9716554 100644 --- a/src/lcode/subagents.py +++ b/src/lcode/subagents.py @@ -461,6 +461,8 @@ def __init__(self, parent: Agent, kind: AgentType, task: str, label: str, isolat show_thinking=False, ) child = Agent(parent.ollama, settings, cwd, console=Console(file=io.StringIO(), width=120)) + if not self.worktree: + child.workspace = parent.workspace # it may cd around the same project, not out of it child.perms = self.relay # type: ignore[assignment] child.sandbox = parent.sandbox child.jobs = parent.jobs # its background commands belong to the session @@ -487,11 +489,10 @@ def execute(self) -> None: last = child.messages[-1] if last.get("role") != "assistant" or last.get("tool_calls") or not last.get("content", "").strip(): # It ran out of steps, or went round in circles: ask for what it has so far, without tools. - why = ( - "You kept making the same call, so lcode stopped you." - if child.turn_status == "loop" - else "You've used all your steps." - ) + why = { + "loop": "You went round in circles, so lcode stopped you.", + "blocked": "A site you needed can't be reached from this machine, so lcode stopped you.", + }.get(child.turn_status, "You've used all your steps.") child.messages.append( { "role": "user", diff --git a/src/lcode/tools.py b/src/lcode/tools.py index 79afb3f..575c7a6 100644 --- a/src/lcode/tools.py +++ b/src/lcode/tools.py @@ -1283,10 +1283,18 @@ def stop() -> None: out.add(last) recent.append(last) if recorded and Path(recorded).is_dir() and Path(recorded).resolve() != self.agent.cwd: - if sandbox and not sandbox.contains(Path(recorded).resolve()): + from lcode.checkpoints import work_tree_for + + moved = Path(recorded).resolve() + if sandbox and not sandbox.contains(moved): status += "\n[That directory is outside the project; the working directory stays the same]" + elif not sandbox and not moved.is_relative_to(work_tree_for(self.agent.workspace)): + status += ( + f"\n[{moved} is outside the project, so the working directory stays {self.agent.cwd}: the next " + "command starts there. To run something elsewhere, cd in the same command]" + ) else: - self.agent.cwd = Path(recorded).resolve() + self.agent.cwd = moved status += f"\n[Working directory is now {self.agent.cwd}]" if sandbox and code != 0 and not sandbox.network and NO_NETWORK.search("".join(recent)): status += ( diff --git a/src/lcode/unreachable.py b/src/lcode/unreachable.py new file mode 100644 index 0000000..3e03d36 --- /dev/null +++ b/src/lcode/unreachable.py @@ -0,0 +1,186 @@ +"""Notice when the network, not the model, is in the way. + +A download that fails however its address is written usually means this machine can't reach the site: a +firewall, a proxy, a regional block, or no internet. Nothing the model writes fixes that, yet a small model +will try address after address for an hour (and lcode's repeat check never fires: every address differs). +So within one request lcode keeps track, per site, of the calls that failed to reach it (an HTTP error, a +name that doesn't resolve, a refused or timed-out connection…) and of the ones that worked. When `NOTE_AT` +different calls failed for a site and none worked, the result gets a note: check one address that must +exist, and if that fails too, tell the user. If `STOP_AFTER_NOTE` more fail, lcode ends the request. + +Which site a failure belongs to is read from the output: the address on the failing line, else the last +address printed before it, else the command's address when the command names only one site. A call that +names several sites and prints a bare status line counts for none of them. A site counts as reached when a +call got an answer from it: a success status, a page from web_fetch, or a call naming only that site that +printed plenty without an error. Local addresses (localhost, private networks, names without a dot) aren't +tracked: they aren't "the internet". +""" + +from __future__ import annotations + +import hashlib +import ipaddress +import json +import re +from dataclasses import dataclass +from urllib.parse import urlsplit + +NOTE_AT = 3 # different calls that failed for a site, and none worked: the note +STOP_AFTER_NOTE = 2 # more such calls after the note: the request ends +PLENTY = 400 # characters of error-free output from a call naming one site: it was reached +URL = re.compile(r"\b(?:https?|ftp)://[^\s'\"<>`|]+", re.I) +OK_STATUS = re.compile( + r"\bHTTP/\d(?:\.\d)?\s+[23]\d\d\b|awaiting response\.\.\. [23]\d\d\b" + r"|\b(?:status(?:[ _]code)?|http_code)[:= ]+[23]\d\d\b", + re.I, +) +FAILED_STATUS = re.compile( + r"\bHTTP/\d(?:\.\d)?\s+([45]\d\d)\b" # curl -I / -v, wget -S + r"|\bHTTP Error ([45]\d\d)\b" # Python's urllib + r"|\breturned HTTP ([45]\d\d)\b" # lcode's web_fetch + r"|\breturned error: ([45]\d\d)\b" # curl --fail + r"|\bERROR ([45]\d\d):" # wget + r"|\b([45]\d\d) (?:Client|Server) Error\b" # requests + r"|\b(?:status(?:[ _]code)?|http_code)[:= ]+([45]\d\d)\b" # what a model's own loop prints + r"|\b([45]\d\d):? \(?(?:Not Found|Forbidden|Unauthorized|Bad Gateway|Service Unavailable|Gateway Time-?out|" + r"Too Many Requests)\b", # npm, wget, error pages + re.I, +) +NETWORK_ERRORS = ( + ( + "the name doesn't resolve", + re.compile( + r"Could not resolve host|Temporary failure in name resolution|Name or service not known|" + r"nodename nor servname|getaddrinfo|ENOTFOUND|EAI_AGAIN|NameResolutionError|Failed to resolve", + re.I, + ), + ), + ("connection refused", re.compile(r"Connection refused|ECONNREFUSED", re.I)), + ("timed out", re.compile(r"Connection timed out|ETIMEDOUT|Operation timed out|Read timed out", re.I)), + ("connection reset", re.compile(r"Connection reset|ECONNRESET", re.I)), + ("no route to it", re.compile(r"Network is unreachable|No route to host|ENETUNREACH", re.I)), + ("a TLS/SSL error", re.compile(r"SSL certificate problem|certificate verify failed|SSLError", re.I)), + ("couldn't connect", re.compile(r"Failed to connect to|Max retries exceeded", re.I)), +) +LOCAL_SUFFIXES = (".local", ".lan", ".home", ".internal", ".localhost", ".test", ".example", ".invalid") + + +@dataclass +class Verdict: + site: str = "" + calls: int = 0 # different calls that failed for the site + what: str = "" # how they failed, e.g. "HTTP 404" + note: str = "" # to add to the tool result + stop: bool = False # end the request + + +class Sites: + def __init__(self, note_at: int = NOTE_AT) -> None: + self.note_at = note_at # 0: off + self.failed: dict[str, dict[str, str]] = {} # site -> {call key: how it failed} + self.reached: set[str] = set() + + def reset(self) -> None: + """A new request.""" + self.failed.clear() + self.reached.clear() + + def check(self, name: str, args: dict, result: str) -> Verdict: + """Count what this call reached and failed to reach; a note when a site looks out of reach.""" + if self.note_at <= 0: + return Verdict() + failed, reached = outcomes(name, args, result) + self.reached |= reached + key = hashlib.sha256(json.dumps([name, args], sort_keys=True, default=str).encode()).hexdigest() + verdict = Verdict() + for site, how in failed.items(): + if site in self.reached: + continue + calls = self.failed.setdefault(site, {}) + calls[key] = how + n = len(calls) + if n < self.note_at or n < verdict.calls: + continue + what = ", ".join(sorted(set(calls.values()))) + if n >= self.note_at + STOP_AFTER_NOTE: + note = ( + f"\n\n[lcode] {n} different calls to {site} failed ({what}) and none worked, so lcode stopped " + "the request here: this machine can't reach that site." + ) + verdict = Verdict(site, n, what, note, stop=True) + elif not verdict.stop: + verdict = Verdict(site, n, what, advice(site, n, what)) + return verdict + + +def advice(site: str, n: int, what: str) -> str: + return ( + f"\n\n[lcode] Every request to {site} in this request has failed ({n} different calls: {what}) and none " + "worked. When a site fails however the address is written, the usual cause is the network, not the " + "address: a firewall, a proxy, a regional block, or no internet on this machine. Check that once: request " + f"an address on {site} that must exist (one the site's own pages link to). If that fails too, stop trying " + "addresses and tell the user what fails and what they could do: use another network or a VPN, set a " + f"proxy (HTTPS_PROXY), use a mirror, or run this step on a machine that can reach {site}, such as CI." + ) + + +def site_of(url: str) -> str: + """The host to track for a URL, or "" for local and private addresses (and anything unparseable).""" + try: + host = (urlsplit(url).hostname or "").lower().rstrip(".") + except ValueError: + return "" + if not host or "$" in host or "{" in host or host == "localhost" or host.endswith(LOCAL_SUFFIXES): + return "" + try: + ip = ipaddress.ip_address(host.strip("[]")) + except ValueError: + return host if "." in host else "" + return "" if ip.is_private or ip.is_loopback or ip.is_link_local or ip.is_unspecified else host + + +def failure(line: str) -> str: + """How a line of output says a request failed ("HTTP 404", "the name doesn't resolve"…), or "".""" + m = FAILED_STATUS.search(line) + if m: + return f"HTTP {next(g for g in m.groups() if g)}" + for what, pattern in NETWORK_ERRORS: + if pattern.search(line): + return what + return "" + + +def outcomes(name: str, args: dict, result: str) -> tuple[dict[str, str], set[str]]: + """The sites a call failed to reach (with how) and the ones it reached.""" + if name == "web_fetch": + site = site_of(str(args.get("url", ""))) + if not site: + return {}, set() + if result.startswith("Error:"): + how = failure(result) + return ({site: how} if how else {}), set() + return {}, {site} + if name != "bash": + return {}, set() + named = {site_of(url) for url in URL.findall(str(args.get("command", "")))} + current = next(iter(named)) if len(named) == 1 else "" + failed: dict[str, str] = {} + reached: set[str] = set() + for line in result.splitlines(): + if line.startswith("[lcode") or line.startswith("[exit code"): + continue + urls = URL.findall(line) + if urls: + current = site_of(urls[-1]) + if not current: + continue + how = failure(line) + if how: + failed.setdefault(current, how) + elif OK_STATUS.search(line): + reached.add(current) + if len(named) == 1 and (site := next(iter(named))) and not failed and site not in reached: + printed = "".join(line for line in result.splitlines() if not line.startswith(("[lcode", "[exit code"))) + if result.rstrip().endswith("[exit code: 0]") and len(printed) >= PLENTY: + reached.add(site) + return failed, reached diff --git a/tests/test_backends.py b/tests/test_backends.py index 82a87a5..023c824 100644 --- a/tests/test_backends.py +++ b/tests/test_backends.py @@ -338,3 +338,12 @@ def test_lcode_runs_against_a_configured_server(server, repo, tmp_path, monkeypa with pytest.raises(SystemExit, match="1"): cli.main(["setup"]) assert "lcode setup downloads models into Ollama, but lcode uses LM Studio" in out.file.getvalue() + + +def test_a_reply_cut_off_by_the_server_says_so(server): + cut = {"choices": [{"index": 0, "delta": {"content": "The build"}, "finish_reason": "length"}]} + FakeServer.replies = [sse(cut, USAGE)] + chunks = list(OpenAICompatible(server).chat_stream({"model": "m", "messages": []})) + assert chunks[-1]["done_reason"] == "length" + FakeServer.replies = [sse(delta(content="Done."), USAGE)] + assert list(OpenAICompatible(server).chat_stream({"model": "m", "messages": []}))[-1]["done_reason"] == "stop" diff --git a/tests/test_sessions.py b/tests/test_sessions.py index 3356d36..70d2857 100644 --- a/tests/test_sessions.py +++ b/tests/test_sessions.py @@ -49,7 +49,7 @@ def test_list_orders_by_last_use_and_filters_by_folder(make_agent, tmp_path_fact run(second, "second task") touch(second, 100) elsewhere = make_agent() - elsewhere.cwd = tmp_path_factory.mktemp("elsewhere").resolve() + elsewhere.cwd = elsewhere.workspace = tmp_path_factory.mktemp("elsewhere").resolve() run(elsewhere, "other folder") assert [s.title for s in sessions.list_sessions(first.cwd)] == ["second task", "first task"] assert len(sessions.list_sessions()) == 3 @@ -115,7 +115,7 @@ def test_resumed_session_needs_fresh_reads(make_agent): def test_resume_switches_to_the_session_folder(make_agent, tmp_path_factory): other = tmp_path_factory.mktemp("other").resolve() agent = make_agent() - agent.cwd = other + agent.cwd = agent.workspace = other run(agent, "work in another folder") here = make_agent() assert choose_session(here, "all work in another") is not None diff --git a/tests/test_stuck.py b/tests/test_stuck.py new file mode 100644 index 0000000..54a1bb5 --- /dev/null +++ b/tests/test_stuck.py @@ -0,0 +1,250 @@ +"""When a request can't make progress: an unreachable site, a reply that goes round in circles or says +nothing, the shell leaving the project, a script written and never run. Each case comes from a real session +that spent an hour on an Android build whose downloads this machine couldn't reach.""" + +import os +import stat + +from conftest import call, output, reply +from lcode import checks +from lcode.repeats import Circles +from lcode.unreachable import Sites, failure, outcomes, site_of + +BUILD_LOG = """# Run 'git config --get remote.origin.url' ... +https://github.com/kivy/python-for-android.git +# Recommended android's NDK version by p4a is: 28c +# Android NDK is missing, downloading +# Downloading https://dl.google.com/android/repository/android-ndk-r26b-linux.zip +Traceback (most recent call last): +urllib.error.HTTPError: HTTP Error 404: Not Found + +[exit code: 1]""" + + +def bash(command): + return {"command": command} + + +# ----------------------------------------------------------------------------- unreachable sites + + +def test_a_failure_belongs_to_the_address_printed_before_it(): + assert outcomes("bash", bash("./build_android.sh"), BUILD_LOG) == ({"dl.google.com": "HTTP 404"}, set()) + + +def test_a_command_naming_one_site_owns_bare_status_lines(): + head = bash('curl -sI "https://dl.google.com/android/repository/android-ndk-r30-linux.zip" | head -3') + assert outcomes("bash", head, "HTTP/2 404 \ncontent-length: 1449\n[exit code: 0]") == ( + {"dl.google.com": "HTTP 404"}, + set(), + ) + assert outcomes("bash", head, "HTTP/2 200 \n[exit code: 0]") == ({}, {"dl.google.com"}) + several = bash("curl -sI https://dl.google.com/a.zip; curl -sI https://github.com/b") + assert outcomes("bash", several, "HTTP/2 404\n---\nHTTP/2 200\n[exit code: 0]") == ({}, set()) + + +def test_a_page_from_one_site_counts_as_reached(): + page = bash("curl -s https://raw.githubusercontent.com/kivy/python-for-android/master/README.md") + assert outcomes("bash", page, "x = 1\n" * 100 + "[exit code: 0]") == ({}, {"raw.githubusercontent.com"}) + assert outcomes("bash", page, "short\n[exit code: 0]") == ({}, set()) + + +def test_network_errors_and_web_fetch(): + assert failure("curl: (6) Could not resolve host: dl.google.com") == "the name doesn't resolve" + assert failure("curl: (7) Failed to connect to x port 443: Connection refused") == "connection refused" + assert failure("Error 404 (Not Found)!!1") == "HTTP 404" and failure("all 404 tests passed") == "" + assert failure(" Status: 404") == failure("http_code=404") == failure("HTTP status code 404") == "HTTP 404" + loop = bash('for u in https://dl.google.com/a.zip https://dl.google.com/b.zip; do curl -o /dev/null "$u"; done') + assert outcomes("bash", loop, "Trying: https://dl.google.com/a.zip\n Status: 404\n") == ( + {"dl.google.com": "HTTP 404"}, + set(), + ) + assert outcomes("bash", loop, " Status: 200\n") == ({}, {"dl.google.com"}) + url = {"url": "https://developer.android.com/ndk/downloads"} + assert outcomes("web_fetch", url, f"Error: {url['url']} returned HTTP 403") == ( + {"developer.android.com": "HTTP 403"}, + set(), + ) + assert outcomes("web_fetch", url, "NDK Downloads\n...") == ({}, {"developer.android.com"}) + + +def test_local_addresses_are_not_tracked(): + for url in ( + "http://localhost:8000/x", + "http://127.0.0.1/x", + "rtsp://10.0.0.5:554/cam", + "http://192.168.1.2/", + "http://nas/x", + "http://printer.local/", + "https://${HOST}/x", + ): + assert site_of(url) == "", url + assert site_of("https://DL.Google.com./x") == "dl.google.com" + + +def test_a_site_that_never_answers_gets_a_note_then_stops(): + sites = Sites() + verdicts = [ + sites.check("bash", bash(f'curl -sI "https://dl.google.com/android-ndk-{v}-linux.zip"'), "HTTP/2 404\n") + for v in ("r26b", "r27", "r28c", "r29", "r30") + ] + assert [("stop" if v.stop else "note") if v.note else "" for v in verdicts] == ["", "", "note", "note", "stop"] + assert "Every request to dl.google.com" in verdicts[2].note and "(3 different calls: HTTP 404)" in verdicts[2].note + assert "HTTPS_PROXY" in verdicts[2].note and "such as CI" in verdicts[2].note + + +def test_a_site_that_answered_once_never_counts_as_out_of_reach(): + sites = Sites() + sites.check("bash", bash("curl -sI https://dl.google.com/ok.zip"), "HTTP/2 200\n") + for v in range(5): + assert not sites.check("bash", bash(f"curl -sI https://dl.google.com/{v}.zip"), "HTTP/2 404\n").note + + +def test_the_same_failing_call_counts_once_and_a_new_request_starts_over(): + sites = Sites() + same = ("bash", bash("curl -sI https://dl.google.com/a.zip"), "HTTP/2 404\n") + assert not any(sites.check(*same).note for _ in range(5)) + sites.check("bash", bash("curl -sI https://dl.google.com/b.zip"), "HTTP/2 404\n") + sites.reset() + assert not sites.check("bash", bash("curl -sI https://dl.google.com/c.zip"), "HTTP/2 404\n").note + + +def test_the_agent_stops_a_request_whose_site_cannot_be_reached(make_agent): + tries = [ + reply(tool_calls=[call("bash", command=f"echo https://dl.example.com/ndk-{v}.zip; echo 'HTTP Error 404'")]) + for v in range(5) + ] + agent = make_agent([*tries, reply("never asked")]) + agent.run_turn("Download the NDK") + results = [m["content"] for m in agent.messages if m["role"] == "tool"] + assert len(results) == 5 and "Every request to dl.example.com" in results[2] + assert "so lcode stopped the request here" in results[4] and agent.turn_status == "blocked" + text = " ".join(output(agent).split()) + assert "Stopped: every request to dl.example.com failed (5 calls: HTTP 404)" in text + assert agent.ollama.scripts == [reply("never asked")] + + +# ----------------------------------------------------------------------------- replies that go nowhere + +LOOP = ( + "Let me try to check the actual download page for the correct URL.\n\n" + "Let me check if there's a way to get the NDK from the Android SDK Manager.\n\n" + "Actually, let me try a completely different approach with the android.ndk_path option.\n\n" +) + + +def test_circles_are_noticed_and_varied_thinking_is_not(): + circles = Circles() + assert not circles.feed("I need to read the build script first, then the spec.\n") + assert any(circles.feed(LOOP) for _ in range(12)) + varied = Circles() + text = "".join(f"Step {n}: read file number {n} and compare it with the spec.\n" for n in range(200)) + assert not any(varied.feed(text[i : i + 9]) for i in range(0, len(text), 9)) + short = Circles() + assert not any(short.feed("Wait.\nHmm.\nOK.\n") for _ in range(100)) + + +def streamed(chunks, closed): + def stream(): + try: + yield from chunks + finally: + closed.append(True) + + return stream() + + +def test_a_reply_going_round_in_circles_is_stopped_and_the_model_asked_again(make_agent): + looping = [{"message": {"thinking": LOOP}, "done": False} for _ in range(200)] + closed = [] + agent = make_agent([None, reply("The NDK downloads fail from this network: use a VPN.")]) + agent.ollama.scripts[0] = streamed(looping, closed) + agent.run_turn("Build the APK") + assert closed == [True] and agent.turn_status == "success" + assert not any(m.get("thinking") for m in agent.messages) # the circles don't stay in the context + assert "went round in circles" in agent.messages[-2]["content"] + assert agent.messages[-1]["content"].startswith("The NDK downloads fail") + assert "the reply went round in circles, so lcode stopped it" in output(agent) + + +def test_an_empty_or_cut_off_reply_gets_one_more_chance_then_the_request_stops(make_agent): + cut = [{"message": {"thinking": "Hmm, let me think about the URL."}, "done": False}] + cut.append({"message": {}, "done": True, "done_reason": "length"}) + agent = make_agent([cut, reply("Done: the build works.")]) + agent.run_turn("Build it") + assert "filled the rest of the context" in agent.messages[-2]["content"] and agent.turn_status == "success" + agent = make_agent([reply(""), reply(" ")]) + agent.run_turn("Build it") + assert agent.turn_status == "loop" + assert "ended without an answer, twice" in " ".join(output(agent).split()) + + +def test_an_answer_cut_off_by_the_context_says_so(make_agent): + cut = [{"message": {"content": "The build fails because"}, "done": False}] + cut.append({"message": {}, "done": True, "done_reason": "length"}) + agent = make_agent([cut]) + agent.run_turn("Why does it fail?") + assert "The answer was cut off: the context window is full" in " ".join(output(agent).split()) + + +# ----------------------------------------------------------------------------- the shell leaving the project + + +def test_the_shell_can_move_around_the_project_but_not_out_of_it(agent, repo, tmp_path_factory): + elsewhere = tmp_path_factory.mktemp("elsewhere") + out = agent.tools.run("bash", {"command": f"cd {elsewhere} && pwd"}) + assert agent.cwd == repo and f"the working directory stays {repo}" in out + assert agent.tools.run("bash", {"command": "pwd"}).startswith(str(repo)) + agent.tools.run("bash", {"command": "cd src && pwd"}) + assert agent.cwd == repo / "src" + agent.messages.append({"role": "user", "content": "hi"}) + agent.save() + from lcode import sessions + + assert [s.id for s in sessions.list_sessions(repo)] == [agent.session_id] + + +# ----------------------------------------------------------------------------- scripts written, never run + + +def test_runs_tells_running_a_script_from_touching_it(): + for command in ( + "./build.sh", + "cd x && bash -n build.sh", + "FOO=1 sudo scripts/build.sh release", + "timeout 60 sh build.sh", + ): + assert checks.runs(command, "build.sh"), command + for command in ("chmod +x build.sh", "cat build.sh | head", "git add build.sh", "ls -la build.sh; echo ok"): + assert not checks.runs(command, "build.sh"), command + + +def test_only_shell_scripts_count(repo): + (repo / "build.sh").write_text("echo hi\n") + (repo / "deploy").write_text("#!/usr/bin/env bash\necho hi\n") + (repo / "main.py").write_text("#!/usr/bin/env python3\nprint(1)\n") + changed = ["build.sh", "deploy", "main.py", "gone.sh"] + assert checks.unrun_scripts(repo, changed, ["chmod +x build.sh deploy"]) == ["build.sh", "deploy"] + assert checks.unrun_scripts(repo, changed, ["./build.sh", "bash deploy"]) == [] + + +def test_a_script_written_and_never_run_is_sent_back_once(make_agent, repo): + write = call("write_file", path="build_android.sh", content="#!/usr/bin/env bash\necho built\n") + run = call("bash", command="chmod +x build_android.sh && ./build_android.sh") + agent = make_agent( + [ + reply(tool_calls=[write, call("bash", command="chmod +x build_android.sh")]), + reply("All done: run ./build_android.sh."), + reply(tool_calls=[run]), + reply("It runs: it printed 'built'."), + ], + checks="auto", + ) + agent.run_turn("Write a build script") + notes = [m["content"] for m in agent.messages if m["role"] == "user" and "never ran it" in m["content"]] + assert len(notes) == 1 and "build_android.sh" in notes[0] + assert agent.messages[-1]["content"] == "It runs: it printed 'built'." and not agent.ollama.scripts + assert os.stat(repo / "build_android.sh").st_mode & stat.S_IXUSR + agent = make_agent([reply(tool_calls=[write]), reply("Done.")], checks="off") + agent.run_turn("Write a build script") + assert agent.messages[-1]["content"] == "Done." diff --git a/tests/test_subagents.py b/tests/test_subagents.py index b18577a..0d3e2b0 100644 --- a/tests/test_subagents.py +++ b/tests/test_subagents.py @@ -395,6 +395,6 @@ def test_a_subagent_that_repeats_itself_still_gives_a_report(make_agent): ) agent.run_turn("go") report = agent.ollama.payloads[6] - assert "You kept making the same call, so lcode stopped you." in report["messages"][-2]["content"] + assert "You went round in circles, so lcode stopped you." in report["messages"][-2]["content"] assert "tools" not in report assert "I kept listing the top level" in agent.messages[3]["content"] and agent.turn_status == "success"