diff --git a/CHANGELOG.md b/CHANGELOG.md index fe7b0a1..dc715fb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,20 @@ All notable changes to lcode are documented here. The format follows ## [Unreleased] +### Changed + +- The model is told what counts as checking its work: run what it changed the way it will be used, + reproduce each reported problem before fixing it, check what it can't see (GUIs, web pages, + layouts) without a screen, for example with Qt's `offscreen` platform, a headless browser or a + rendered image, and never call something tested or done without having run it. See + [Checking its own work](https://nasser1941.github.io/lcode/how-it-works/#checking-its-own-work). + +### Added + +- `lcode bench --tasks runtime-bug`: a bug that shows only when the code runs, a camera wall whose + window grows with every video frame. Reading the code suggests band-aids that fail its hidden + test. It isn't in the default set. + ## [0.24.0] - 2026-10-10 ### Added diff --git a/docs/bench.md b/docs/bench.md index d6fb513..fac0173 100644 --- a/docs/bench.md +++ b/docs/bench.md @@ -38,8 +38,10 @@ A model usually takes 4 to 12 minutes; each task stops after 5 minutes (`--timeo ### More tasks and features `--tasks find-concept` adds a task in a larger repository: 40 files of a small backend where the -model has to find a function by what it does. It isn't in the default set, so results stay -comparable with earlier versions. `--repo-map` gives the model the [repository map](search.md) and +model has to find a function by what it does. `--tasks runtime-bug` adds a bug that shows only +when the code runs: a camera wall's window grows taller with every video frame and can't be made +smaller again. Reading the code suggests band-aids that fail the hidden test; running it shows the +cause. Neither is in the default set, so results stay comparable with earlier versions. `--repo-map` gives the model the [repository map](search.md) and `--code-search` indexes each task's folder for [semantic search](search.md#semantic-code-search); both are off in `lcode bench` by default, for the same reason. diff --git a/docs/how-it-works.md b/docs/how-it-works.md index 75d4e17..f51297c 100644 --- a/docs/how-it-works.md +++ b/docs/how-it-works.md @@ -29,6 +29,19 @@ All tools return text, and failures come back to the model as `Error: …` messa crashing the session, so the model can correct itself (for example re-read a file when an edit didn't match). +### Checking its own work + +Small models like to call work done after a syntax check. The system prompt tells the model what +counts as checked: + +- run what it changed the way it will be used; +- reproduce a bug before fixing it, then show it's gone; +- check what it can't see (a GUI, a web page, a layout) without a screen: Qt's `offscreen` + platform, a headless browser, or a script that prints sizes or renders an image it then looks + at; +- never mark a task done or call something tested without having run it, and tell you what to + look at when it couldn't check. + ### When the model repeats itself Small models sometimes fall into a loop: they make a call, get a result, and make the very same call diff --git a/pyproject.toml b/pyproject.toml index 07cc448..1bdaaee 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -82,6 +82,7 @@ ignore = [ "src/lcode/context.py" = ["E501"] "src/lcode/lsp.py" = ["E501"] "src/lcode/repomap.py" = ["E501"] +"src/lcode/bench_runtime.py" = ["E501"] # the task's files and hidden test "src/lcode/codesearch.py" = ["E501"] "src/lcode/bench_backend.py" = ["E501"] "src/lcode/backends.py" = ["E501"] diff --git a/src/lcode/agent.py b/src/lcode/agent.py index 7f9c4de..9a84892 100644 --- a/src/lcode/agent.py +++ b/src/lcode/agent.py @@ -81,14 +81,18 @@ - Use your tools to gather facts. Never guess file contents, APIs, or command output — read, search, or run them. - To understand a repository: look at the layout, README/docs, entry points and config, then grep and read the relevant files. Cite code as `path:line`. - Read a file before editing it. Use edit_file for targeted changes and write_file for new files or full rewrites. old_string must be copied exactly from the file (without the line-number prefixes). -- After writing or changing code, verify it: run the script, the tests, or at least a syntax check (e.g. `{python} -m py_compile file.py`). Fix what fails. - Match the existing code style, naming and structure. Don't add unrequested features. - For multi-step tasks, plan with todo_write and keep it updated. - Work autonomously until the task is done. Only stop to ask the user when you are genuinely blocked or the decision is theirs. - Don't run destructive or irreversible commands (rm -rf, git reset --hard, git push --force, dropping data) unless the user explicitly asked. -- Never claim something works without having run it. If something fails, say so and show the relevant error. - If the user denies a tool call, don't retry the same thing; adjust or ask. +# Verifying your work +- A syntax check or an import only shows that code loads. Run what you changed the way it will be used (the script, the tests, the app) and fix what fails. +- To fix a bug, first reproduce each problem the user reports with a test, a command or a short script, then run the same checks to show they're gone. +- For what you can't see, like a GUI, a web page or a layout, check it without a screen where you can: `QT_QPA_PLATFORM=offscreen` for Qt, a headless browser, or a script that prints sizes and positions or renders to an image{look}. Measure instead of assuming. +- Never claim something works, mark a todo done, or call it tested unless you ran it. If you couldn't check something, say so and tell the user exactly what to look for. If something fails, say so and show the relevant error. + # Communication - Be concise and direct; no filler. Use GitHub-flavored markdown. - When you finish, give a short summary of what you found or changed and anything the user needs to do. @@ -317,6 +321,7 @@ def system_prompt(self) -> str: skills=extensions.skills_prompt(self.extensions().skills, self.settings.context), code=self.code_prompt(), map=self.map_prompt(), + look=" you look at with view_image" if self.vision_model() else "", ) def repo_map(self) -> repomap.RepoMap | None: diff --git a/src/lcode/bench.py b/src/lcode/bench.py index 19442b8..038b990 100644 --- a/src/lcode/bench.py +++ b/src/lcode/bench.py @@ -746,7 +746,7 @@ def check_feature(folder: Path, answer: str) -> Check: def extra_tasks() -> list[Task]: """Tasks that only run when asked for with --tasks (they're slower, or test optional features).""" - from lcode import bench_backend + from lcode import bench_backend, bench_runtime return [ Task( @@ -755,7 +755,14 @@ def extra_tasks() -> list[Task]: bench_backend.PROMPT, bench_backend.FILES, bench_backend.check, - ) + ), + Task( + "runtime-bug", + "Fix a bug that shows only when it runs", + bench_runtime.PROMPT, + bench_runtime.FILES, + bench_runtime.check, + ), ] diff --git a/src/lcode/bench_runtime.py b/src/lcode/bench_runtime.py new file mode 100644 index 0000000..2c8f227 --- /dev/null +++ b/src/lcode/bench_runtime.py @@ -0,0 +1,156 @@ +"""A bug that only shows when the code runs, for `lcode bench --tasks runtime-bug`. + +A camera wall lays out video panels in a window. While video plays, the window grows taller frame after +frame, past the screen, and can't be made smaller again: each panel's minimum size is taken from the +frame it shows, and the frame is scaled to the whole cell, status bar included. Reading the code, the +obvious fixes are band-aids (ignore the minimum sizes, or only fix the scaling) that fail part of the +hidden test. Running it, the window sizes show the ratchet. It's the kind of bug a GUI has, without +needing one, and rewards reproducing a problem before fixing it. +""" + +from __future__ import annotations + +import re +from pathlib import Path + +WALL = '''\ +"""A camera wall: a window with a grid of panels, each showing its camera's latest video frame.""" + +SPACING = 8 # pixels between and around the panels +STATUS_BAR = 20 # under its video, each panel shows the camera's name and status + + +class Panel: + def __init__(self, name): + self.name = name + self.min_width = self.min_height = 1 # the panel can't get smaller than this + self.video = (0, 0) # the size of the frame on screen + + def show(self, frame_width, frame_height, cell_width, cell_height): + """Show a frame scaled to fit the panel's cell, keeping its aspect ratio.""" + scale = min(cell_width / frame_width, cell_height / frame_height) + self.video = (int(frame_width * scale), int(frame_height * scale)) + # The video and the status bar under it must stay visible. + self.min_width, self.min_height = self.video[0], self.video[1] + STATUS_BAR + + +class Wall: + def __init__(self, names, rows, cols): + self.rows, self.cols = rows, cols + self.panels = [Panel(name) for name in names] + self.width = self.height = 0 + + def resize(self, width, height): + """Ask for a window size. The window can't be smaller than its panels need.""" + need_width = max(p.min_width for p in self.panels) * self.cols + SPACING * (self.cols + 1) + need_height = max(p.min_height for p in self.panels) * self.rows + SPACING * (self.rows + 1) + self.width, self.height = max(width, need_width), max(height, need_height) + + def cell(self): + """The size of one panel's cell.""" + return ( + (self.width - SPACING * (self.cols + 1)) // self.cols, + (self.height - SPACING * (self.rows + 1)) // self.rows, + ) + + def new_frame(self, frame_width, frame_height): + """Every camera delivered a frame: show it in each panel, then let the layout settle.""" + cell_width, cell_height = self.cell() + for panel in self.panels: + panel.show(frame_width, frame_height, cell_width, cell_height) + self.resize(self.width, self.height) +''' + +SIMULATE = '''\ +"""Play the camera wall without a screen: python simulate.py [WIDTH HEIGHT [ROWSxCOLS]]""" + +import sys + +from wall import Wall + + +def main(): + width, height = (int(sys.argv[1]), int(sys.argv[2])) if len(sys.argv) > 2 else (2560, 1080) + rows, cols = map(int, sys.argv[3].split("x")) if len(sys.argv) > 3 else (3, 3) + wall = Wall([f"camera {n + 1}" for n in range(rows * cols)], rows, cols) + wall.resize(width * 9 // 10, height * 9 // 10) + print(f"{rows}x{cols} wall on a {width}x{height} screen, window {wall.width}x{wall.height}") + for n in range(1, 6): + wall.new_frame(1920, 1080) + print(f"after frame {n}: window {wall.width}x{wall.height}, video {wall.panels[0].video}") + + +if __name__ == "__main__": + main() +''' + +FILES = {"wall.py": WALL, "simulate.py": SIMULATE} + +PROMPT = ( + "Users say the camera wall in wall.py keeps growing taller while video plays, until it's bigger than " + "their screen, and after that it can't be made smaller again. Fix it." +) + +HIDDEN_TEST = """\ +import unittest + +from wall import SPACING, STATUS_BAR, Wall + +SCREENS = [(2560, 1080, 3, 3), (1920, 1080, 2, 3), (1366, 768, 3, 3), (3440, 1440, 2, 4), (1280, 1024, 2, 2)] + + +def play(width, height, rows, cols, frames=30, frame=(1920, 1080)): + wall = Wall([f"camera {n}" for n in range(rows * cols)], rows, cols) + wall.resize(width, height) + for _ in range(frames): + wall.new_frame(*frame) + return wall + + +class HiddenTest(unittest.TestCase): + def test_keeps_its_size(self): + for width, height, rows, cols in SCREENS: + with self.subTest(screen=(width, height), grid=(rows, cols)): + wall = play(width * 9 // 10, height * 9 // 10, rows, cols) + self.assertEqual((wall.width, wall.height), (width * 9 // 10, height * 9 // 10)) + + def test_can_be_made_smaller(self): + wall = play(2304, 972, 3, 3) + wall.resize(1280, 720) + wall.new_frame(1920, 1080) + self.assertEqual((wall.width, wall.height), (1280, 720)) + + def test_video_fits_and_fills_its_cell(self): + for width, height, rows, cols in SCREENS: + for frame in [(1920, 1080), (1280, 960), (720, 1280)]: + with self.subTest(screen=(width, height), grid=(rows, cols), frame=frame): + wall = play(width * 9 // 10, height * 9 // 10, rows, cols, frames=3, frame=frame) + cell_width = (wall.width - SPACING * (cols + 1)) // cols + cell_height = (wall.height - SPACING * (rows + 1)) // rows + for panel in wall.panels: + video_width, video_height = panel.video + self.assertLessEqual(video_width, cell_width) + self.assertLessEqual(video_height + STATUS_BAR, cell_height) + fills = video_width >= cell_width - 2 or video_height + STATUS_BAR >= cell_height - 2 + self.assertTrue(fills, "the video doesn't use the room it has") + self.assertAlmostEqual(video_width / video_height, frame[0] / frame[1], delta=0.02) +""" + +PROBLEMS = { + "test_keeps_its_size": "the window still grows", + "test_can_be_made_smaller": "the window can't be made smaller", + "test_video_fits_and_fills_its_cell": "the video doesn't fit its cell above the status bar (or doesn't fill it)", +} + + +def check(folder: Path, answer: str): + from lcode.bench import Check, last_line, run_python + + if not (folder / "wall.py").is_file(): + return Check(False, "wall.py is missing") + (folder / "test_hidden_wall.py").write_text(HIDDEN_TEST) + r = run_python(folder, "-m", "unittest", "-q", "test_hidden_wall") + if r.returncode == 0: + return Check(True, "keeps its size, can be made smaller, and the video fits") + failed = [text for name, text in PROBLEMS.items() if re.search(rf"\b{name}\b", r.stderr)] + return Check(False, "; ".join(failed) or last_line(r.stderr) or last_line(r.stdout)) diff --git a/tests/test_bench.py b/tests/test_bench.py index fac9757..3a30527 100644 --- a/tests/test_bench.py +++ b/tests/test_bench.py @@ -310,3 +310,28 @@ def test_rounds_get_one_row_per_task_with_a_mark_per_run(monkeypatch, tmp_path): rows = dict(bench.summary_rows([run], tasks)) assert rows["find-code: Answer with file:line"][0].startswith("✓✗ ") assert rows["Passed"] == ["1/2"] + + +def test_runtime_bug_needs_the_real_cause(tmp_path): + from lcode import bench_runtime + + task = bench.select_tasks("runtime-bug")[0] + scaled = bench_runtime.WALL.replace( + "min(cell_width / frame_width, cell_height / frame_height)", + "min(cell_width / frame_width, (cell_height - STATUS_BAR) / frame_height)", + ) + fixed = scaled.replace( + "self.min_width, self.min_height = self.video[0], self.video[1] + STATUS_BAR", + "self.min_width, self.min_height = 1, STATUS_BAR + 1", + ) + for wall, passed, detail in [ + (bench_runtime.WALL, False, "the window still grows"), + (scaled, False, "the window can't be made smaller"), + (fixed, True, "keeps its size"), + ]: + folder = tmp_path / str(len(detail)) + folder.mkdir() + for name, content in {**task.files, "wall.py": wall}.items(): + (folder / name).write_text(content) + result = task.check(folder, "") + assert result.passed is passed and detail in result.detail diff --git a/tests/test_vision.py b/tests/test_vision.py index 0a4487d..6b58d89 100644 --- a/tests/test_vision.py +++ b/tests/test_vision.py @@ -133,3 +133,12 @@ def test_image_names_without_a_folder_may_be_new_files(make_agent): agent = make_agent() text = "rename logo.png to icon.png and see https://example.com/banner.png" assert agent.expand_mentions(text) == text + + +def test_the_model_is_told_how_to_check_what_it_cant_see(make_agent): + blind = make_agent() + prompt = blind.system_prompt() + assert "# Verifying your work" in prompt and "QT_QPA_PLATFORM=offscreen" in prompt + assert "first reproduce each problem" in prompt and "view_image" not in prompt # it has no such tool + agent = seeing(make_agent(), "qwen3.6:35b-a3b-coding") + assert "renders to an image you look at with view_image" in agent.system_prompt()