Repository navigation
Verification: test what the model can't see before calling it done #93
Description
Problem
In a GUI task, the model marked "Test the fix by running the app" as done after only py_compile and an import test. It never ran the app. It changed the window's layout several times without seeing it, and called the work finished each time, while the actual bug went unfound.
The bug could be reproduced in about a minute headless (QT_QPA_PLATFORM=offscreen, fake frames, measuring the window and taking screenshots), as it was later.
Proposal
Teach the model, in its instructions, how to verify work it can't see:
- What counts as tested: a todo or claim of testing is only true for what was run. Say what was checked and what wasn't.
- GUI, web and visual changes: reproduce headlessly where possible: Qt's
offscreenplatform, a headless browser, rendering to an image and looking at it withview_image. Otherwise ask the user to check, and say what to look for. - Bug fixes: reproduce the bug first, then show it's gone.
Keep it short, since the instructions go with every request.
Acceptance criteria
- Instructions for verifying, including visual changes
- A benchmark task that rewards reproducing before fixing, run before and after
- Docs
Activity
Tell the model what counts as checking its work (#100)
nasser1941 commentedon Oct 10, 2026
Shipped in 0.25.0 (#100), with
lcode bench --tasks runtime-bug. The measurements are in #100. In short, for qwen3.6-35b:The PyPI build runs as expected. In a manual
--no-thinkrun, though, the model still edited before reproducing, so following the guidance depends on the model. The task stays as a harder benchmark for stronger models.