Skip to content

Look at a window before calling it done: screenshot actions and Qt warnings, unseen-window check, steadier repeats, project-Python checks - #146

Merged
nasser1941 merged 7 commits into
mainfrom
feat/look-at-windows
Oct 21, 2026
Merged

nasser1941 merged 7 commits into
mainfrom
feat/look-at-windows

Conversation

@nasser1941

@nasser1941 nasser1941 commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

Why

A real session asked lcode (qwen3.6-35b) for a double-click feature in a PyQt6 camera wall: double-click a camera to fill the window, double-click again for the grid. The model "fixed" it four times, while the user saw a blank window each time and finally a grid of blank cameras.

How the model checked its work: an import and the project's 75 tests, which cover the config and the camera reader, never the window. It never ran the program or took a screenshot. It also ran the same tests four times in a row.

The cause was printed by Qt the moment the double-click happened: QWidget::setLayout: Attempting to set QLayout "" on FocusedCamera "", which already has a layout. The full-screen widget created a layout and then tried to install a second one, so it stayed empty.

Why lcode didn't catch it:

  • The screenshot tool could only capture a program as it starts, and this bug shows only after a double-click.
  • Nothing asked the model to look at a window it had changed.
  • The repeat check saw 75 passed in 0.14s, then 0.12s, as different results.

What changes

  • Screenshot actions (Python Qt programs):
    • actions run in the Qt hook before the picture: click / double_click / right_click at a point (300,200, window coordinates) or on a widget by its text (double_click Camera 1), key Escape / key ctrl+s, type …, wait 2.
    • Steps go through QtTest, with a 0.4 s pause after each. The report lists what each step hit, or why it couldn't, before the window measurements.
    • Typing goes where the keyboard focus is, as for a user.
    • Web pages and Xvfb programs refuse actions with a clear message.
  • Windows it never looked at (checks.unseen_windows): when a request changed a .py file that imports PyQt5/6, PySide2/6, tkinter, Kivy, wx or gi.repository, and the model used neither screenshot nor view_image, it's sent back once. The note says to look at the window, with actions if the change shows after a click, or to say plainly that it's unchecked. It runs where the checks run (main agent, not plan mode) and is off with checks = "off".
  • Steady repeats: timings (0.14s, 250 ms, 2 minutes) and clock times are masked before results are compared, so counts and other output still have to match.

Added after the first real-model runs (below):

  • Qt's warnings come first in a screenshot's report. They're taken from everything the program printed, without repeats or offscreen noise. In run 3 the cause was in the output, among 30 log lines, and the model looked past it.
  • A dropped circling reply is saved (thinking and text), and the message gives the path. Run 1 ended after two dropped replies that nobody could inspect.
  • The note after a stalled reply adds: if you were about to check something by running it, run it now; for a window, take a screenshot. In run 2, the dropped thinking said "let me create a test config and run the app" 10 times.
  • The checks use the project's own Python (.venv/bin/python, venv, env). Tests that import pytest, or a requirements file listing it, get pytest without a config. In run 4, python -m unittest discover from PATH failed on the project's imports, and the model chased that.

Real-model runs (qwen3.6-35b, 64K, -p, isolated copy of the camera wall in its broken state)

The project ran in its own fresh venv, with cameras at TEST-NET addresses (192.0.2.x), and got the user's own words: "I double click a camera: it fills the whole window but I see only a white screen…". The user's original session made 4 "fixes", never ran the app, and never found the cause.

Run lcode What happened
1 actions + unseen-window check + steady repeats Two circling replies before any change; stopped (loop)
2 + saved dropped replies One wrong edit, then a loop. The saved replies were real loops: one sentence said 10 times, the same doubts 4–5 times each
3 + stall note points at running it The unseen-window check fired. The model took screenshots with actions: ["double_click 194, 160"] and saw FocusedCamera holding only the ✕ button. Its saved thinking noticed the setLayout warning, then drifted and looped
4 + Qt warnings first The unseen-window check fired, then 6 screenshots. The model removed the extra layout, the real cause, so the focused view shows the camera (verified with fake cameras). Going back to the grid was still broken in its version, and the run ended on the wrong check command, fixed afterwards in this PR

So, with this branch the model finds and fixes the cause of the white screen, which it never did before. The small model doesn't finish the whole feature in one request: the return to the grid needs another round. The user's camera wall has a separate, simpler fix (hide the other cameras), verified with screenshots and 3 window tests.

Checks

  • Unit tests: tests/test_look.py (17 tests): action parsing and its error messages, where actions are refused, the report lines, the tool passing actions through and rejecting bad ones, the nudge (once; not after a screenshot; not for non-GUI files; not with checks off), and timing-only repeats.
  • A real PyQt6 run in the same file, enabled with LCODE_TEST_QT_PYTHON (skipped in CI, which has no Qt): it double-clicks a label by its text, types into the focused field, clicks a button twice, and gets an error for a missing widget. Run locally and passed.
  • The user's actual camera wall, with fake cameras, through screenshots.take with actions=["double_click Camera 1"]:
    • The broken version: the report shows the step hitting "Camera 1", and "What the program printed" contains the setLayout warning, which is the cause.
    • The fixed version: the camera fills the window and nothing is printed.
  • Unit tests for each addition (tests/test_look.py, tests/test_stuck.py, tests/test_checks.py).
  • Full suite and strict docs build pass.

🤖 Generated with Claude Code

nasser1941 and others added 6 commits October 21, 2026 11:05
…indow check, steady repeats

From a real session that 'fixed' a double-click feature in a PyQt6 camera wall four
times, checking only imports and unrelated tests, while the user saw a blank window
each time. The cause was printed by Qt (setLayout on a widget that already had a
layout), and only showed after a double-click.

- screenshot actions (Python Qt programs): click / double_click / right_click a point
  or a widget by its text, key (with modifiers), type, wait, run by the Qt hook before
  the picture; the result says what each step hit.
- A request that changed a Python file importing a GUI toolkit, with no screenshot or
  view_image, is sent back once before the answer counts.
- The repeat check ignores timings and clock times, so the same test run four times
  ('75 passed in 0.14s', '0.12s') counts as a repeat.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
A real run on the camera wall ended after two dropped replies, and nobody could see
what the model had been thinking: the session doesn't keep them. Now the dropped
reply (thinking and text) is saved with the other long outputs and the message
gives the path.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…f reasoning about it

In a real run the model's dropped thinking said 'let me create a test config and run
the app' ten times without doing it. The note after such a reply now says: if you
were about to check something by running it, run it now; for a window, take a
screenshot (with actions for clicks or keys).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
In a real run the model double-clicked the camera with the new actions and the
cause, 'QWidget::setLayout: ... which already has a layout', was in what the
program printed, among 30 lines of camera logs. The model noticed it and then went
back to its own theories. Qt's warnings are now taken from everything the program
printed (not only the shown end), without repeats or offscreen noise, and listed
first: 'Qt printed warnings (they often name the cause)'.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
A real run on the camera wall ended with lcode's checks failing: they ran
'python -m unittest discover' with the Python on PATH, which has neither the
project's dependencies (PyQt6, PyAV) nor a way to run its pytest-style tests. The
model then 'fixed' that instead of the user's problem.

- Python checks run with the project's .venv/bin/python (or venv, env) when there is one.
- Tests that import pytest, or a requirements file listing it, get pytest without a
  pytest configuration.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@nasser1941 nasser1941 changed the title Look at a window before calling it done: screenshot actions, unseen-window check, steady repeats Look at a window before calling it done: screenshot actions and Qt warnings, unseen-window check, steadier repeats, project-Python checks Oct 21, 2026
@nasser1941
nasser1941 merged commit b019175 into main Oct 21, 2026
12 checks passed
@nasser1941
nasser1941 deleted the feat/look-at-windows branch October 21, 2026 10:23
@nasser1941 nasser1941 mentioned this pull request Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant