look-decide-act

Drive real web apps over the Chrome DevTools Protocol by looking at the page, not by replaying a recorded sequence of clicks.

Why

A click script says: click here, wait three seconds, click there. It is correct exactly once — on the layout it was written against. Worse, every later failure is silent, because element.click() returns without complaint whether or not anything happened. A broken flow looks identical to a working one until somebody watches the screen.

This library is the other shape: an observe → decide → act loop that re-reads the page every iteration and picks the one action that screen deserves. Nothing is timed. Nothing is assumed.

from look_decide_act import attach, run, click_text, upload_file, screenshot

def observe(s):
    text = (s.evaluate('document.body.innerText') or '').lower()
    return {
        'screen': 'editor' if 'add video' in text else 'unknown',
        'attached': bool(s.evaluate(
            '[...document.querySelectorAll("button")].some('
            'e => e.innerText.trim().toLowerCase() === "delete")')),
    }

def choose(state, ctx):
    if state['screen'] != 'editor':
        return {'do': 'wait'}
    if state['attached']:
        return {'do': 'done', 'why': 'media is in'}
    return {'do': 'upload'}

def perform(s, action, ctx):
    node = upload_file(s, 'clip.mp4', lambda: click_text(s, 'add video'),
                       on_failure=lambda n, s: screenshot(s, f'shots/fail_{n}.jpg'))
    return f'uploaded (node {node})'

session = attach(port=9222)          # a browser that is ALREADY running
result = run(session, observe, choose, perform)

run() returns {'ok', 'why', 'steps'}, where steps is the full trail — every screen, every decision, every result. When a flow fails, the trail is the deliverable: a flow that cannot say where it stopped has to be debugged from scratch every time.

The four things that actually break automation

Each function here exists because its obvious version failed silently on a production app.

1. element.click() is not a click. The control may be below the fold; the element matching your text is often a zero-width wrapper whose box is meaningless; and coordinates read before scrolling are stale the moment you scroll. click_text() finds by visible text, climbs to an ancestor with a real box, scrolls it into view, re-reads the box after the scroll, and dispatches move/press/release at its centre.

2. Native file dialogs have no DOM node. Many apps have no input[type=file] at all — clicking their upload control opens the operating system's file picker, which CDP cannot see as an element. The obvious approach finds nothing to set, forever, while reporting a successful click each time. upload_file() uses Page.setInterceptFileChooserDialog so the browser hands the dialog to you, then reads the backendNodeId out of the resulting event.

3. Dialogs cannot be clicked away. A beforeunload ("Leave site?"), alert or confirm is drawn by the browser, not the page — no DOM node, no button any script can reach. Until something answers it, every later command on that session blocks, so a flow that is merely waiting looks hung. Session answers them the instant they open.

4. Page text is a poor witness. A successful file upload can change no visible text at all while the DOM quietly gains a Delete control and a preview. A text-only check reports failure on a flow that worked. Take screenshots; one image answers instantly what an hour of log reading does not.

Why attach instead of launch

attach() connects to a browser that is already running. Launch flags like --remote-debugging-port are start-up only, so any plain invocation that starts first owns the profile and permanently denies the control port. Attaching has no flag to lose and no race to lose it to — and the browser keeps whatever sessions a human signed into by hand.

Install

pip install -e .

Requires websocket-client. Python 3.9+.

Start a browser with a debug port:

chromium --remote-debugging-port=9222 --remote-allow-origins='*' --user-data-dir=/tmp/profile

Scope

This is deliberately small: one session class, five actions, one loop. It is not a Playwright replacement — it is what you want when you must drive a browser a human is also using, whose session was established by hand, and where a launched-fresh automation profile would be logged out of everything.

Licence

MIT