36 runs · 290 tests · $13.50 total API spend.
Published GUI-grounding numbers rarely say what exact prompt, output format, or coordinate convention produced them, so a reported score mixes model quality with evaluation choices - this project measures those confounds directly instead of treating them as noise. The task is point grounding on ScreenSpot-Pro (1,581 samples, targets averaging 0.07% of screen area - small enough that implementation details move the score by several points instead of disappearing into noise).
Four controlled ablations, each isolating one variable while holding prompt wording, coordinate space, and image encoding fixed:
- Output format (native vs. raw coordinates), replicated in opposite directions on two model families
- That same format effect re-measured across four model sizes (3B/8B/22B/32B active parameters), with a numeric prediction made before the fourth run and checked against it
- Inference-time scaffolds - single call, 2-call crop-and-relocate (ReGround), and an 8.5-call uncertainty-gated zoom reproduced from UI-Zoomer - compared on cost and accuracy
- Active vs. total parameter count as predictors of grounding accuracy, across one model family at four sizes
What came out of it:
- Output format alone shifts accuracy by 3-7pp in both directions (McNemar p<1e-4), a variable no model card reports
- That format effect looked to scale linearly with model capacity across three sizes; a pre-registered fourth measurement missed the predicted value by 2.3pp, falsifying that story - the effect instead splits along MoE-vs-dense architecture, reported here as a hypothesis, not a conclusion
- Unconditional 2-call ReGround (73.3%) beats 8.5-call uncertainty-gated zoom (59.5%) at a quarter the cost - the gating logic works, but sits on a voting scheme that discards correct minority answers
- Active parameters predict accuracy while total parameters barely do - a 30B-total/3B-active model scores 15.75pp below an 8B dense model (p=6.8e-31)
None of this is a leaderboard entry - it's a methodology and its evidence: two prompt-level variables worth 3-7pp that the field currently doesn't report, a demonstration that a clean 3-point scaling trend can fail a held-out prediction reported straight rather than reframed after the fact, and an evaluation harness (290 tests, config-sidecar reproducibility, three-baseline external calibration) built so every number above can be independently re-derived.
- Experimental sub-tasks - table of every ablation, its independent variable, and what was held fixed
- Results - headline numbers, plus external calibration against three published baselines
- Four findings - the evidence and statistics behind each claim above
- Cost - $/correct-click, and where the API spend went
- Quick start - install, run the test suite, run one sample against a real model
- Engineering decisions - why the harness is built the way it is, and what broke before it was
- Layout - what's in each folder
- Known limitations - what this project does not resolve, stated plainly
- References - related work and where each zoom method comes from
Each ablation below is a self-contained experiment with its own independent variable, controls, and sample size. Full evidence and statistics for each are in Four findings (linked per row).
| Sub-task | Models | Independent variable | Held fixed | n | Finding |
|---|---|---|---|---|---|
| Output format ablation | Qwen3-VL-32B, UI-TARS-1.5-7B | Output format (native <tool_call>/<point> vs. raw [x,y]) |
Wording, identical line-by-line | 1,581 each | #1 |
| Bare vs. ReGround, full set | Qwen3-VL-32B, UI-TARS-1.5-7B | Scaffold (single call vs. 2-call ReGround) | Model, prompt | 1,581 (Qwen3-VL), 200 (UI-TARS) | #2 |
| Scaffold comparison incl. gated zoom | Qwen3-VL-32B | Calling strategy (bare / sample-and-vote / gated zoom / ReGround) | Model, prompt, sample subset | 200 (all 4 methods, same subset) | #3 |
| Active vs. total parameters | Qwen3-VL 3B/8B/22B/32B active | Model size (active vs. total parameters) | Prompt, coordinate space, architecture family | 1,581 each | #4 |
| Format effect across model scale | Qwen3-VL 3B/8B/22B/32B active | Model size × output format (interaction) | Prompt pair, coordinate space, image encoding | 1,581 each | #4 |
| External calibration | UI-TARS-1.5-7B, ShowUI-2B, GPT-4o | Model (three coordinate conventions) | Each model's own official prompt/wrapper | 200-1,581 | Results |
Two calls beat eight and a half. That is the practical headline: unconditional crop-and-reground reaches 73.3%, while UI-Zoomer's uncertainty-gated variant - which spends 8 sampled calls before deciding whether to zoom - reaches 59.5%. Sampling eight times and voting, without zooming, is worse than a single call (the "sample+vote" bar - not one of the seven Results-table rows, added here from finding 3 since this is the claim it backs). The non-blue bars (UI-TARS, ShowUI-2B, GPT-4o) are single-call baselines on different models, shown for scale - not part of the calls-vs-accuracy comparison, which is a same-model, same-prompt argument about Qwen3-VL-32B.
| Model | Configuration | n | Accuracy | $/correct click |
|---|---|---|---|---|
| Qwen3-VL-32B | toolcall prompt + ReGround (2 calls) | 1,581 | 73.31% | $0.00071 |
| Qwen3-VL-32B | toolcall prompt, single call | 1,581 | 56.42% | $0.00051 |
| Qwen3-VL-32B | official harness prompt | 1,581 | 56.29% | $0.00055 |
| Qwen3-VL-32B | uncertainty-gated zoom (8.5 calls) | 200 | 59.50% | $0.00394 |
| UI-TARS-1.5-7B | official GROUNDING template | 1,581 | 47.12% | $0.00155 |
| ShowUI-2B (local) | official wrapper config | 394 | 8.38% | $0 (9.4h GPU) |
| GPT-4o | official gpt4x.py prompt |
400 | 1.00% | $0.308 |
This is the evidence that the numbers above mean anything. Three models, three different coordinate conventions, spanning a 47× capability range - all land near their published values:
| Model | Coordinate convention | Ours | Published | Δ |
|---|---|---|---|---|
| UI-TARS-1.5-7B | absolute pixels, post-smart_resize |
47.12% | 49.6 (model card) | -2.5 |
| ShowUI-2B | normalized [0,1] | 8.38% | 7.7 (paper) | +0.7 |
| GPT-4o | normalized [0,1] | 1.00% | 0.8 (paper) | +0.2 |
One model matching could be luck. Three, under three different conventions, is not. Two of those conventions are undocumented - they were found by probing, and getting one wrong costs you the entire run silently (see Silent failures).
Each below states the takeaway first, then the evidence and what was ruled out.
Takeaway: match the model's fine-tuned output shape, not "whichever format seems simpler" - the wrong choice costs several points either direction.
Two prompts, identical word for word, differing only in the shape of answer requested. A test enforces the wording match line by line, so the delta is attributable to format alone.
| Model | Direction | That model's native format | Effect | McNemar p |
|---|---|---|---|---|
| Qwen3-VL-32B | bare -> native | <tool_call>{"name": "computer_use", …} |
+3.35pp | 6.2e-05 |
| UI-TARS-1.5-7B | native -> bare | Action: click(point='<point>x y</point>') |
-7.34pp | 3.6e-14 |
This is a reverse replication, not the same measurement twice: adding the native format helps one model, removing it hurts the other. And the two "native formats" are entirely different strings - so what matters is not that some format is intrinsically better, but matching the shape that model was fine-tuned on.
On Qwen3-VL this single change accounted for the entire advantage of the official harness prompt (toolcall-only vs full harness: +0.13pp, p=0.935). Its system prompt, tool schema, and "guess a most likely one" instruction contribute approximately nothing.
Ruled out, on the UI-TARS pair:
- Not format non-compliance - 1,581/1,581 obeyed the bare format; zero fell back to the native one.
- Not parsing - 2/1,581 parse failures. Cannot produce 7pp.
- Not chain-of-thought - output tokens 10.9 (bare) vs 19.0 (native). The bare arm is shorter and worse; the extra tokens are boilerplate, not reasoning.
Practical consequence: any reported grounding number needs its output format stated, or it is not reproducible. Most model cards omit even the prompt.
Takeaway: a cheap 2-call crop-and-relocate outperforms an 8.5-call uncertainty-gated alternative, and its gain does not shrink as the base model improves - contradicting a published claim.
ReGround = locate once, crop around that guess, locate again. Two calls.
| Model | bare | +ReGround | Δ |
|---|---|---|---|
| UI-TARS-1.5-7B | 45.0% (n=200) | 57.0% | +12.0pp |
| Qwen3-VL-32B | 56.42% (n=1,581) | 73.31% | +16.89pp (p=9.7e-54) |
Qwen3-VL has both the higher baseline and the larger gain, contradicting UI-Zoomer's reported "gain shrinks as the base model gets stronger." Their comparison was cross-architecture and confounded with training data; this one is at least same-family. Caveat stated plainly: not fully controlled - the two arms used different prompts, so some of the difference could be prompt-driven.
Where the gain lands: icon +19.4pp and tiny targets +25.6pp, versus text +15.4pp and large +10.2pp. Zoom helps most where the model was weakest - but does not close the gap. Icons remain at 49.5% against text's 88.0%.
Takeaway: the gating logic itself works (it routes hard samples correctly); it's bolted onto a voting pipeline that loses to doing nothing, so the whole thing loses to skipping it entirely.
Implemented from UI-Zoomer, whose constants are absent from the paper and were read out of the reference implementation.
Same 200 samples, all with real box extent:
| Method | Calls | Accuracy |
|---|---|---|
| bare, single call | 1× | 47.50% |
| sample-and-vote (8 draws, no zoom) | 8× | 46.50% |
| gated zoom, best τ | 8.53× | 59.50% |
| ReGround | 2× | 70.50% |
Sample-and-vote at 8× is 1pp worse than one call. Gating rescues it to 59.5%, and its routing logic verifiably works - the tau sweep is monotone, and the premise holds (gated-out samples have 5× more recoverable headroom). But ReGround beats it by 11pp at a quarter the cost.
So the conclusion is not "gating fails." It is that gating is bolted onto a pipeline that loses to doing nothing. Two independent defects compound: the provider returns logprobs for only 1 of 35 output tokens (making the confidence term a constant, so tie-breaking is dead), and majority-IoU voting actively discards minority-correct draws.
A related finding worth its own line: upstream's τ=1.5 degrades to a no-op on both models, in opposite directions - 7.5% pass rate on UI-TARS (nearly everything zooms), 91% on Qwen3-VL (nearly nothing does). A threshold tuned on one model's score distribution does not transfer.
Takeaway: judge a MoE model's grounding accuracy by its active parameters, not its total size - and be skeptical of any 3-point "scaling law," including one from this project (see below).
Four sizes, one architecture, identical prompt and coordinate space, full n=1,581:
| Model | Active | Total | Accuracy | $/correct |
|---|---|---|---|---|
| 30B-A3B | 3B | 30B | 35.23% | $0.00283 |
| 8B | 8B | 8B | 50.98% | $0.00073 |
| 32B | 32B | 32B | 56.42% | $0.00051 |
| 235B-A22B | 22B | 235B | 60.53% | $0.00138 |
30B-A3B has nearly 4x the total parameters of 8B and scores 15.75pp worse (p=6.8e-31). Ordered by total parameters the curve is badly non-monotone; ordered by active parameters it is monotone apart from 235B-A22B's 22B active edging past 32B dense. All three adjacent comparisons are significant.
Verified this was not a technical fault before believing it: 1,581/1,581 format-compliant, zero parse failures, output tokens matching 32B, coordinates mapping correctly. The failure mode is pointing at the wrong element.
32B dense is the cost-efficiency sweet spot - 235B buys +4.11pp for 2.9x the money, while the output-format change costs nothing and is worth +3.35pp, about 80% of what a 7x-larger model buys.
A fourth point breaks the clean story - and that's the more interesting
result. Finding 1 measured a large format effect on 32B (toolcall vs
rawpair, +3.35pp). Rerunning that same ablation at 8B, 30B-A3B, and
235B-A22B, all n=1,581, strictly paired (same 1,581 sample_ids scored under
both prompts):
| Model | Active | Architecture | toolcall | rawpair | Δ (format effect) | McNemar p |
|---|---|---|---|---|---|---|
| 30B-A3B | 3B | MoE | 35.23% | 34.60% | +0.63pp | 0.67 (n.s.) |
| 8B | 8B | Dense | 50.98% | 48.77% | +2.21pp | 0.011 |
| 235B-A22B | 22B | MoE | 60.53% | 60.09% | +0.44pp | 0.68 (n.s.) |
| 32B | 32B | Dense | 56.42% | 53.07% | +3.35pp | 6.2e-05 |
I originally fit a line through the first three points (3B/8B/32B) and predicted the format effect at 22B active would land around +2.7pp - the whole point of running 235B-A22B was to test that prediction. It didn't hold: the effect at 22B active is +0.44pp, statistically indistinguishable from zero, landing next to 3B active rather than between 8B and 32B. Ordered by active parameters alone, the format effect is not monotone (+0.63 -> +2.21 -> +0.44 -> +3.35).
What is monotone: grouped by architecture, both MoE points (30B-A3B, 3B active; 235B-A22B, 22B active) show a non-significant format effect, and both dense points (8B, 32B) show a significant one, an order of magnitude larger. That regrouping fits the same four numbers with zero free parameters and no outliers - but it's two points per architecture, so "MoE dampens format sensitivity" is a pattern consistent with the data, not a finding established by it. I'm reporting it as the leading hypothesis, not a conclusion.
I looked for a more mundane explanation before trusting either story. 235B's
toolcall accuracy (60.53%) is the highest of the four models, so a ceiling
effect is a candidate: if the model is already scoring near its own ceiling,
there is less room for a format effect to show up regardless of the
mechanism. I don't have a way to rule that out with the current data - it
would need a task easier than ScreenSpot-Pro's icon-heavy split to check
whether 235B's format effect reappears somewhere below ceiling.
I also checked whether toolcall is even a fair thing to call "native" for
these models. The label was reverse-engineered from what 32B actually
replied under the official ScreenSpot-Pro harness system prompt - which
itself explicitly instructs the model, via a <tools> schema, to answer
inside <tool_call></tool_call> tags. That is 32B complying with an explicit
instruction, not an unprompted preference; the "native" label was always an
inference, not a measurement. I tried to test it directly: 20 samples on 32B
with a prompt that names the task and gives no format instruction at all
(scripts/probe_native_format_preference.py). Result: 20/20 replies were
conversational prose ("To modify the highlights of the photo... the user
should..."), 0/20 resembled tool_call, JSON, or any structured shape. That
kills the probe, not the hypothesis - an instruction-tuned model's default
completion under an open-ended question is prose regardless of which
structured format it was fine-tuned to use in a tool-calling context, so this
design can't distinguish "prefers tool_call" from "prefers nothing
in particular." I did not run the 235B side of this probe, since a null
design would just spend money to reproduce the same null. The tool_call
schema in the harness prompt is Qwen's documented Agent/function-calling
format, publicly specified independent of anything measured in this repo -
that's the actual basis for calling it "native," not the probe.
Practical consequence, revised: this is not a clean capacity × format scaling law. What holds up: (1) the format effect is real and large on at least two dense models (8B, 32B), (2) it is measurably absent on both MoE points tested, (3) a linear "bigger model, bigger effect" story fit to three points did not survive a fourth, pre-registered test. Report format effects per model, not as a property of a model family, and treat any curve fit through three points as a hypothesis to break, not a law.
Literature gap: of the three zoom papers this work implements (ZoomClick, UI-Zoomer, GUI-Lens), none reports token or dollar cost. One reports latency.
| Method | Calls | Accuracy | $/1,581 samples | $/correct click |
|---|---|---|---|---|
| Qwen3-VL bare | 1× | 56.42% | $0.45 | $0.00051 |
| Qwen3-VL +ReGround | 2× | 73.31% | $0.82 | $0.00071 |
| Qwen3-VL gated zoom | 8.53× | 59.50% | ~$3.71 | $0.00394 |
| UI-TARS bare | 1× | 47.12% | $1.15 | $0.00155 |
| GPT-4o | 1× | 1.00% | ~$4.87 | $0.308 |
ReGround buys +16.89pp for a 1.8× cost increase. Gating buys +3.1pp for 8×. Specialised grounding models are one to two orders of magnitude cheaper per correct click than a general VLM.
One counterintuitive result: for Qwen3-VL, not downscaling images is
cheaper. Its factor=32 patches are larger than Qwen2.5-VL's 28, which
offsets the pixel increase - $0.244 vs $0.306 over 400 samples.
# Python 3.12 (3.10 is too old for some dependencies)
uv venv --python 3.12
uv pip install -e ".[dev]"
# 290 tests, no network and no API key required - they drive the pipeline
# with a scripted model, so they cost nothing and run in ~15s
pytestEvaluating against the real benchmark additionally needs the ScreenSpot-Pro
images (3.38 GB, manual download). Local ShowUI-2B inference needs
pip install -e ".[local]" plus a torch build matching your CUDA driver - both
are optional and neither is required for the test suite or for any API-based
model.
To run against a real model, add an OpenRouter key:
echo "OPENROUTER_API_KEY=sk-or-..." > .envfrom ssp_eval import (
load_samples, stratified_subset, QwenVLModel, CoordSpace,
ReGroundScaffold, ScaleStrategy, evaluate, summarize,
)
from ssp_eval.prompts import QWEN_NATIVE_POINT_1000_TOOLCALL
samples = stratified_subset(load_samples(), 50, seed=0)
model = QwenVLModel(
"qwen/qwen3-vl-32b-instruct",
CoordSpace.FIXED_GRID, # Qwen3-VL always answers on a 1000x1000 grid
grid=(1000, 1000),
prompt_template=QWEN_NATIVE_POINT_1000_TOOLCALL,
factor=32, # 32 for Qwen3-VL, 28 for Qwen2.5-VL
)
results = evaluate(
samples, model,
ReGroundScaffold(window=(1024, 1024), scale=ScaleStrategy.TARGET_PIXELS),
out_dir="runs", budget_usd=1.00, # hard spend cap
)
print(summarize(results).accuracy)Reproducing a published run: every runs/*/*.config.json sidecar is committed
and records the model, full prompt text, coordinate space, image encoding and
resolution policy for that run.
Coordinate spaces differ per model, and getting one wrong raises no error - it just clicks the wrong place.
| Model | Coordinate space | How it was determined |
|---|---|---|
| UI-TARS-1.5-7B | absolute pixels, post-smart_resize |
probing; undocumented |
| Qwen3-VL-32B | 1000×1000 virtual grid | probing; the README example (1016 > 1000) misleads |
| ShowUI-2B / GPT-4o | normalized [0,1] | official wrappers |
So coordinate space is a required attribute of every model adapter, never
inferred at parse time. tests/test_geometry.py contains a test that
demonstrates the failure mode: treating post-resize coordinates as original-image
coordinates shifts the click by 960 pixels, and the resulting point looks
entirely plausible.
This project paid that price once, before the test existed: assuming Qwen3-VL's
bbox_2d was absolute pixels produced 0/8 - see
runs/_stale_nativebox_n8_absolute_image/.
Reading a run script to determine what a run actually sent has produced a wrong
published claim on this project: backfill_qwen3vl_failures.py describes only 70
of 1,581 rows, so a mixed PNG/JPEG encoding went unnoticed and the difference
between two arms was reported as "prompt only." Another script turned out to
evaluate a different model entirely.
Hence every run writes a *.config.json, and those are committed even though the
per-sample outputs are not.
Dropping them would flatter models with unreliable output formats. The
fallback parser requires the reply to contain exactly two numbers - otherwise
it would happily extract (3, 2) from "I see 3 buttons and 2 menus." Better to
score a miss than invent a click.
For a 6pp effect:
| n | Δ | 95% CI | Verdict |
|---|---|---|---|
| 50 | +0.060 | [+0.000, +0.140] | not significant |
| 200 | +0.060 | [+0.030, +0.095] | significant |
| 1,581 | +0.060 | [+0.049, +0.072] | significant, 2.3pt wide |
Two independent groups of 50 (52% vs 58%) give overlapping intervals and detect nothing. Paired designs let expensive models be evaluated on a 200-sample stratified subset.
But an underpowered null is not a null, and this project fell for that twice: at n=400 two prompts looked tied (p=0.38) when the full set showed +3.23pp (p=7.3e-04); and a 40-sample encoding control that "agreed 39/40" was treated as ruling out an effect, when that interval spans [0.1%, 13%] - wide enough to contain the entire 3pp gap being investigated. A ~3pp effect needs the full 1,581.
test_differs_from_point1000_only_in_output_format asserts line by line that two
prompts are identical apart from the output format. Without it, someone tidying
the wording would quietly convert a format ablation into a multi-variable
comparison - and nothing would fail. Given that format is worth 3.35pp and the
template is worth 4.68pp, such an edit has a measurable cost.
- Hard spend cap, default $5. Tools in this space routinely ship without one (browser-use still has this open), and unattended batch runs are how surprise bills happen.
- Streaming JSONL + resume by
sample_id. A crash does not lose work already paid for. Learned the hard way: misjudging a run as dead and relaunching it duplicated 297 samples, so run status is now determined by counting rows on disk, not by looking at the process. - Sampling separated from scoring. Crop windows are a pure function of the sampled points, so draws were bought once ($0.90) and every crop-geometry variant was recomputed offline for $0 - including the most informative diagnostic of the project.
ssp_eval/ harness: dataset loading, coordinate transforms, model
adapters, scaffolds, uncertainty gating, metrics
tests/ 290 tests, network-free (scripted model, no API key needed)
scripts/ one script per experiment or analysis (51 scripts)
runs/ config sidecars (committed) + per-sample outputs (gitignored)
docs/ README figures, plus literature review and write-up notes
reference/ third-party wrapper copies, for byte-comparison; not executed
Key modules: geometry.py (the only path back to original-image coordinates)
and prompts.py (verbatim transcriptions of official templates - the file whose
contents are worth several points of accuracy).
-
Coordinate-scaling arithmetic is not fully closed-loop. Verifying it needs a model with absolute-pixel output and a published number from a known harness. UI-TARS satisfies only the first (ByteDance's harness is not public). Under normalized coordinates the sent size cancels algebraically, so the GPT-4o and ShowUI checks bypass the resize path entirely.
-
The icon gap is unresolved and is the largest effect in the project. 42.5pp bare (text 72.7% vs icon 30.1%), 38.5pp after ReGround. None of the 31 runs targeted it. Known ineffective: more pixels (icons gain +1.3pp), prompt wording, a stronger model. Diagnosis points to localisation, not recognition - 82% of failed boxes do not overlap the target at all, while their sizes are roughly right.
Every intervention here lifts both curves and leaves the gap between 38 and 43pp. That is the strongest evidence that the remaining headroom is not a prompting or resolution problem.
-
Scaffold results are two models, not four. GPT-4o and ShowUI were calibration points only; their baselines (1% and 8%) are too low for a gain to clear the confidence interval.
-
Gating on Qwen3-VL could only be run single-signal. The provider returns logprobs for 1 of 35 output tokens, making UI-Zoomer's confidence term a constant. This is a provider limitation, not a result about the method.
-
temperature=0, single run per arm. No seed-variance estimate.
| Work | arXiv | Relevance |
|---|---|---|
| SeeAct | 2401.01614 | grounding, not planning, is the bottleneck |
| ScreenSpot-Pro | 2504.07981 | the benchmark; defines ReGround |
| UI-TARS | 2501.12326 | end-to-end GUI agent |
| UGround | 2410.05243 | vision-only, modular |
| Set-of-Mark | 2310.11441 | coordinates - discrete choice |
Zoom methods implemented or studied here: ReGround (ScreenSpot-Pro §4.1), UI-Zoomer's uncertainty-gated zoom, ZoomClick's crop parameters.
crop-and-zoom is not this project's invention - ScreenSpot-Pro publishes the ladder itself (18.9 -> 31.0 -> 40.2 -> 48.1). The contribution here is implementation, controlled comparison, and the cost and reproducibility dimensions the papers leave out.