rreu
Can VLMs Review Mobile UI? Introducing Scry Design Diff Eval
July 6, 2026
In most mobile teams the flow of UI Development is often modeled as a one directional flow of Designer -> Frontend Dev. In reality there is often a ping ponging back and forth between the nodes on final implementation often due to discrepenancy in ui or limitations feasibility.With the advent of LLM Generated UI there is often the promise of a conversion into a single node.
While coding agents are able have generate UI from UX design documents and images, they still often have gaps that often lead to human prompters having to identify and guide models to fix these. While we currently have tools to diff at a pixel level these can be overly noisy when comparing between design files and actual screenshots, and often not useful for UI refinement. Our goal was to see which vision language models would be best suited for this task, and in doing so we build a benchmark.
The benchmark
Scry Design Diff Eval contains 311 human-reviewed mobile UI pairs: a reference screenshot and a generated implementation of the same screen. 234 pairs carry 557 human-tagged defects — each one a list of detected tags (Icon/Nav, Typography, Missing Content, ...) plus a hand-drawn selection box — and 77 pairs with no tagged issues serve as controls. A model must return a structured issue list: tags and normalized boxes, not prose.
Scoring is deterministic and recall-first. A model issue counts only if it shares a tag with a human issue and its box overlaps (IoU ≥ 0.10) on the same image, matched one-to-one. Human annotations are known positives, not exhaustive ground truth, so extra model findings are reported as diagnostics rather than penalties.
Results
Three numbers to know, in plain English:
- Known-Issue Recall — of all the defects human reviewers tagged, how many did the model also catch (matching tag, overlapping box)? Higher means fewer missed defects. This is the primary score.
- Diagnostic Precision — of everything the model flagged, how much lined up with a known human-tagged defect? Low precision means lots of extra flags a human still has to triage — though some extras may be real defects the annotators didn't tag, which is why we call it diagnostic.
- Issue F1 — a single combined score that balances the two. It is only high when a model catches most known defects without burying them in extra flags.
We evaluated seven models on the full 311-pair set:
| Model | Known-Issue Recall | Diagnostic Precision | Issue F1 |
|---|---|---|---|
| Claude Fable 5 | 40.4% | 16.0% | 22.9% |
| Kimi K2.7 Code | 38.2% | 15.9% | 22.5% |
| Gemini 3.5 Flash | 37.5% | 20.2% | 26.3% |
| Codex GPT-5.5 xhigh | 37.3% | 13.6% | 19.9% |
| MiniMax M3 | 21.9% | 11.5% | 15.0% |
| Gemma 4 26B A4B | 20.3% | 12.4% | 15.4% |
| Gemma 4 31B | 17.8% | 13.3% | 15.2% |
Claude Fable 5 was run as Claude Code subagents (one agent per image pair with schema-enforced JSON output), so like the Kimi row it measures a model-plus-harness setup rather than a bare API endpoint.
Four things stood out:
- The best models catch roughly two fifths of known issues. Claude Fable 5 leads at 40.4% recall with Kimi, Gemini, and Codex a point or two behind — but Gemini 3.5 Flash produces hundreds fewer predictions than the other leaders, giving it the best precision and F1 of the high-recall group.
- Noticing a diff is easy; enumerating defects is hard. The leaders find something on 62–67% of defective screens, but issue-level recall tops out at ~40% — on dense screens most annotated defects go unrecovered.
- Defect families differ wildly.
Missing Content(77.3%) and changed illustrations (Image/Asset, 76.7%) are the easiest;Typographytops out at 20.0%. - Abstention is basically absent. The high-recall models flag 92–100% of the no-tagged controls. A review tool that flags every screen still needs a human triage pass. This is due to the fact that there are many small defects that would pass human review but are picked up by the model. Also models often have a high hallucination rate of defect detection.
The takeaway: Current VLM endpoints while capable of generating strong mobile UI designs are nowhere near a drop-in replacement for a human reviewer. We hope to release a full paper on this issue along with further works on improving the performance on this task.
Get the data
- Dataset (311 pairs, images + annotations): huggingface.co/datasets/Scrymore/scry-design-diff-eval