Scry Design Diff Eval

60-row pilot analysis

AuditPaper
Pilot Set
60 rows
45 diff screens, 15 no-tagged controls
Tagged Issues
140
10 tag families
Top KIR
Kimi K2.7 Code 23.6%
33/140
Kimi Format
42/60 valid
2 invalid, 16 empty

Model Comparison

Known-Issue Recall

Label Recall

MiniMax M3

Color/Background7/50
14.0%
Icon/Nav6/48
12.5%
Shape/Size8/33
24.2%
Typography1/33
3.0%
Spacing/Layout5/21
23.8%
Missing Content5/11
45.5%
Image/Asset3/6
50.0%
Separator1/6
16.7%
Extra Element0/1
0.0%
other0/1
0.0%

Defect Density

MiniMax M3

Single15
Issue 26.7%Screen 26.7%
Multi15
Issue 19.4%Screen 40.0%
Dense15
Issue 13.5%Screen 53.3%

Output Health

Parser Stability

MiniMax M3
60 valid0 empty0 invalid
Gemma 4 31B
60 valid0 empty0 invalid
Gemma 4 26B A4B
60 valid0 empty0 invalid
Kimi K2.7 Code
42 valid16 empty2 invalid
Kimi K2.7 Code failures18

results/kimi-k2.7-code-pilot/output_health.json

Empty outputs
cleo__116coinbase__86craft__272craft__366craft__450creme__62flighty__22grab__440instacart__141klarna__223monzo__280shopify__118todoist__96todoist__107todoist__266vivid__226
Malformed outputs
headway__60Expecting property name enclosed in double quotes: line 5 column 154 (char 219)
instacart__61Expecting ',' delimiter: line 27 column 48 (char 1168)
Runner reasons
ok: 42empty content: 16invalid JSON: Expecting property name enclosed in double quotes: line 5 column 154 (char 219): 1invalid JSON: Expecting ',' delimiter: line 27 column 48 (char 1168): 1

Recall-First Diagnostics

Unmatched Prediction Audits

Extra model issues are review queues, not primary-score false positives.

MiniMax M3reports/pilot/minimax-m3-15x4-unmatched-predictions.csv
189
Gemma 4 26B A4Breports/pilot/gemma-4-26b-a4b-15x4-unmatched-predictions.csv
160
Kimi K2.7 Codereports/pilot/kimi-k2.7-code-15x4-unmatched-predictions.csv
135
Gemma 4 31Breports/pilot/gemma-4-31b-15x4-unmatched-predictions.csv
130

Screen Outcomes

60 rows

TPFNFPTN
ScreenSplitTagsMiniMaxG31BG26BKimi
Amazon Shopping ios Aug 2025amazon-shopping__801No Diff
none
TN0 match / 0 predFP0 match / 3 predFP0 match / 4 predFP0 match / 5 pred
Apple Fitness ios Feb 2026apple-fitness__148No Diff
none
FP0 match / 4 predFP0 match / 1 predFP0 match / 2 predFP0 match / 4 pred
Binance ios Dec 2025binance__100No Diff
none
FP0 match / 2 predFP0 match / 3 predFP0 match / 2 predFP0 match / 5 pred
Bumble ios Jun 2024bumble__133No Diff
none
FP0 match / 3 predFP0 match / 1 predFP0 match / 1 predFP0 match / 3 pred
Cash App ios Nov 2025cash-app__7No Diff
none
TN0 match / 0 predFP0 match / 1 predFP0 match / 2 predFP0 match / 3 pred
Cash App ios Nov 2025cash-app__217No Diff
none
FP0 match / 3 predFP0 match / 3 predFP0 match / 2 predFP0 match / 1 pred
Cleo ios Feb 2024cleo__1Single
Color/Background
FN0 match / 4 predFN0 match / 3 predFN0 match / 3 predTP1 match / 4 pred
Cleo ios Feb 2024cleo__116No Diff
none
FP0 match / 5 predFP0 match / 5 predFP0 match / 3 predTN0 match / 0 pred
Coinbase ios Jan 2025coinbase__86No Diff
none
FP0 match / 3 predFP0 match / 2 predFP0 match / 2 predTN0 match / 0 pred
Craft ios Dec 2025craft__60No Diff
none
FP0 match / 3 predFP0 match / 2 predFP0 match / 2 predFP0 match / 2 pred
Craft ios Dec 2025craft__272Multi
Color/BackgroundIcon/NavMissing Content+2
FN0 match / 0 predTP1 match / 4 predTP2 match / 4 predFN0 match / 0 pred
Craft ios Dec 2025craft__366Single
Color/BackgroundIcon/Nav
FN0 match / 5 predFN0 match / 5 predFN0 match / 6 predFN0 match / 0 pred
Craft ios Dec 2025craft__410Dense
Color/BackgroundIcon/NavShape/Size
FN0 match / 0 predFN0 match / 3 predFN0 match / 5 predFN0 match / 5 pred
Craft ios Dec 2025craft__450Dense
Icon/NavMissing ContentSpacing/Layout+1
FN0 match / 4 predTP1 match / 3 predTP3 match / 5 predFN0 match / 0 pred
Craft ios Dec 2025craft__461Dense
Color/BackgroundIcon/NavShape/Size+2
FN0 match / 0 predTP1 match / 3 predTP1 match / 3 predTP1 match / 4 pred
Craft ios Dec 2025craft__462Dense
Icon/NavMissing ContentTypography+1
TP1 match / 4 predTP1 match / 4 predTP1 match / 4 predTP1 match / 5 pred
Craft ios Dec 2025craft__513Single
Typography
FN0 match / 0 predFN0 match / 0 predFN0 match / 3 predTP1 match / 5 pred
CREME ios Jul 2025creme__62No Diff
none
FP0 match / 4 predFP0 match / 3 predFP0 match / 4 predTN0 match / 0 pred
Flighty ios Dec 2025flighty__22Dense
Color/BackgroundMissing ContentTypography
FN0 match / 3 predFN0 match / 4 predFN0 match / 2 predFN0 match / 0 pred
Flo ios Mar 2024flo__105No Diff
none
FP0 match / 5 predFP0 match / 3 predFP0 match / 4 predFP0 match / 4 pred
Gojek ios Jul 2024gojek__450Multi
Missing ContentShape/Size
TP2 match / 4 predFN0 match / 4 predTP1 match / 4 predTP2 match / 5 pred
Grab ios Aug 2025grab__226Multi
Color/BackgroundIcon/NavImage/Asset
TP1 match / 5 predTP1 match / 4 predTP1 match / 6 predTP2 match / 4 pred
Grab ios Aug 2025grab__440Multi
Color/BackgroundShape/SizeSpacing/Layout
FN0 match / 3 predFN0 match / 2 predFN0 match / 4 predFN0 match / 0 pred
Grab ios Aug 2025grab__565No Diff
none
FP0 match / 3 predFP0 match / 3 predFP0 match / 4 predFP0 match / 3 pred
Headway ios Nov 2024headway__60No Diff
none
FP0 match / 3 predFP0 match / 1 predFP0 match / 3 predTN0 match / 0 pred
IKEA ios Jun 2025ikea__124Multi
Spacing/Layout
TP1 match / 5 predFN0 match / 3 predTP2 match / 3 predFN0 match / 3 pred
IKEA ios Jun 2025ikea__338Dense
Extra ElementMissing ContentSeparator+1
FN0 match / 3 predFN0 match / 2 predFN0 match / 2 predFN0 match / 3 pred
Instacart ios Mar 2025instacart__61No Diff
none
FP0 match / 5 predFP0 match / 2 predFP0 match / 2 predTN0 match / 0 pred
Instacart ios Mar 2025instacart__141Single
Shape/SizeSpacing/Layout
FN0 match / 4 predFN0 match / 0 predFN0 match / 2 predFN0 match / 0 pred
Instagram ios Jan 2026instagram__370Single
Color/BackgroundIcon/Nav
FN0 match / 5 predFN0 match / 4 predFN0 match / 4 predFN0 match / 5 pred
Klarna ios Nov 2024klarna__223Single
Icon/Nav
FN0 match / 4 predFN0 match / 4 predFN0 match / 2 predFN0 match / 0 pred
Lyft ios May 2022lyft__172Dense
Color/BackgroundIcon/NavMissing Content+3
TP2 match / 4 predFN0 match / 3 predTP1 match / 4 predTP3 match / 5 pred
Monarch ios Jul 2025monarch__85Dense
Icon/Nav
FN0 match / 3 predFN0 match / 1 predFN0 match / 2 predFN0 match / 4 pred
Monarch ios Jul 2025monarch__143Dense
Icon/NavSpacing/LayoutTypography
TP1 match / 3 predFN0 match / 0 predFN0 match / 2 predTP3 match / 4 pred
Monzo ios Jul 2024monzo__280No Diff
none
FP0 match / 4 predFP0 match / 1 predFP0 match / 2 predTN0 match / 0 pred
Nike ios Mar 2025nike__132Single
Typography
FN0 match / 3 predFN0 match / 4 predFN0 match / 3 predFN0 match / 4 pred