Dark Factory Tiers: How Strong a Model Set Does Each Job Need?
A dark factory run takes one message and ships to production with no further human input: builder, reviewer and verifier agents, a scripted acceptance test, and a release switch. The question for each new job is which model set to staff it with. Small models are cheap and fast, but a run that fails costs more than it saves.
We graded every feature and fix we had shipped, 134 items as of 13 Oct 2026, by the weakest model set that could finish them. The cheapest set, Haiku 5.5 builders under a Sonnet 5.5 orchestrator, covers six one-file fixes. Most of the work, 87 items, is longer than any measured model can finish reliably in one attempt. Those shipped because separate builder, reviewer and verifier agents caught the misses, and large items were sliced into shorter tasks. Twelve large features are beyond one autonomous run today.
All numbers in this post are as of 13 Oct 2026.
| Items | Share | |
|---|---|---|
| T1 Haiku tier | 6 | 4% |
| T2 Sonnet, one pass | 29 | 22% |
| T3 and T4, past the reliable horizon | 87 | 65% |
| T5 not autonomous today | 12 | 9% |
| Not completed (graded the same way) | 28 |
Where each item sits
Where each item sits
Estimated human-expert hours per item (log scale) against METR time horizons · 134 shipped, 28 not completed · as of 13 Oct 2026
| Tier | Shipped | Hours (range) | Median | Not completed |
|---|---|---|---|---|
| T1 Haiku tier | 6 | 42 min to 1.5 h | 1.5 h | 1 |
| T2 Sonnet, one pass | 29 | 2 h to 6 h | 4 h | 4 |
| T3 Frontier and a verify loop | 51 | 3 h to 20 h | 12 h | 11 |
| T4 Frontier and slicing | 36 | 20 h to 90 h | 45 h | 7 |
| T5 Not autonomous today | 12 | 30 h to 250 h | 120 h | 5 |
- Shipped (134)
- Not completed (28), placed at the whole job's estimate
- METR measured horizon
- Trend extrapolation to Oct 2026
- Past 16 h: unreliable
Each dot is placed by the hours a senior engineer who knows the codebase would need for the whole job, from build to verified release. The vertical lines are METR time horizons: the task length at which a model succeeds 80% or 50% of the time on METR's suite. The solid ink dots are the 28 items that were not completed, graded on an estimate of the whole job and placed by the same rule.
The five tiers
One rule was applied to every item. Hours set the base tier. Desktop or VM GUI work adds one tier. Two or more of security or money, data migration and race conditions set a floor of T3. A step only a person can legally or physically do makes it T5.
| Tier | Model set that succeeds | Rule | Benchmark anchor |
|---|---|---|---|
| T1 Haiku tier | Haiku 5.5 builder, reviewer and verifier under a Sonnet 5.5 orchestrator | Up to 1.5 h in one component, with no second repo or service, security or money, data changes, races, desktop or VM work, or vendor accounts | Inside the 80% horizon of every frontier model METR measured in 2026 (Opus 4.6: 70 min; Mythos Preview: 3.1 h) |
| T2 Sonnet, one pass | Sonnet 5.5 builders at max effort, one review round, a standard release | 1.5 to 6 h, in one repo or across two with a clear contract | Above the best measured 80% horizon, up to the 50% horizon of late-2025 frontier models (Opus 4.5 4.9 h, GPT-5.2 5.9 h) |
| T3 Frontier and a verify loop | Opus 5.5 or Fable 5.1 orchestrating Sonnet builders, with a separate reviewer and a scripted-UAT verifier | 6 to 20 h; or shorter work stacking two risk drivers; or desktop work that would otherwise be T2 | Between the best model's 80% (3.1 h) and 50% (17.4 h) horizons: one attempt succeeds roughly half the time |
| T4 Frontier and slicing | The same models, with the work cut into T2 and T3 slices across sessions and shipped dark behind release switches | 20 to 100 h, or desktop work that would otherwise be T3 | Longer than the 50% horizon of any measured model, and past 16 h, where METR says its numbers stop being reliable |
| T5 Not autonomous today | No model set finishes these in one run | 100 h or more, or a step only a person can do: identity checks, legal agreements, vendor contracts | Beyond the top Artificial Analysis score (Opus 5.5, 58) acting alone, and beyond the extrapolated 48 h horizon |
Medians by tier: T1 1.5 h, T2 4 h, T3 12 h, T4 45 h, T5 120 h.
What the data says
The Haiku tier is small and specific. Six of 134 items: a CSS overflow, a navigation order, a copy sweep, spacing, icons. One trial ran it end to end. Four Haiku 5.5 agents built, reviewed and verified a text-overflow fix on a card, and it reached production in 22 minutes against a 45-minute estimate, with no fix rounds. The Haiku models' share of the spend was $0.11.
Sonnet 5.5 in one pass covers ordinary bugs and small features (22%), including security fixes with a single code path, such as a token leak in a command-line login and a missing membership check on an image proxy. Sonnet 5.5 scores 56 on the Artificial Analysis index, two points under Opus 5.5, so for this tier a stronger model buys little.
Most work sits past the reliable horizon of any measured model (65%). These 87 items need 6 to 100 human hours. They shipped through the review and scripted-UAT loop and through slicing. The spend is split between Sonnet 5.5 and Opus 5.5 in T2 to T4, and the mix moves only modestly with the tier (table below). In our data, task length and the number of moving parts separate these tiers more than the builder model does.
Twelve features are beyond one autonomous run today (9%). All of them shipped, but over many sessions and with founder decisions. An image generation product took 32 PRs and 4 rollbacks; a dashboard redesign took 55 PRs against a 21-PR plan. Two needed a step only a person can do: a telecom vendor's identity check for alert escalation, and vendor accounts plus a data-processing agreement for product analytics. T5 is the only tier where Opus 5.5 carried most of the spend.
Agent spend by tier, for items whose agent sessions are linked to the work:
| Tier | Items traced | Median spend per item | Sonnet 5.5 share | Opus 5.5 share |
|---|---|---|---|---|
| T2 | 26 of 29 | $9.46 | 50% | 33% |
| T3 | 42 of 51 | $23.65 | 33% | 37% |
| T4 | 32 of 36 | $69.56 | 40% | 41% |
| T5 | 12 of 12 | $233.53 | 21% | 58% |
T1 has one traced item, so it has no row. The rest of each tier's spend is older Sonnet and Opus versions and Fable 5.1.
What made it hard
What made it hard
Share of each tier's shipped items that list the reason · 134 items · as of 13 Oct 2026
| Reason | T1 (n=6) | T2 (n=29) | T3 (n=51) | T4 (n=36) | T5 (n=12) |
|---|---|---|---|---|---|
| founder call | 0% | 45% | 55% | 78% | 100% |
| cross-service | 0% | 14% | 53% | 89% | 100% |
| flaky infra | 33% | 21% | 49% | 75% | 92% |
| multi-repo | 0% | 7% | 35% | 92% | 92% |
| security/money | 0% | 21% | 37% | 67% | 92% |
| vendor account | 0% | 10% | 27% | 67% | 83% |
| taste | 67% | 45% | 25% | 19% | 58% |
| perf | 0% | 10% | 41% | 44% | 25% |
| data | 0% | 14% | 27% | 44% | 50% |
| race | 0% | 10% | 22% | 19% | 42% |
| unclear spec | 0% | 7% | 10% | 11% | 42% |
| VM GUI | 0% | 0% | 6% | 22% | 42% |
| desktop | 0% | 0% | 6% | 19% | 33% |
Founder decisions, cross-service work, multi-repo changes and vendor accounts grow steadily from T2 to T5; every T5 item needed founder decisions and crossed services. Taste is the exception: it marks four of the six T1 fixes, fades through the middle tiers and returns in T5. Desktop and VM GUI work never appears below T3, which is where the rule's one-tier bump puts it.
The dark factory runs so far
Twenty items had shipped through a complete dark factory run by 13 Oct: five in T1, three in T2, seven in T3, five in T4 and none in T5. T1 and T2 runs took 0 to 3 rounds. Every T4 run needed slicing plus several fix or acceptance-test rounds.
A second opinion: re-grading with Haiku
The grades came from Sonnet 5.5. To see whether a cheaper model can do the grading itself, three independent Haiku 5.5 raters got the same brief and inputs. The Sonnet grades were locked away from them, and we audited their transcripts afterwards; they were clean.
Taking the median of the three Haiku raters against the published tiers:
| Measure | Result |
|---|---|
| Same tier | 66% |
| Within one tier | 100% |
| Quadratic weighted kappa | 0.83 |
| Hours rank correlation with Sonnet (Spearman) | 0.79 to 0.92 |
| T5 items (Sonnet vs Haiku) | 12 vs 3 |
Every miss was one tier lower. Haiku put hours at about 0.5 to 0.67 times Sonnet's estimates, so the 100-hour-plus T5 items fell to T4. The T1 set matched, 6 of 6. Haiku can rank the work; its hours would need scaling by roughly 1.6, or a Sonnet check on T4 and T5 calls.
Benchmarks behind the lines
METR measures how long a task can be before a model fails. Artificial Analysis measures how capable a model is per attempt. METR had not measured any Claude 5 model as of 13 Oct 2026, so the tier lines use the newest models it did measure.
| METR model | 50% horizon | 80% horizon |
|---|---|---|
| Claude Mythos Preview (Apr 2026) | 17.4 h | 3.1 h |
| Claude Opus 4.6 | 12.0 h | 70 min |
| Gemini 3.1 Pro | 6.4 h | 90 min |
| GPT-5.2 | 5.9 h | 66 min |
| GPT-5.3-Codex | 5.8 h | 55 min |
| GPT-5.4 | 5.7 h | 54 min |
| Claude Opus 4.5 | 4.9 h | 49 min |
| Gemini 3 Pro | 3.7 h | 54 min |
| GPT-5 | 3.4 h | 38 min |
| Claude Opus 4 / 4.1 | 100 min | 20 to 24 min |
| Claude 3.7 Sonnet | 60 min | 12 min |
| Any Claude 5 model | not measured | not measured |
METR-Horizon-v1.1, page last updated 8 Sep 2026 and no longer actively updated. Doubling time 129 days since 2023 (CI 104 to 158). Mythos Preview's 50% CI is 8.5 to 55 h. Only 5 of 228 tasks run 16 h or more. Source: metr.org/time-horizons. METR's Opus 5.5 pre-deployment note (22 Sep 2026) is qualitative only: an incremental improvement over Fable 5.1.
| Artificial Analysis model | Index (max effort) | Lower effort |
|---|---|---|
| Opus 5.5 | 58 | xhigh 56 · high 54 · medium 51 · low 42 |
| Sonnet 5.5 | 56 | xhigh 52 · high 47 |
| Muse Wonder Full | 56 | |
| Fable 5.1 | 53 | |
| GPT-6 Astra | 53 | |
| Gemini 4 Argon | 53 | |
| GPT-6.1 Sol | 52 | |
| Grok 4.7 | 46 | |
| Haiku 5.5 | 43 | xhigh 41 · high 38 · medium 34 · low 29 |
| GPT-5.6 Terra | 42 |
Intelligence Index v4.3, read 13 Oct 2026. Haiku 5.5 at max effort (43) matches Opus 5.5 at low effort (42). The Fable 5.1 entry shows 53; a changelog line mentions 66, which we could not reconcile, so we used 53. Source: artificialanalysis.ai.
How this was graded
We read 164 feature and bug folders: status timelines, plans, retrospectives and release notes. 134 had shipped to production, been published, or landed as tools in use. The other 30 were research, waiting at a gate, or unfinished; 28 of those are the solid dots in the strip chart.
Five Sonnet 5.5 graders estimated human-expert hours per item from what was built, in the METR style, and tagged difficulty drivers. Hours are estimates, mostly of medium confidence. One rule then replaced the graders' own tier calls, which differed between batches; 23 items moved by one tier.
METR's horizons come from its own task suite: self-contained software and research tasks with automatic scoring. Our tasks also include release tails, vendor consoles and desktop acceptance testing, which METR does not cover, so real horizons on our work are likely shorter than METR's.
No Claude 5 model has a METR measurement. Extending METR's 129-day doubling from Mythos Preview (April 2026) gives about 48 h at 50% and 9 h at 80% for October 2026. That is a trend line, not a measurement. If it holds, every tier boundary moves up about one band.
Some items overlap. Seven onboarding-tour entries are one programme, two back-navigation bugs shared one PR, and one release entry is a bundle of fixes.
Where the founder supplied a credential or an approval (a minted token, a vendor key, an account login), the step could have been provisioned in advance, so it does not raise the tier on its own.
Spend comes from priced agent transcripts. The first version of this grading priced Sonnet 5.5 cache reads at $0.20 per million tokens; the correct price is $0.10, which made Sonnet-heavy items read up to about a third high. Every spend figure above was recomputed from the transcripts' token counts at the corrected price. The correction changed one finding: Sonnet 5.5 no longer carries the largest share of spend in T3 and T4, where it is now roughly level with Opus 5.5.
Related: Dark Factory Patterns: A Developer's Guide, the patterns these runs use.