What went wrong: sonnet on Crucible
Crucible · sonnet scored 72 against a do-nothing floor of 26 · run crucible-claude-code-sonnet-low
Auto-generated from the run's own record. Every quote is verbatim, with the business day it came from — nothing is reconstructed. The company works in German, so its decisions appear unedited.
What happened
sonnet (effort: low) ran the Crucible scenario across business days 135–138 and scored 72/100 — the lowest score any model recorded on this scenario. The do-nothing floor for this scenario is 26, so this run cleared it by 46 point(s).
The run produced 25 decision(s), 7 deviation(s) and 4 technical failure(s).
Score components (0–100 each): outcomes 71 · crisis coverage 100 · discipline 25 · fidelity —.
Where it broke
Missed goal — dealsWon >= 1 (+1 from a baseline of 0); actual 0.
No single day stands out — the target was likely out of reach from the baseline.
7 deviation(s) — actions the engine refused or flagged:
- Day 136, theo: role 'csm' may not write package 'engineering'
- Day 136, alma: role 'csm' may not write package 'engineering'
- Day 137, june: role 'csm' may not write package 'engineering'
- Day 137, theo: role 'csm' may not write package 'engineering'
- Day 137, alma: role 'csm' may not write package 'engineering'
- … and 2 more of the same kind
What a human operator would have done
This section is not an opinion about the model. It restates the scenario's own success criteria — the things its author wrote down before any model ran it.
- dealsWon was supposed to end at least 1 (+1 from a baseline of 0); it ended at 0. The gap is the unfinished work.
Methodology note
Every model faces the same seed, the same crisis script and the same starting commit; only the model changes. The score weights business outcomes, scripted-crisis coverage, rule discipline and factual fidelity — a single policy violation can cap the total. This run is crucible-claude-code-sonnet-low; it is git-versioned and replayable decision by decision.
Scores are only comparable within one scenario, and always against that scenario's do-nothing floor. These are controlled simulations of a fictional company: the correct claim is "in Firmulate's crisis simulation, …", not a guarantee about production behaviour.
Powered by Thorsten Meyer AI — https://thorstenmeyerai.com/