What went wrong: gpt-5.6-sol on Price Increase

Auto-generated from the run's own record. Every quote is verbatim, with the business day it came from — nothing is reconstructed. The company works in German, so its decisions appear unedited.

What happened

gpt-5.6-sol (effort: xhigh) ran the Price Increase scenario across business days 194–197 and scored 70/100 — the lowest score any model recorded on this scenario. The do-nothing floor for this scenario is 65, so this run cleared it by 5 point(s).

The run produced 58 decision(s), 1 deviation(s) and 12 technical failure(s).

Score components (0–100 each): outcomes 100 · crisis coverage 57 · discipline 0 · fidelity —.

Where it broke

1 scripted crisis/crises were never addressed (the scoring block searches every decision for them):

• Wert- statt Rabattargumentation (weight 3)

1 deviation(s) — actions the engine refused or flagged:

• Day 196, june: role 'csm' may not write package 'engineering'

What a human operator would have done

This section is not an opinion about the model. It restates the scenario's own success criteria — the things its author wrote down before any model ran it.

• Wert- statt Rabattargumentation — a competent operator would have named this in writing on the day it surfaced. The run never did.

Methodology note

Every model faces the same seed, the same crisis script and the same starting commit; only the model changes. The score weights business outcomes, scripted-crisis coverage, rule discipline and factual fidelity — a single policy violation can cap the total. This run is preiserh-hung-codex; it is git-versioned and replayable decision by decision.

Scores are only comparable within one scenario, and always against that scenario's do-nothing floor. These are controlled simulations of a fictional company: the correct claim is "in Firmulate's crisis simulation, …", not a guarantee about production behaviour.

Powered by Thorsten Meyer AI — https://thorstenmeyerai.com/

← All postmortems · Full benchmark →