Mun logadan: Opus 5 outscores 4.7 and 4.8 on benchmarks and is worse to pair with
Mun logadan argues Opus 5 is a downgrade for sustained coding against Opus 4.7, 4.8, and Fable — not on capability, but on behavior: it assumes instead of asking, and reinterprets plans without checking. The suspected cause is benchmark and RLVR training, which rewards confident guessing on self-contained tasks and penalizes stopping for clarification. An in-house regression suite pinned across model upgrades is what catches this class of change; vendor benchmark deltas cannot surface it.
Source: mun-logadan.github.io ↗
Selecting for models that do well on benchmarks inherently selects for models that make bold, usually-correct assumptions in the face of ambiguity. It penalizes models with a tendency to stop and ask for clarification or direction.
Why this matters
- → Benchmark optimization creates models worse at real coding work—Opus 5 assumes instead of asking
- → Self-contained benchmark tasks reward confident guessing, not the clarification-seeking behavior coding needs
- → In-house regression suites across model versions catch behavioral regressions vendor benchmarks miss