SlopCodeBench 10/2026 – Opus 5.5, Astra wins overall, xhigh considered harmful

Astra writes well (which is why I often like to work with it over Claude), but it tends to produce code golf that's borderline unreadable, like completely omitting line breaks. I thought I had something funky with my config, but it seems to be a known issue.

https://lucumr.pocoo.org/2026/9/7/astra-why/

[Sorry for X] https://x.com/tenobrus/status/2096636223414808719

The generated code certainly does what's asked of it, but it's as sloppy as anything else I've seen. I don't think this benchmark can control for that. Maybe if it had a metric for minimum diff to satisfy the new requirement with the existing constraints on dead code etc?

Is there a reasonable way to run a representative subset myself?

Edit:

> For folks who want to try to repro these findings: I used the pinned v1.0 catalog, the stock just-solve prompt, seed 42, the any-case runner pass policy, and Docker with Python 3.12 and uv.

And https://github.com/gabeorlanski/scb-problems/

For most things I find medium to be the highest quality output. However if I want it to do spatial reasoning want it be funny / write something (not code) of almost-passable quality, then xhigh is where it's at.

First of all, what happened to Max and Ultra?

Secondly, I’m mostly using Astra Ultra or Sol 6.1 when tight on quota. I pay $200/month and rarely if ever run out. Am I doing something wrong? Or perhaps I’m not that productive as you guys?

Related. Others?

Benchmarking Opus 5 on SlopCodeBench - https://news.ycombinator.com/item?id=49076391 - July 2026 (119 comments)

Great benchmark!

I believe effort can mean both "solve harder problems" and "produce more stuff". I mostly don't want more stuff than I asked for, so a capable model at lower effort can indeed be the wiser choice. Faster response too.

I wish a company named human layer won't write their reports with so much Claudisms.

The irony is the article is very sloppy. Can we at least write it and review it properly..

[dead]