Artificial Analysis v5 and the benchmarker's dilemma
interconnects.ai50 points by 6thbit 2 days ago
50 points by 6thbit 2 days ago
I wish AA benchmarked the models at the reasoning efforts people actual use them at. For example, it's very rare for people that I know to use the "max" reasoning setting - specially because Claude Code itself scares you away from using it. But all of these coding benchmarks are based on just that reasoning effort.
I get it, they can only benchmark so much. But I'd rather have a benchmark of how the model is actually used and offered to customers rather than the best case scenario for it.
Maybe you need to enable them in the filters, but AA is the best website I found with benchmarks numbers on all reasoning effort levels of recent frontier models.
I dug into it more, and it seems like the reasoning efforts are available for the Intelligence Index, but not the Coding Agents benchmark I was interested in.
The other reasoning levels are almost always benchmarked, AA just doesnt show them in the default charts.
I see them for GPT 6.1 Terra, but not Opus 5.5. I'm looking on the Coding Agents page.
"The final point I need to make, which does flatter the closed labs, is that the likes of Anthropic and OpenAI moved on from caring about these indices months, or even years, ago. Their focus is to a large majority on supporting internal uses – like AI R&D automation – and the use-cases of customers who are spending billions of dollars."
Is this true? If yes it makes a lot of sense. Time and time again, when I try some newly released open weights model I am disappointed. Even some older Claude models are beating them on quality of answers, even though they score less on intelligence in benchmarks.
> Is this true? If yes it makes a lot of sense. Time and time again, when I try some newly released open weights model I am disappointed. Even some older Claude models are beating them on quality of answers, even though they score less on intelligence in benchmarks.
Interesting. My experience is that Opus 5.5 is smarter than a model like Qwen3.8 Flash Next. But given the choice, I'd actually rather work with a model like Qwen3.8 Flash Next.
Even on medium reasoning, modern Qwen and DeepSeek models are ridiculously diligent. If I ask them a question about which library is best for a given task, they'll download 4 different candidates, read the source, and write test programs. Meanwhile Opus will look at a couple of web pages and give me glib bullshit answer. Opus is often like a brilliant coworker who can't be bothered to do their job. The modern Qwen and DeepSeek models are like someone who is merely smart, but who has an alarmingly good work ethic.
And I'm not just writing simple React CRUD GUIs. I'm doing stuff like modernizing ancient software forks, working on high throughput distributed systems, or finding bugs in other people's parser generators. And even the 120-240B open weight models are normally entirely adequate for these tasks.
One of the benchmarks where open models tend to perform badly is Terminal-Bench 4.0. I looked at some of the sample tasks on that benchmark, and some of they are really hard and specialized. Look, all I actually need out of a model is things like "tell me where this distributed algorithm might break", "get this ancient C code to build on modern Debian with modern dependencies", or "write a plan to migrate this database from MySQL to PostgreSQL and to test that the app still works." The modern open models can do all that, no problem.
It's perhaps rather expected that the compute constrained providers have lazy models, and the compute available providers are happy to let you burn a gazillion tokens.
> modern Qwen and DeepSeek
It's worth checking different behaviours across the available models from time to time. What you end up preferring may be very dependent on what you do. For example while being diligent is nice in a lot of situations, sometimes I ask a personal agent a question and mimo spends 10min validating the answer in a few different ways. Sometimes I want that big research, sometimes I just want to know what that transformers toy is called.
There's also a huge difference is writing style. I'll take GPTs version of a commit summary over Claude and its copies any time.
> If I ask them a question about which library is best for a given task, they'll download 4 different candidates, read the source, and write test programs. Meanwhile Opus will look at a couple of web pages and give me glib bullshit answer. Opus is often like a brilliant coworker who can't be bothered to do their job.
I’ve been using Qwen3.8FN and DeepSeek. I would not trust them to evaluate 4 different libraries from scratch unless I gave them extremely specific benchmarks to run. They’re very good at consuming tokens for long stretches and then giving you what looks like thoroughly researched output, but their intelligence just isn’t at a level I trust.
But really, the difference you’re describing is easily shaped by prompting. If you want Opus to download and compare the libraries, just tell it to. It will. All of the models benefit from being clear in your intentions. If you try to find models that will read your mind where spending 20 extra seconds to type out your intentions would accomplish the goal, you’re ignoring the very important intelligence and efficiency differences between models.
> Look, all I actually need out of a model is things like "tell me where this distributed algorithm might break", "get this ancient C code to build on modern Debian with modern dependencies", or "write a plan to migrate this database from MySQL to PostgreSQL and to test that the app still works." The modern open models can do all that, no problem.
I actually read the code output for projects that matter. There is a huge difference between a model that can eventually accomplish a shallow goal like “fix this bug” and a model that does it cleanly. Even with Opus or Astra I will spot cleanup issues frequently.
When using models down to Qwen3.8-FN or DS-Flash size, the code can feel like the kind of spaghetti mess that you get from an overly caffeinated and confident junior developer who accomplished the goal via brute force and staying up all night. Yeah it compiles and does the thing you asked, but all of that spaghetti code compounds with each change until a couple months later your harness is spending half a day and countless tokens doing simple tasks because it has to navigate the mess of a codebase.
That problem is much less of an issue at the Opus level and I can trust Opus to clean up better when directed to do so. It’s those differences that compound.
But if I’m doing a throwaway task in a sandbox that just needs to work for a day? I’ll throw it to the open models and see what they can do. They are interesting to play with.
I believe so, this checks out with my experience, especially when comparing between the US models and Chinese models which benchmark scores are neck to neck with US ones.
My experience with Chinese open models is that they couldn't cope at all with my production work; both GLM 5.3 and Deepseek v4 ( or was it 4.1? Couldn't remember too well) went into infinite loop and wasted my tokens until my OpenRouter wallet reached 0; good thing I didn't enable the auto topup. US models, by contrast, breezed past them.
Even for simpler tasks, Chinese models took long time to complete, and I needed to supervise closely. The price for Chinese models, in the end, didn't come cheap, mainly because too much time was wasted on thinking, and this was regardless of whether they finished the tasks or not.
If they don't care, why are them presenting them when launching a new model, every single time?
Because people, for some stupid reasons, still read and make decisions by these public benchmarks, which makes zero sense. Create your own benchmarks and evaluations, then use those, it's pretty the only way to get any qualitative answers to how well a model will work for you and your user cases.
How many users are willing to spend the time and repeated cost (once for each new potential model+version+variant) for that - 1 in hundreds? (Edit: I was thinking about this as an individual dev. Especially for AI uses in a biz workflow, custom benchmarking makes perfect sense yes.)
Probably the same amount of users who actually care about figuring out how well a model works for their use cases, instead of what amounts of guessing.
Most companies I've consulted for when it comes to LLM usage in various ways all have their own evaluations, not sure why others would avoid them.
This is to save time, for what it's worth. Doing that testing and evaluation will take you MUCH longer than doing so automatically, for obvious reasons.
> Time and time again, when I try some newly released open weights model I am disappointed
The article is correct that the major labs are not benchmaxxing, but I think it overstates their focus being on something else.
They don’t play the benchmark manipulation game because they’re comfortably in the lead and they have their own teams producing internal benchmarks and targets that are representative of what their users expect.
Your experience with the open weights models is not missing anything. Nearly all of them are disappointing if you set your expectations according to where they fall in the benchmark charts. They can be useful for some tasks once you’ve developed an intuition for their capabilities, but you have to start learning their relative cost per task and their limits, which takes a lot of observation and honestly might not be worth it.
One of my unpopular open weight model hot takes is that, right now, OpenAI’s Luna model is often cheaper per task than even DeepSeek and others (paying API prices, not based on the fluctuating subscription prices).
The real advantage of open models has always been that you can run them on your own hardware or a rented server and have more control over where your data goes. Some of the hype about them being 3 months behind frontier models or people making claims that 27B local models have replaced their Claude subscription become extremely questionable if you actually use both open weight and frontier models frequently.
I’ve been burned by this so many times that I’ve pretty much given up on open models for general use.
How does AA make money? Who are their investors?
We just need transparency. It's okay if you change criteria, but pre-register the trigger for when you're going to change them.
Example could be, we'll update X criteria when top-N models are within 2 points of each other.
AA has dramatically altered their methodology twice in just a few weeks. I really don’t see the utility in their composite scoring anymore - it’s basically the US News college rankings of LLMs.
For the time being, LiveBench seems to provide a better calibrated signal.
Bench switcherooing seems worse than bench maxxing, at least goodhart law shenanigans still has target everyone agrees on.
I’m tempted to say that ideally, we’d have more in-depth, qualitative reviews of AI models so that it’s possible to form an impression of how suitable each model would be to one’s particular use case (instead of just “this model gets you 34 intelligence per dollar; this one, 38” as if they could be reduced to such numbers), but that would be a lot of effort given the sheer number and frequency of model releases.
I personally would like more reviews of new capabilities unlocked. Like there was a lot of talk when o3 got released that it can do geolocation. Feels more interesting than being better at an existing capability.
AA's intelligence score is based on many benchmarks, you can go look at them all individually if you'd rather focus on specific areas.
Those are still numerical scores and, as far as I know, essentially based on “does it solve the task correctly or not”. By “qualitative”, I meant for example some analysis of how it succeeds (e.g. quality of prose or code) or its failure modes (lack of persistence, etc.).
I just realized benchmarks are the trading card PSA of AI.
The numbers are fuzzy and hard to verify, especially if benchmarks are private, putting authenticators in a powerful position to determine market value of AI in countless ways; a recipe for corruption. I'm not saying it's happening, I just realized the social ingredients and temptation are there.
Tangent but the fact that quen3.8 27B even places on these top rankings versus hundred B and T models is incredible.
It’s the crow of current LLMs: tiny brain, outsized performance.
Mentioned in the article: https://www.intelligence.ai
Not sure if the scores are better, but the website is nicer to use.
AA has such terrible UX, especially on mobile, trying to select a subset of models you care about is an exercise in frustration.
My own (first world) problem with AA: deep linking to the specific models + benchmarks has been hit or miss (for comparing a new model vs my standards).