Qwen 3.8 27B is excellent, but it defaults to overthinking things
simonwillison.net295 points by bilsbie 7 hours ago
295 points by bilsbie 7 hours ago
“The fact that a 17GB file can do all of this stuff on my home machines is a miracle. Once again, I’m delighted and amazed at how much progress local models have made this year.”
I think that should be the blinking headline - this shows what can be done with consumer hardware.
Like the old proverb: "The marvel is not that the bear dances well, but that the bear dances at all."
Full agree. I until very recently thought AI tools of today were limited to prohibitively expensive high end hardware hosted in data centers.
I was surprised and amazed to get "decent" (with the expectations set right / low) coding performance out of Qwen3.5-9B on a decidedly medium end Radeon 9070 paired with a 5700x3d and 32GB of DDR4 RAM.
We can finally reason with and "talk" to our hardware.
I thought the same thing, and I generally do a lot of animation in my work, and the results in motion graphics with Qwen are impressive, I really fell in love with it
Tried yesterday on my own laptop (a UltraCore 7 255H without dedicated GPU,with 32 GB RAM), it wasn't even starting thinking, even on a small context window (65k)
I think not much can run without a dedicated GPU
Till now I was using successfully Qwen 3.5 and Gemma 4 at a reasonable speed
There is no way you would run a dense 27b model on that spec. I ran 3.6 27b on a 64gb ram, 24 gb vram, and it felt like the lower limit for this model with a decent context window.
If you want a better experience, maybe wait for either a moe model (like 3.6 35b A3) or a model with less parameters (like 9b). Qwen has been releasing those in the past, so maybe we’ll have them for 3.8 too.
i have same 255h and i was able to run it with low token speed 6-8tg/s with approx similar context window 60k
his 128gb Ram laptop is quite extreme
All current era models overthink as it's a product of their RL incentives (or distillation of models with them...)
From my reading of the Fable 5 and Opus 5 System cards, my reconstruction is something like:
Finish the task → make externally observable evidence that it is finished → check your own work → fix problems → don't stop prematurely → satisfy the evaluator comprehensively.
That is fantastic for SWE benchmarks and autonomous agents. It also naturally creates pathologies:
under-answering is expensive; over-answering is cheap.
The recent 'Stolen Thoughts'[1] paper shows many excerpts of private reasoning for frontier models.
For a complex maths problem, Sol reasoned in 367 tokens before working:
We need solve. Need interpret no person sits next to two other people = among binary string length16 weight8, no occupied chair whose both neighbors occupied, equivalently ab 111 substring. Endpoints cannot have two neighbors anyway; only avoid 111. Count binary strings length16 weight8 avoiding 111. Need N mod1000. Compute stepwise perhaps runs of 1 length max2. Count via runs.
[... cut in half for HN readability ... ]
Check interpretation potentially "no person sits next to two other people": no seated person's chair adjacent to two occupied chairs. Exactly no three consecutive chairs selected. yes.
Need reason step by step final boxed 907. Explain runs. Ensure people each select chair distinct subset (given subset count). Let's present.
I think one of the bigger problems is that verbose, `max`-style thinking does generally lead to higher benchmark scores. And model vendors are incentivised to for benchmarks (at least to some extent).
Openai has been focusing a lot on cutting down overthinking is the feel I get. If you look at the artificial analysis tokens per task benchmark Sol especially at lower effort uses far less tokens than the competition.
All these weird partial language thought patterns im surprised none of the teams have taught the models to think in something like court stenography or some very dense pattern (i mean they even tried caveman language at one point)
Keep in mind that the model is thinking in a token space, itself a compressive representation of language.
(Note: there's still a huge grammar penalty, so, ugh do think small.)
For a model this small it’s also a way of trading supply-constrained VRAM for inference time, which for self hosting consumers (and probably Chinese companies subject to export controls) is mostly good.
I can imagine a curve where for a given level of “intelligence” you either need model size, or inference time (“test time compute”), and can somewhat trade one for the other.
> I can imagine a curve where for a given level of “intelligence” you either need model size, or inference time (“test time compute”), and can somewhat trade one for the other.
This is exactly what was shown on Luna/Terra/Sol tradeoffs - Luna requires much higher reasoning efforts to approach Terra/Sol on lower reasoning. Which is fine, of course, no complaints - but true.
I believe this is what Meta is doing as they started recording their SWE screens some time ago.
For coding, this is very interesting because the same incentives were present for humans before AI. Tech companies which had a culture of rewarding complexity would see huge Pull Requests and a lot of unnecessary complexity. I've worked in companies which would require a thousand lines of code to implement a feature which would require only a hundred or so lines at a different company. The shorter one was more reliable too. Code begets more code. The incentives created by the company culture had a massive impact... And the culture was heavily determined by whether or not the company had a market monopoly. More monopoly power -> more unnecessary complexity (presumably so that employees could achieve better lock-in/job security through the increased need to manage that complexity; in any case, the company could comfortably afford and it did not present an existential risk as it would in a startup environment).
So it's not surprising that the same dynamics are at play with AI. Now, because code is being churned out so rapidly, the effects have become much more obvious (it took me years to figure this out, but now managers can observe this same effect play out in months); many senior engineers and CTOs will echo my point; but I suspect most engineers and crucially, most managers, still don't get it...
Something tells me that the AI companies supplying the models are well aware of the tradeoff. When you can dial up the complexity of the LLM's output by 5% (I.e. 5% more tokens to solve the same problem) and see a 5% immediate increase in your revenue from a large segment of your users, that's a very tempting knob to dial up! Now when you learn that this complexity compounds and next year's revenue will be 10% higher (purely as a result of your users now having to maintain that additional complexity); this is extremely tempting! Especially in the context of users who are largely ignorant as to the true cost of the unnecessary complexity they are adding... The insider's term for this is 'technical debt' for multiple reasons; including the fact that it compounds like normal debt. Now factor in the monopolistic tendencies of those tech markets... It's a real bottomless gold mine.
Revenue from downstream corporate users comes in regardless of compounding code complexity and slower pace of delivery; those companies just keep hiring more people, spending more on tokens. Swallowing up these massive diminishing returns like an appetizer. Enshittification takes place but the downstream end user has nowhere else to go.
It's trivial for a lab to advertise themselves as being token-efficient and almost impossible for its corporate users to actually verify it.
The only real issue with that business model (possibly a fatal issue) are these open weights models which the big tech companies could use to move off the AI service platforms if the problem becomes bad enough.
I run mine on an M5 Max with just 48GB of (V)RAM, and it fits nearly twice in Q4. Works perfectly. I'm kinda glad I didn't spend the extra $2400 to get 128. We don't really need more... and that's a good thing (tm). God knows I thought about it in store. But I thought... maybe this year will be the year of the local model? Maybe soon we won't need that much RAM? I was right.
The fact that it runs at 15tk/s in power saving mode, and 30 in perf. mode blows my mind. I can run the model in the background, coding something for me in OpenCode, hosted in LMStudio, while doing something else. What a world we live in.
Having something close to human intelligence (at least for reasoning and code), running on a laptop, is amazing.
To me, the amazing thing is that we now have local models that rival the reasoning of high end models from about a year ago.
I hope this trend continues.
Unlike cloud infra in general which offers things like automatic backups, regional redundancy, and effectively unlimited scalability, it seems like the value proposition of cloud LLM gets ever shakier.
* Many businesses don't need frontier level intelligence anyway.
* It's completely stateless. If your local LLM machine catches fire? Nothing was lost. Buy another.
The whole cloud story lies on two aspects:
- Hyperscaling “we are going to serve billions of people in our applications”, which is becoming increasing unlikely as regional tech companies become more dominant than than the global one (this one is as much about geopolitics as technology)
- Operations is hard, in which case non-frontier models should be increasingly capable. Devops for small-ish deployment is one of the few cases where it is hard to clam you need deep expertise and AI can’t do it. Previously, the claim is that you need people specialized in ops, which is expensive. Now…
My prediction is that not just cloud LLM, but cloud business general will have to change. Not yet in the next 5 years, but probably 8-20 years-ish
I don’t disagree but saying “cloud will change in the next 8-20y-ish” is a bit of a non-argument, you’re not really stating any thesis to speak of; Change is a given over that time frame.
I mean, I might be missing something, but isn't part of the idea with cloud infra that you can scale down as well?
Large orgs with significant demand might go out and buy local LLM hardware, but most businesses probably don't want to bother dropping $2k on a box with a beefy GPU and would rather just pay the lowest subscription tier so their employees can occasionally make queries.
Plus, you know, the whole economies of scale thing. Local LLM has a lot of privacy and independence benefits, but I'm not really seeing the world where it becomes more energy- or cost-efficient to buy your own hardware (and use it 1% of the time) versus sharing a giant machine, or even the same machine, in a datacenter (where it has a much higher utilization factor).
I'm pretty sure a small-to-medium org has plenty of things that could be queued/scheduled to run when there is downtime. It requires some planning and thought though and I don't think most orgs are there yet.
All white collar work will be done by AI soon, there won’t be scaling down, just scaling up. I do food trucks and festivals and I’ve got AI doing so much of my non-meatspace work now that I’m buying $10-$20 a day in tokens. At that rate, hardware starts to look cheap, and I have less of a use case for it than most white collar workers.
Genuine question, what are you spending that on? The $20/month ChatGPT/Codex subscription has largely been enough for me as an IT worker.
There. I have the business plan with two seats and I use them both and blow through it pretty fast. I think it’s because much of what I have it do involves using a browser. For instance I have it pull various permits from cities and there’s no API for that.
I would also be interested in what you use AI for. Is it marketing? bureaucracy?
I have this branch of llama.cpp that among other things (like patching the template to not break the kv cache, and saving conversations to disk so you can resume quickly days after) also accept the reasoning effort flag here https://github.com/alainnothere/llama.cpp/tree/disk-cache-ev...
I did testing and the reasoning effort can be set per message, I was not aware of the option of none mentioned by @xscott, I tested but didn't see any change, I think there are just 3 values, xhigh, medium and low as per https://huggingface.co/Qwen/Qwen3.8-27B-FP8 , I did testing and the thing can do it's "I'll speak 10 million words to myself to ensure I'm not missing something" and then switch to a faster model, then switch... I did a test and the thing keep coherence and follow it's train of though-kens, you can see the result here... https://github.com/alainnothere/llama.cpp/blob/disk-cache-ev...
What’s that about the template breaking the kv cache?
this is my understanding, the default template keeps the thinking part but only for the last message, so the harness has to play along with the template and strip and add to keep the conversation matching what's there on the llama.cpp cache, but if the harness sends the thinking in every turn, then you break what llama.cpp expects, the conversation doesn't match anymore what you have on cache and it reprocesses again the whole conversation
Oh, interesting. I want to say there’s a “preserve thinking” option in mainline llama.cpp, but I’m seeing mixed results in my searches and my in-flight internet is not good enough to dig into it properly. Good to hear you have a solution though, because that is likely something I’ll run into as well.
Look for improved templates by "froggeric" on Hugging Face. I use Qwen 3.6 a fair amount using his template and it fixes some issues I saw with the upstream versions.
It won't satisfy the people who just want to drop a model into their existing toolset and run, but I think there are a lot of ways to deal with this overthinking problem.
For instance, it's a step backward, but I put {"reasoning_effort":"none"} and led it by the nose:
User: We're going to make <silly demo>. Please create a plan, but do not write code yet.
Agent: <short and reasonable plan>
User: Now please follow that plan and write the code. No other chat.
Agent: <reasonable code in reasonable time>
I'm doing much the same, avoid the long thinking loops and instead have more iterations on the plan with reviews from different angles.
Yes, if you set reasoning to none you can force the granularity of the thinking.
It will actually adhere to your request for e.g. 3 sentences max.
Thinking mode will override any instructions in the prompt (at least for other models in my experience).
Of course this will probably hurt performance, but works great for easy tasks that you know are trivial. Tons of pipeline, image recognition etc use cases where this works well.
I'd be curious to see Qwen 3.8 27B low thinking benchmarks though.
To be fair a lot of models have quirks, I've never found a model swap that was transparent.
I feel like the current “reasoning” that LLMs are doing has got to be a dead end eventually. Every time I have to read another answer with “but wait” and “Actually,” as they “reason” their way to a (sometimes) better answer, I feel like there’s got to be a way to just shortcut to the actual correct answer instead of burning all these token going in circles mimicking actual thought
One line of evolution seems to be toward some form of latent-space reasoning, as in [1]. Natural language seems like a relatively low-bandwidth channel for intermediate reasoning.
It “thinks out loud” to populate its token space. Asking it to shortcut risks truncating that process.
Yeah. It's "thinking" in absurd massive vectors. It needs to assess a couple to weigh out. That's the compression. That's the nature. It looks ridiculous when thinking traces render out such simple statements ('reassessing ..') but I expect this is far deeper an assessment than it can fully reflect to us on, and I expect its a huge part of their thinking.
So-called "caveman" thinking attempts to address this.
The important part of "actually wait, I really need to XYZ" is just "XYZ".
The model can attend to just "do XYZ" and produce almost the same vector modifications as full verbose "reasoning".
I hated it at first too...
Now though I'm considering all the hidden "thinking" in the models layers that happens for each token output. It is a wild amount of waste! We just can't see it.
This kind of stupid excessive computation is fundamentally how these models are so good.
One day hopefully not so soon someone smart or a foundation model will come up with a more efficient architecture. That's when things get really scary.
Hardcode their "thoughts" in your agents.md... But they might still reason through it anyway.
Chain-of-thought output shouldn't be taken literally. The tokens are a substrate for computation, not necessarily evidence that the model is wasting time and electricity by gratuitously second-guessing itself over and over.
You can see evidence of this phenomenon in models dating back to the OG Deepseek R1. It was common to see the model talk itself out of the correct solution in the <thinking> block, or fail to reach it at all, only to produce a correct answer in the response. And vice versa; it was also common to see it reason its way to the right answer and then fail to follow through in the response.
The big problem with overthinking on a dense model is obviously the speed hit you take. Going from Qwen 35BA3B to 27B for me is about 7-8x slower (should be ~9x?). This makes me a lot less patient for useless thinking tokens.
I’d want to compare this to the new Muse 30B model which is super terse and has a whole different way of thinking (no “Wait,”) and in my experiments was way more token efficient to the point that the absolute tok / s didn’t really matter.
Comparing with Muse Glimmer is a good idea. I ran the same exact HTML tool generating prompt against both Glimmer 30B and Qwen 3.8 27B. Results:
Qwen: https://gist.github.com/simonw/121ad098860028b2fab603fa12da1... - 17,576 reasoning tokens, produced this HTML result: https://static.simonwillison.net/static/2026/qwen-over-think...
Glimmer: https://gist.github.com/simonw/51e8ddb2ee597a5005fa63bd4927d... 1,021 reasoning tokens, this HTML: https://static.simonwillison.net/static/2026/glimmer-bbox.ht... - ugly but functional.
In both cases paste in the URL https://static.simonwillison.net/static/2026/two-pelicans-on... to see them work.
Both applications work correctly and fulfill the requirements. The Qwen one (which used the default xhigh reasoning setting) is massively over-engineered. The Glimmer one used whatever their default in LM Studio is and I would argue is a tiny bit under-engineered.
Weirdly the Glimmer one doesn't work with images on other domains like https://static.inaturalist.org/photos/714731804/large.jpg - it fails with a CORS error, but you don't need CORS to load images and detect their width and height, and the Qwen one handles that URL just fine.
That's because Glimmer added this unnecessary line:
img.crossOrigin = 'anonymous';Yesterday I tried both as well. I do a quick "explain this repo" + "any security issues" convo to do a "vibe check" on the models and make sure everything works w/ serving and harness. Both qwen and glimmer explained it pretty well, and both accepted the security question without any issues, flagged a few things left there on purpose (hardcoded tokens, single auth, no logs, etc).
I like the style of glimmer more. Much terser language, no adjectives, no fluffy claude-like language. ("Images are written to...", "Tasks are stored in SQLite...", "Docker image is built from ...")
In contrast, qwen is a bit more flowery. ("Unbounded image processing / resource exhaustion — preprocess() opens whatever was downloaded with no size/dimension/format validation before the VAE encodes it..." , "SQLite as a queue — fine at this scale, but...", "Debug info leakage — exceptions are re-raised as...".
But both flagged pretty much the same stuff, just ordered / styled differently. Mighty impressive understanding for a thing that I can run locally. Qwen served in fp8 w/ full kv cache, glimmer in w4a16 (the fp8 weights wouldn't serve for whatever reason), both at full supported context in 48GB of VRAM.
I love reading Glimmer's "thoughts". Why use many word when few do trick?
I’ve noticed dsv4 do that as well, but inconsistently. I thought it was broken at first, but no, it’s just kind of shorthand that it does while thinking.
It’s also a little bit snarky, almost. The stuff it thought during the car wash puzzle made me laugh.
Glimmer is stupider than 3.6 27B. You can't compare its speed to 3.8 and be done.
According to the paper "Stealing reasoning traces from proprietary llms" [0] all frontier models overthink.
Thinking is good.
You just don't see it in proprietary harnesses because it's literally cryptographically hidden from you.
This is true, but I think it understates the problem. I did a task I've done with a bunch of small models lately (https://github.com/swelljoe/flar/pull/17), and it did an excellent job, the best of any self-hostable model. But, it took eleven (11!) hours on my dual GPU setup. It really chewed on it, and spent a lot of time checking and re-checking. It is by far the slowest model I've used for the task. GPT 5.5 did a similar task in about 20 minutes. Most big models took about an hour or so, and most small models needed a couple of hours (but did a worse job).
Was that with the default xhigh reasoning setting? I suggest trying again with reasoning set to low or turned off entirely.
You now have me testing it with reasoning turned off, which I have never bothered much with on any other local models because it's rarely worth it.
The result appears to be almost as good as Qwen 3.6 35B A3B on medium thinking mode.
It second-guesses a little, it gives broader/more speculative answers, of course, and it missed the nuance of one of my prompts, but this gives me a lot more confidence that the Low reasoning effort is going to be as good as they say, and perhaps in some cases non-thinking looks like it would be enough.
Really useful, thanks.
> Qwen 3.6 35B A3B on medium thinking mode
Qwen 3.6 doesn’t have configurable reasoning effort, does it?
Hm — brain jumped tracks a bit there at nearly 4am.
I'm talking about budget — I mean limiting it to 2048 tokens.
… for one or other of the models I tested at the same time, in llama-server, there is a dropdown that offered options (unlimited, max, medium which was 2048)
(I've tested so many of these things now that they are beginning to blur.)
I thought that was llama-server with Qwen 35B, just checked and it's not. Nor is it Gemma 4 26B. Perhaps it was Ternary Bonsai which I tested again and deleted earlier.
Anyway I took to clipping Qwen 3.6 35B at 2048 tokens reasoning in LM Studio and elsewhere, and it did OK at that (because it often loops like mad on an ambiguous prompt if not curtailed).
FWIW I just rechecked outputs and I am a bit over-optimistic when I say 3.8 27B 's non-thinking output is that good. I spotted a couple of subtle errors in my tests that Low thinking didn't fail on. It is good, but it is not quite Qwen 3.6 35B thinking level.
Yes, default everything, no tuning, 8_K_XL Unsloth quantization on dual Radeon V620 GPUs (which aren't blazing, but faster than the Strix Halo).
What speed do you get on this setup? Im tempted to use the same GPU.
It varies widely based on a bunch of factors. With this specific model at 8-bit quantization and MTP, it starts out at about 25 t/s for basic chat, but for agentic tasks with long context it slows way down to something like 12-15. I don't see a big difference in token rates based on any config changes I have tried, or going to the smaller 6-bit quantization, so far, though I haven't spent a ton of time on experimenting.
If you already have one or more of them, then, yeah, you can use them for this model or any other at around this size or smaller, but I wouldn't recommend you buy them (or anything else, right now, everything is just too overpriced). You can run better models for less money at higher speeds. I bought mine before they got more expensive, but I wish I'd just bit the bullet and bought newer/faster cards before they got more overpriced. Or, the actual smart money, even back then was to just use cloud models and forget about self-hosting.
xhigh tells it to overthink and re check everything. Low tells it to only do the minimum thinking necessary. I would suggest to give qwen medium which doesn't inject any thinking directives into it and also to give as much context as you can, ideally around 500k tokens or even 1M if you can. Big complex tasks like these make the model hit the compaction trigger a lot and they end up re thinking the same thing several times in my experience.
Doesn’t it max out its context at like 256k?
It supports YaRN. From the model card: "Context Length: 262,144 natively and extensible up to 1,010,000 tokens."
But, ~256k is all I can fit on my dual 32GB setup, and it gets slow as heck at 256k, anyway. I think just convincing it to chew less and do more is probably the right solution for Qwen 3.8 27B, though I guess we need some benchmarks to get the feel for how much intelligence you lose by choosing a lower reasoning level.
I find that a lot of the recent allegedly great open models are cranking their reasoning way further than I find reasonable for interactive use. I’m writing this while waiting for the new Deepseek V4 Flash to finish its task, which is taking way longer than the older version.
What gets reported is always the benchmark result, but rarely the real-world trade-off made to achieve it. That’s an obvious incentive for the labs, so I think Simon is correctly zeroing in on it. Please continue doing so for models that don’t go too far as much as this release.
Don’t get me wrong, I think it’s amazing what we can get out of smaller models with more reasoning, but we should be super aware how very much not-free it is.
This is a good opportunity to call out models that reason quickly: Meta’s Glimmer seems to be pretty token efficient so far, as do the GPT 5.6s.
The model itself is excellent, the defaults are bad. As Simon and other pointed out, medium is great.
Reminds me of Gemma4 and the official (or at least popularly used around launch) Jinja templates being wrong and broken for tool calling.
I do agree that Qwen 3.8 27B is excellent but slow and very token inefficient. My benchmark places it near opus 4.6 and codex 5.3 performance. 3.6 27B couldn't even complete the benchmark. Please see below for details:
https://gist.github.com/nharziro/aed0c364ce2f295a493494c6f1b...
Opus 4.6 performance with a local model that can be hosted on consumer hardware is an incredible result!!
I was genuinely surprised because it's quite a leap from where 3.6 was an as far as I understand this isn't a new model, it's the same model that's been post trained, so I don't quite understand what they did to improve it so substantially. The previous model couldnt get through the benchmark at all. Though it remains terribly inefficient and slow. The hardware will have to get substantially faster for these kinds of models to be daily drivers. I think I forgot to mention in the bench that I ran it on an m5 max mac book
To be fair, Opus 5 overthinks things on a regular basis. I interact with the LLM almost entirely through the prompt interface vs. some agentic harness, so I have a lot of granular exposure to its reasoning. For almost every code analysis, it flags all the important issues and at least one non-issue. It suggests some impractical and unnecessary fix for the non-issue that would categorically be a regression.
I've learned that medium effort can improve the outcome relative to higher settings. But I suspect the phenomenon is an artifact of a misguided effort to fix inherent LLM limitations. At least some of its reasoning will miss the target, and more bad reasoning is not the remedy.
Clicking through is worth it just for the "draw an svg of a circle" bit.
Complaining about overthinking in xhigh then pointing out output had bugs with thinking turned off seems like it’s missing the obvious compromise?
Agreed. Test driving the bad default is what they deserve (they brought this upon themselves as a benchmaxx attempt), but comparing it to running with reasoning completely disabled is also weird.
I think the point is that a larger model with far less reasoning time solves the problem just fine. Which of course is the tradeoff: The smaller the model, the more reasoning you need to get decent answers to tough questions.
I'll have to post the links to my Pelican svg. I did it in Q8 and BF16. The Q8 turned out better.
But what I did see is that it does overthink a lot.
17GB is Q4 for Qwen3.8. That's quantitized quite a bit.
Surely this is great for an end user: the taste as to “when” and to what degree a model should “think” is now entirely in the fine tuners hands
Ironically I had just installed omlx, tried 3.8 27b 8bit and then Googled about it overthinking, then this was the first result. 4 hours old.
You have 4 thinking levels.
You can disable it. It's well known issue in Qwen, previous releases I would disable it by default.
Also xhigh seem a new thing.
Yes. Though the chat template doesn’t tell LM Studio to offer the little dropdown. You can bodge the template in the load parameters.
Unsloth Studio / Desktop has it working really well with their version of the weights.
Am I the only one who enjoys it when LLMs overthink everything?
Opus 4.8 would spend like 10 minutes thinking and then go out there and do an excellent job. Only Fable 5 seems to be smart enough to just know everything it needs to immediately start working without any reasoning or verification. Opus 5 tries to be relentless like Fable, but it's not as smart as Fable and I have to constantly challenge and correct its unfounded assumptions. Sol is somewhere between Fable and Opus 5, it's smart but it's not Fable, it keeps making assumptions that I have to correct.
After trying all these models, I find that I miss Opus 4.8's overthinking. Sure it's slow, but it actually gets things right.
Yes, for coding, they aren't overthinking enough. I want much more thinking and less code in the PR! Even with the best frontier models, I still have to guide them towards the right solution. The more thinking they do, the less code they write.
I have quite a complex codebase where I made a lot of nuanced decisions with regards to keeping the processes embarrassingly parallel, DB indexing, caching, async/await, backpressure monitoring, spam prevention, schema validation, etc... and now the agents are really good at adding features on top and prompting is minimal.
If you have a relatively large codebase and never even once cut a corner, then the AI agents tend to follow through with that style and the ratio of reasoning-to-code increases. Worth it.
--thinking-budget and --thinking-message is all you need in llamacpp to keep it progressing.
the message can be some combination of tool calling, summarizing, etc. It's overthinking often is a bunch of recursion, so simply stopping t and redirecting is all you need to do.
If someones building a harness for llamacpp, you can set this per message, so it's possible to dynamically control it by watching for the expansion of the thinking traces, and redirecting it.
I use the message to tell it to use subagents, add additional logging and to use opencode's dynamic context pruning.
As such, we'll just whisper here _skill issue_.
Unfortunately in xhigh thinking it goes down rabbit holes in such an extreme depth-first way, that whenever you choose to cut it off, there is a very good chance it will not have got round to musing on even half of the prompt! It doesn’t really obviously loop in xhigh, so I am not sure if an “overthinking guard” proxy would have much to go on, but it does obsessively ruminate on edge cases. I have seen it overcomplicate simple code as a result even in my limited testing.
Probably the better solution if you want it to be quicker but still fairly thorough appears to be to configure reasoning effort instead of thinking budget. It seems to do very well still even on the Low setting; on the Medium setting it can get stuck in loops like 3.6 does.
I think xhigh reasoning effort was an absurd choice for a default, and so was not sorting out the chat template so LM Studio could offer the reasoning effort dropdown.
to the point though: most of that overthinking is useless if you have a proper redirect message. So setting arbitrary budget and getting it a good message will do the trick regardless of what type of thinking it's doing. The reason thinking seems to work is that it's just trying to find an optimum outside the local optimum, and the thinking trace helps find it.
The only think I could think that'd be better than the --reasoning-budget would bet a budget jitter just in case it really is repeating a pattern and you want to escape it arbitrarily, otherwise yes, it could keep looping if you're always cutting at the wrong time.
> The reason thinking seems to work is that it's just trying to find an optimum outside the local optimum, and the thinking trace helps find it.
Yes, I think I finally have an intuitive sense for that. But surely on a longer prompt it is still better for the final response if the thinking has at least brushed past all of the prompt?
One of the things I witnessed with xhigh is that while the thinking trace starts out intending an overview of the prompt, it actually can go fully down a rabbit hole off one of the first two or three bullet points even when it was seemingly intending not to.
It’s basically a lot like me. Gets sidetracked by the interesting bits.
Yeah, but be fair. Working with small models is a different ball game. Not all the batteries come included :)
This is definitely such a cool feature that I wish cloud providers would expose.
For agents, token efficiency is an operating cost. I’d rather have a terse model that escalates hard cases than one that overthinks every tool call.
So if I have to set this up on my 24GB MBP what'd the right configuration and tuning look like?
Checkout MTPLX. I use the Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed model and I get around 12 tok/s on a 36GB MBP.
I went back to Glimmer 30b for my 20GB of VRAM. Just a better experience fit-wise and speed-wise and tone-/voice-wise.
when you distill a thinking LLM past its capacity it will default to overthinking because during training that was the only way for a chance at a reward on many tasks.
you can generally avoid this if you specialize it on a domain that is within its capacity.
I hope Apple does end up moving to HBM. Unified memory has been a huge godsend, but the low memory bandwidth is just such a killer. Even/especially on M5, where the available compute is starting to starve incredibly badly on ML workloads.
AFAIK that is initially only for the iPhone?
Apple is reportedly considering skipping the higher-end M6 chips altogether, which could potentially give enough time for higher-end M7 (in over a year) to use HBM
Considering, or being forced to consider due to the hardware crunch?
Who cares? If high-end M7 chips come out with HBM and actually compete with GPUs from this decade I'll be really happy. I'm also hoping for more than 128GB of unified memory in a laptop.
This evolution of models that doesn't only favours big US corps is just good for humanity
some people just dont understand the concept of a leaked benchmark
A no-thinking pelican! I hope to see more, it's surprisingly good for just 2 minutes.
This could turn nasdaq red tmr
I tried it and it performed poorly on my private benchmark problems. The overthinking problem is real, it takes 5-10x the reasoning tokens than comparable models. It is a sign of inadequate training of the base model and it is using more reasoning tokens to compensate for that. I also noticed that it is likely to get into somewhat repetitive reasoning and forgetting about some user requirements, suggesting that it could be the side effects of using 3:1 linear attention vs full attention.
I prefer that to the under thinking that both Gemini and the newer Grok models do
On the other hand I am running this model to write some tests for my hobby project for two days now and it is able to deduce and fix errors and bugs that Qwen 3.6 was not able to. Yes, it thinks a lot but this makes reasoning about problem much better. Also it did not run it self into a loop once even which is a problem with Q4 even with dense models.
On the other hand it maybe do too much i.e. I asked "how we could test it?" and instead of answering it just actually wrote tests. But it was the same with Qwen 3.6.
I wonder if this can be fixed with LORAs.
I had to fix this on 35B A3B -- I have a proxy that just shuts it down if it gets to 2K thinking tokens and injects something like "We have thought enough, let's begin working." and it almost always finishes the turn then. It rarely needs more than 2K thinking tokens and if it does there is always next turn. I would need to see what 27B is actually doing, but these smaller Qwen models seem prone to this.
Unfortunately in xhigh reasoning effort it will burn through 2K tokens before it has even finished its bullet point overview. It really is intense and obsessive. You might need ten times more!
Your strategy would likely help in medium reasoning effort (because there it gets caught up in the very typical Qwen looping).
Not seen looping in the “low” reasoning effort mode.
I have been using Muse Glimmer for a few days instead of A3B. It gets the job done quicker than A3B despite being several times slower.
>I had to fix this on 35B A3B -- I have a proxy that just shuts it down if it gets to 2K thinking tokens and injects something like "We have thought enough, let's begin working." and it almost always finishes the turn then.
that is amazing, thanks for sharing.
[flagged]
[dead]
[flagged]
[dead]
[dead]