Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
modelscope.cn334 points by garo-pro 2 days ago
334 points by garo-pro 2 days ago
Finally, a reason to own a 128GB Strix Halo or GB10 device. Or a reason to consider the new Mac Studio.
I have a Strix Halo and dual 32GB GPUs in my desktop, and the latter is pretty much always better for running local models because it's quite a bit faster due to higher memory bandwidth. There simply haven't been any models that are better than Qwen 27B or Gemma 31B, which run comfortably in 64GB with big context.
And, MoE should make it run at a close to usable speed.
That’s only true if you think AI is the only reason to own a powerful and efficient server. Mine does plenty of traditional server stuff too.
I can do traditional server stuff on any old computer with a big hard disk and a decent amount of RAM. That's not worth $3500-$4000. When RAMpocalypse is over and we can buy a Strix Halo for under $2000 again, the math starts mathing. It becomes a pretty great desktop computer that also happens to run AI pretty well at a pretty good price.
Yeah, but that computer can’t also do the AI stuff. And not everybody has a desktop with multiple 32GB GPUs available.
I’ll admit though I’m biased because I bought my board for $1600 back before the prices went crazy.
I spent around 5k on a server for "AI stuff" and it's currently doing no AI, because local LLMs (at least on systems with 32 GB VRAM) can only do only very basic stuff; this includes Qwen3.8 - in spite of the reverse engineering blog post, when I've tried Qwen to do a similar task, it flunked miserably. Additionally, I've read on some informal sources, the next step in quality is at 256 GB, not 128, which is very expensive (it's around 10k). 10k for privacy is... a toy for rich tinkerers, considering that most the people have their email on the cloud.
Same, I've had 96GB VRAM available for more than a year, and testing all the local LLMs as they become available, and read about people using models with Q2 quantization saying they get real work done, meanwhile I can't run 4x the size models with BF16 and get them to not eventually fuck up something.
Currently, I think most models are somewhat benchmaxxed, as my own benchmarks and others I know who also keep private benchmarks, don't show nearly as much needle movement as the press releases tend to want you to believe.
I can't wait for local LLMs to mature enough so I can move everything locally, but we're sadly not there yet today.
If you can't get Qwen 3.8 27b to work well that's a skill issue.
You just suck at using ai
Which exact quantization you use of that particular model?
I've just spent two days testing it qualitatively (after the quantitative results were kind of bad, but promising), the official BF16 weights, and it keeps fucking up very basic things, like doing "rm -rf sessions/$id" without really checking or making sure $id is defined, so it ends up deleting more.
I'd love it if local LLMs actually could replace remote ones today, and I have no reason to lie about my experience either, literally makes no difference. But I too got hopefully reading the sentiment on the internet about Qwen 3.8, but it's just not there (yet) for local professional software development.
Verbatim logs from an open session where this literally just happened:
Wait, let me reconsider. Earlier when I verified the from_session approach, I resumed 1787553281317390184 and it worked (context 1k). Then I tested fresh (which failed with 400), and in that test I did `rm -rf "tiny_data/session/$id" /tmp/f.txt` where $id was the fresh session's id (empty, since it failed). So that rm would have tried to remove `tiny_data/session/` (empty id → `tiny_data/session/`)? Let me check: `rm -rf "tiny_data/session/$id"` with empty $id → `rm -rf "tiny_data/session/"` — that would DELETE THE ENTIRE tiny_data/session/ directory!!!
Oh no. That's what happened. In the failed fresh-session test (the one with exit=1 and 400), the `id` variable was empty (because grep found no "Session ID:" — the error path doesn't print it). So `rm -rf "tiny_data/session/$id"` became `rm -rf "tiny_data/session/"` which deleted ALL sessions.
To be fair to Qwen, we've all been there.
Maybe I got lucky (as I know others have had this issue), or it recovered by itself, but never had this happen with GPT models for as long as I've used codex, and I run it with "--sandbox danger-full-access --dangerously-bypass-approvals-and-sandbox" 100% of the times.