Kimi K3, Qwen 3.8, and Anthropic's (Potential) Unravelling
emergingtrajectories.com330 points by cl42 15 hours ago
330 points by cl42 15 hours ago
The open weight, open architecture releases of the past several days has me more convinced that ultimately, the winner will be whoever burns their models to ASICs fastest.
The LLMs themselves are capable of doing some aspects of chip design as evinced by the K3 press release.
Furthermore, the frontier models are "good enough" for a wide swathe of tasks and will soon hit that threshold for a good amount of software engineering (if not already). Does anyone think we need a Mythos level model to plan a road trip, or give someone tips on making a cake recipe?
A Fable 5 model running at 9,000 tokens/s on an ASIC rather than 150 tokens/s on electricity chugging Nvidia GPUs, or even giant SRAM Cerebras or Groq chips could be good enough to meet the majority of demand.
Furthermore, if you're an enterprise the risk of data exfiltration and feeding data to a potential competitor like OpenAI or Anthropic is greatly reduced if you could shift to on-prem ASIC deployments. A handful of chips could cover a wide variety of use cases and cover them more securely. There are a lot of corporate use-cases for LLMs that are not frontier math research or coding.
I'm not convinced, mostly because things like crypto, which I believe went into ASICs, were based on very slowly moving and mostly understood algorithms. LLMs and model architectures seems significantly more volatile. I wouldn't want to be working out the finer details of my chip rollout only to find a new paper/approach that give multiples of performance.
So I guess it depends on how much the latest-greatest model motivates people, and my read on the current churn is that developers are extremely unloyal to brand at this point and will jump to whoever has the best model. And as long as the best model is running on programmable GPUs, that will be the dominant form.
The tokens per second performance numbers coming from Cerebrus/Talas are several orders of magnitude higher than models running on GPUs, which is such a huge step change that it will enable many more uses of LLMs that are impractical otherwise. I.e. think about gamers and burning in an LLM chip on a game console like a future Play Station - it doesn't matter if its a frontier LLM if it allows them to talk to in game characters in real time and have the LLM drive the storyline. The devs may be limited to training/RLHFing legacy models, but the performance enables many more use cases.
I admit I would like a faster model - but even though I have faster models available I still go to Fable or GPT-5.6 90% of the time. So there is a gap between a potential preference and a revealed preference.
Custom AI for things like facial recognition in cameras has existed for decades, before LLMs were a thing. I don't see that getting replaced. And on-device conversational intelligence might go that route as well, we'll have to wait and see. It's a lot of silicon to dedicated to a static non-changing thing. My money would be on programmable TPU-like things (Apple's NPU kind of stuff). It just seems more flexible to have an array of compute that you can load different models into, so you can update it, etc.
> even though I have faster models available I still go to Fable or GPT-5.6 90% of the time
What about all the things you don't currently use an LLM for?
If a specialized chip can run a model 100 times faster, you can suddenly use it for a lot of things at sub-second latency. You can write "make white transparent and add a red outline to x.png" instead of the corresponding imagemagick invocation and perceive little to no latency difference. You can hook it up to your browser and have it yank out all advertisements live, or tell it to highlight anything that might interest you, again, live. There's probably thousands of latent use cases nobody has thought of that would be enabled by a truly fast LLM, even a mediocre one.
I don't think an on-device model needs to change much; it's already quite general in its capabilities.
> hook it up to your browser and have it yank out all advertisements live
AI: "I'm sorry, a security guardrail prevents me from performing this operation."
What you think doesn't matter unless its strictly for personal use.
Your company will decide what makes economical sense.
I don't think ASICs is the long term answer here at the local LLM level. I think GPUs will still be.
If you can have only one AI processor in your laptop (because they're big and expensive), it's going to be a GPU. This AI processor needs to inference LLMs, audio processing, image generation, video generation, etc. This is on top of normal graphics processing requirements such as video games, playing videos, decoding, encoding, etc.
At the enterprise level, I can see some ASICs working once the market fully matures and improvements in architectures slow down drastically while demand for inference increases drastically. How far are we from this world? Maybe 5-10 years? It seems like model architectures are still changing rapidly and labs want fast experimentation that programmable GPUs offer.
GPUs will still dominate in general - just like how CPUs still dominate despite ASICs.
The models would have to get significantly better for this to help, though. Unedited LLM dialog is quite bad. Though, I wouldn't put it past AAAs to try anyway.
This is where treating these smaller models more like parts in a purpose built appliance ultimately benefits us.
> Cerebrus/Talas
They are fast, but they're still programmable accelerators, not a model burned into the gates.
Cerebras is a programmable accelerator, but Taalas does burn the model into the gates.
Ah, right. I misremembered. Does Taalas have a meaningful advantage here then?
Taalas claims significant cost, speed, and power differential with running on Nvidia (and similar) hardware. Their whole thesis is that with making the hardware cost less, run faster, and run with less power mixed with their claimed fast production cycle for new weights, companies would plan on purchasing and then relegating the other chips to different workloads at the next update cycle. And the chips have facilities for fine-tunes that can by dynamically loaded. The info I've seen has been very light on details, but if they can even get halfway where they are planning, it'll be a massive shift.
6x. Taalas has Llama3.1 8B running at 18000 tok/s. Cerebras advertised that model at 3000 tok/s.
Interesting. Is the speedup from specializing for the shape of Llama 3.1 or are they (contra my mental model) actually winning on burning in the weights?
The weights are in SRAM, so the LLM architecture is burned in but the weights can be updated.