Janas: Run LLMs larger than RAM by streaming experts from SSD
github.com44 points by khalidkhair 5 hours ago
44 points by khalidkhair 5 hours ago
This could be a game changer for me. The best I've been able to do so far is ~8 tok/s with Qwen 35B A3B, 32GB RAM. But then again I just got an OpenCode Go plan yesterday and have yet to figure if I can even consume my quota on DeepSeek Flash as the need to plan things well is bottlenecking me. Might be a while before I try the local thing again.
Qwen 3? (Jumanji) What year is it?
What a waste of time.
Strata already does this with Qwen3.8-Flash-Next: https://github.com/Niko1221/Strata
If anyone needs to see what 62 tokens/second looks like: https://token.qusai.pro/?speed=62
Which seems sufficient for mortals who are not running gas town, but want a local model.
Not sure if the measurement method used for this is actually valid or reliable enough, but for reference, Codex 6.1 Sol Standard runs at 19.9 tok/s according to a benchmark I recently saw: https://www.reddit.com/r/codex/s/b4mUszSOv1
60 toks/sec may be ok for pure chat or token generation, but not for token preprocessing nor any sort of agentic work.
>.. nor any sort of agentic work.
may I ask why?
oai luna runs at 60-70 toks/s; astra runs around 35-45 , are you just worried that your agentic work can't spread across a lot of parallel work?
The numbers you povided are for output/generation. Input is around 5-20 thousand for Astra, and 16-60k for Luna. So a few hundred times faster.
Most of the tokens consumed by agents is reading stuff - docs, tool outputs, code etc. Previewing a content of a dumb single page of html is a few thousand tokens.
If you use claude or some other harness you can ask it how many cache writes/input tokens are there in avg session, and how much time it would take if preprocessing speed was just 60 tok/s.
I would guess the assumption is that "agentic work" should be happening asynchronously in the background on a machine that's also used for foreground interactive work, where the slower the inference is, the longer the machine's GPU and [V]RAM are "hogged" by it at a time, and so the more the machine's performance seems to degrade.
My self hosted LLM runs at 60-ish tok/s and does great at agentic tasks. I think that’s about the threshold where it stops feeling annoyingly slow.
Why are the models being used so old (in llm years anyway). Why no Qwen 3.8 ?
Based on how vibe-coded this whole thing is, I’m guessing the newer models weren’t in the training data and the agent just picked what it knew existed.
I wonder when we will start seeing small commercial accelerators integrated with NVMes; could potentially use far more bandwidth than typical NVMe drive can push thru the four lanes.
Given that you'd have probably a lot more capacity just to get to required bandwidth it could be used to switch very quickly between models (as all stored there would essentially be "loaded")
When High Bandwidth Flash comes out, most likely: https://chipsandcheese.com/p/hot-chips-2026-applying-high-ba...
Seems like a useful innovation on the desktop level, but on the server level, isn't it already pretty normal to see machines with 24x high-performance (i.e. individually high-bandwidth) NVMe drives in them, that can then be used in software RAID0 to achieve pretty massive read bandwidth?
I think what we're going to see first is essentially gpus with nvme (or at least flash) bolted on, not sure if that's also what you're saying or not.
would this work on a virtual disk, say w s3 or dropbox
Sure, it’ll just be absurdly slow.
Maybe next you can run an LLM off of some old MFM drives
Not online services. The overhead of TCP and latency I imagine would really lag performance. A SAN over SFP, maybe.
There are many of these stream-experts-from-SDDD projects now. What's needed is a guide to all of them and maybe for someone to take the best features of each of them and merge them into one uber project.
[flagged]