Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s): 50 tok/s with MTP
piszczek.pl21 points by pich 2 hours ago
21 points by pich 2 hours ago
Can you please try and see how many tokens you get with some form of concurrency. Pretty much ALL the benchmarks I've seen on the more accessible cards are just single request.
Because concurrency with a single "accessible" card quickly diminishes. I have dual 3090s, and on Qwen 3.6 35B A3B at 80k a single card with max concurrent set to 4 will get 70-80tps single request, 50-60 TOTAL tps with 2, 45-50 with 3, and around 40tps with all 4 going.
I know this doesn't exactly match your request, but I'm happy with my 3.6's performance. I've seen people claim they can get 120+tps on a 3090, but I'm not impatient when it comes to streaming text faster than I can read it.
You have something misconfigured then. Concurrency has never lowered my overall TPS. Also have dual 3090s. Generally use vllm though.
The whole site looks like and reads like AI slop. The outcomes also don't make any sense and don't feel rigorously tested (no, having claude test for you doesn't count as rigorous).
quantization level?
Its egregious the quant level isnt disclosed along the "Qwen" string. Everyone knows theres huge difference in speed/quality along the quant axis, I now attribute the ommision of such to deliberate choice to not curb the hype of the tittle.
they talk about the quants they tried in the article and settle on a Qwen3.8-27B-Hermes-iMatrix-NVFP4-Balanced.gguf which they calibrated on their own session traces and they pulled in 5 different llama.cpp pull requests to their local llama-server.
The article mentions Q4, Q5, Q8, and NVFP4. It's total AI slop though, tough read.
In my testing I got 150 tokens/sec with a single 5090 RTX.
Which model/quant/command line did you use? I can barely get 100 token/secs and for sure clearly not a full context. With vllm, I am limited to 130k tokens with vllm + nvfp4.