GPU-unlock: FP/BF16 instruction issue rate on Nvidia 3000's cards
github.com231 points by fragebogen a day ago
231 points by fragebogen a day ago
Hey this is my work, although the exploits etc aren't my ideas by any means. This is just a way to wrap the whole thing up in a much less annoying package than having to patch the nvidia driver. I originally got inspired by the CMP90/50 windows unlocks that run as standalone UEFI modules that chainloads into the OS.
Happy to answer questions!
Thanks so much. 27b is a favourite model and last night my dev ran some benchmarks to achieve 133tks about 3.2x which changes the economics.
great to hear, which quant + KV quant + KV size are you on? is it llama.cpp?
Here's the writeup from the experiments that claude did: https://claude.ai/artifact/X6WJVtKdL1LrhYuCMqdXvG#b9a4d3e2-8...
Super well done, I wished all mods took your approach. Patched drivers are a nuisance, especially if you have more than one kind of card. Thank you.
edit: now do the 4090 ;)
Yeah I hate the patched drivers too, I have a CMP170 as well I want to see if I can find a way to get fully working the same way without or minimal driver patching.
4000s is WIP, forks has already unlocked it but I'm doing some refactoring to better support different families and after that I'll port it over :-)
Is this relevant for 4090? It has FP8 which is already double the rate of FP16. Can this hack double the rate for FP8 too?
Can't wait, I have two 48G modded ada 4090's and 4 stock Blackwell 4000's (though I doubt those will be as easy). The 4090's alone would be an amazing thing to have boosted, they're my main workhorses and effectively any improvement there will immediately make my days more productive. If I can help out with testing, my email is in my profile.
Blackwell and hopper have a whole new boot process so those are unlikely to fall over anytime soon from what I've seen.
If you can't wait for 4090 someone has a fork with a seemingly working unlock :D
Yes, but it's pinned to a driver version that I unfortunately can not use. That's precisely why I love your solution, it is the best way to do this.
I'm running 8 GPUs attached to a single CPU and that driver is a pretty essential part of getting this to work with P2P between as many cards as possible. The modded 4090 is a problem because the BAR1 it advertises (32G) is smaller than the VRAM so not all software is able to use it (you need two memory segments rather than a single 48G one to make this work). That driver is part of the solution for this problem.
FWIW the gpu unlock ada repo doesn't pin to a driver version, Codex for me just compiled it, and it works with my 615.xx driver even though the fork is for 610.x. I think the repo just uses the firmware from that version. My GPU is 4070ti super.
Ok, trying it. I'm using a pretty heavily modified 590.48.01 to be able to use those 48G 4090s, they work but they do require a bit of handholding. The main problem is the BAR1 just isn't large enough, it's capped at 32G.
Heh, got it working, thank you so much!!
Overall 30% reduction in wall clock time, that's pretty neat (and I'm still not running this at full power but 450 W is a bit much for that puny connector).
I have two of modded RTX 4090 with 48GB VRAM. But I'm using default drivers. I'm using models, which uses almost full 48GB.
Now i'm curious - am I missing some settings? Do I need to check any of these BAR things?