Show HN: Shoehorn – Quantize any model down to run on your machine

notactuallytreyanastasio.github.io

67 points by rhgraysonii 3 days ago

Working on Mac, Linux, and Windows now. I include a simple GUI to find new models and get things built and set up. It is working quite well across a few models for me. The GitHub README and DESIGN.md files go into detail of the how/why and it's working remarkably well so far. https://github.com/notactuallytreyanastasio/shoehorn

puttycat - an hour ago

This is really impressive. Can you say a bit about the underlying process? I'm guessing this is post-training qantization? Isn't PTQ also resource-intensive? (Ie might not work on any machine)

hmokiguess - 3 days ago

Reminds me of https://github.com/AlexsJones/llmfit

sscarduzio - 2 hours ago

The project name is perfect!

jedbrooke - 3 hours ago

I gotta laugh at some of the models it suggests, for example:

> AnkitAI/Parable-Qwen3-4B-Claude-Fable-5-GGUF

you’re telling me you managed to fit Fable 5 into just 4B?

akshay_akula - 2 days ago

This is interesting. I wonder how it could work with something like https://github.com/JustVugg/colibri.

mbuchel-hn - 3 days ago

does this work similar to airllm? i am wondering how it would handle something like quantizing kimi k3 on a budget of 8 gbs, or is that something you are not attempting to solve yet?

jaylane - 3 days ago

tried it out but based on the model sizing result i got i got an insufficient memory error when the server started running

kelvo_ran - 2 days ago

[dead]