Show HN: I built a tiny LLM to demystify how language models work
github.com591 points by armanified 12 hours ago
591 points by armanified 12 hours ago
Built a ~9M param LLM from scratch to understand how they actually work. Vanilla transformer, 60K synthetic conversations, ~130 lines of PyTorch. Trains in 5 min on a free Colab T4. The fish thinks the meaning of life is food.
Fork it and swap the personality for your own character.
How does this compare to Andrej Karpathy's microgpt (https://karpathy.github.io/2026/02/12/microgpt/) or minGPT (https://github.com/karpathy/minGPT)? I haven't compared it with anything yet. Thanks for the suggestion; I'll look into these. https://bbycroft.net/llm has 3d Visualization of tiny example LLM layers that do a very good job at showing what is going on (https://news.ycombinator.com/item?id=38505211) have little to do with this, but i have to say your project are indeed pretty cool! Consider adding some more UI? This really makes me think if it would be feasible to make an llm trained exclusively on toki pona (https://en.wikipedia.org/wiki/Toki_Pona) It's genuinely a great introduction to LLMs. I built my own awhile ago based off Milton's Paradise Lost: https://www.wvrk.org/works/milton This is probably a consequence of the training data being fully lowercase: You> hello
Guppy> hi. did you bring micro pellets. You> HELLO
Guppy> i don't know what it means but it's mine. Great find! It appears uppercase tokens are completely unknonw to the tokenizer. But the character still comes through in response :) Finally an LLM that's honest about its world model. "The meaning of life is food" is arguably less wrong than what you get from models 10,000x larger Meaning/goal of life is to reproduce. Food (and everything else) is only a means to it. Reproduction is the only root goal given by nature to any life form. All resources and qualities are provided are only to help mating. Then why are reproductive rates so low in western countries? https://en.wikipedia.org/wiki/List_of_countries_by_total_fer... I was going to suggest implementing RoPE to fix the context limit, but realized that would make it anatomically incorrect. I like the idea, just that the examples are reproduced from the training data set. How does it handle unknown queries? It mostly doesn't, at 9M it has very limited capacity. The whole idea of this project is to demonstrate how Language Models work. Could it be possible to train LLM only through the chat messages without any other data or input? If Guppy doesn't know regular expressions yet, could I teach it to it just by conversation? It's a fish so it wouldn't probably understand much about my blabbing, but would be interesting to give it a try. Or is there some hard architectural limit in the current LLM's, that the training needs to be done offline and with fairly large training set. > A 9M model can't conditionally follow instructions How many parameters would you need for that? My initial idea was to train a navigation decision model with 25M parameters for a Raspberry Pi, which, in testing, was getting about 60% of tool calls correct. IMO, it seems like around 20M parameters would be a good size for following some narrow & basic language instructions. Ok. This makes me wonder about a broader question. Is there a scientific approach showing a pyramid of cognitive functions, and how many parameters are (minimally) required for each layer in this pyramid? > you're my favorite big shape. my mouth are happy when you're here. Laughed loudly :-D This is a direct output from the synthetic training data though - wonder if there is a bit of overfitting going on or it’s just a natural limitation of a much smaller model. Love it! I think it's important to understand how the tools we use (and will only increasingly use) work under the hood. Why are there so many dead comments from new accounts? Because despite what HN users seem to think, HN is a LLM-infested hellscape to the same degree as Reddit, if not more. But what should we do? The parent company isn't transparent about communicating the seriousness of this problem You’re absolutely right! HN isn’t just LLM-infested hellscape, it’s a completely new paradigm of machine assisted chocolate-infused information generation. Just let me know which type of information goo you'd like me to generate, and I'll tailor the perfect one for you. It really seems it's mostly AI comments on this. Maybe this topic is attractive to all the bots. Great and simple way to bridge the gap between LLMs and users coming in to the field! This is so cool! I'd love to see a write-up on how made it, and what you referenced because designing neural networks always feel like a maze ;) how's it handle longer context or does it start hallucinating after like 2 sentences? curious what the ceiling is before the 9M params * How creating dataset? I download it but it is commpresed in binary format. * How training. In cloud or in my own dev * How creating a gguf ```
uv run python -m guppylm chat Traceback (most recent call last): meybe add training again (read best od fine) and train again ```
# after config device
checkpoint_path = "checkpoints/best_model.pt" ckpt = torch.load(checkpoint_path, map_location=device, weights_only=False) model = GuppyLM(mc).to(device)
if "model_state_dict" in ckpt:
model.load_state_dict(ckpt["model_state_dict"])
else:
model.load_state_dict(ckpt) start_step = ckpt.get("step", 0)
print(f"Encore {start_step}")
``` Hm, I can actually try the training on my GPU. One of the things I want to try next. Maybe a bit more complex than a fish :) I could fork it and create TrumpLM. Not a big leap, I suppose. Would have been funny if it were called "DORY" due to memory recall issues of the fish vs LLMs similar recall issues :) I love these kinds of educational implementations. I want to really praise the (unintentional?) nod to Nagel, by limiting capabilities to representation of a fish, the user is immediately able to understand the constraints. It can only talk like a fish cause it’s very simple Especially compared to public models, thats a really simple correspondence to grok intuitively (small LLM > only as verbose as a fish, larger LLM > more verbose) so kudos to the author for making that simple and fun. > the user is immediately able to understand the constraints Nagel's point was quite literally the opposite[1] of this, though. We can't understand what it must "be like to be a bat" because their mental model is so fundamentally different than ours. So using all the human language tokens in the world can't get us to truly understand what it's like to be a bat, or a guppy, or whatever. In fact, Nagel's point is arguably even stronger: there's no possible mental mapping between the experience of a bat and the experience of a human. [1] https://www.sas.upenn.edu/~cavitch/pdf-library/Nagel_Bat.pdf IMO we're a step before that: We don't even have a real fish involved, we have a character that is fictionally a fish. In LLM-discussions, obviously-fictional characters can be useful for this, like if someone builds a "Chat with Count Dracula" app. To truly believe that a typical "AI" is some entity that "wants to be helpful" is just as mistaken as believing the same architecture creates an entity that "feels the dark thirst for the blood of the living." Or, in this case, that it really enjoys food-pellets. Id highly disagree with that. Were all living in the same shared universe, and underlying every intelligence must be precisely an understanding of events happening in this space-time. Different argument I’m not going to argue other than to say that you need to view the point from a third party perspective evaluating “fish” vs “more verbose thing,” such that the composition is the determinant of the complexity of interaction (which has a unique qualia per nagel) Hence why it’s a “unintentional nod” not an instantiation [flagged] Training data is here: https://huggingface.co/datasets/arman-bd/guppylm-60k-generic I think this is a nice project because it is end to end and serves its goal well. Good job! It's a good example how someone might do something similar for a specific purpose. There are other visualizers that explain different aspects of LLMs but this is a good applied example. How much training data did you end up needing for the fish personality to feel coherent? Curious what the minimum viable dataset looks like for something like this. Great work! I still think that [1] does a better job of helping us understand how GPT and LLM work, but yours is funnier. Then, some criticism. I probably don't get it, but I think the HN headline does your project a disservice. Your project does not demystify anything (see below) and it diverges from your project's claim, too. Furthermore, I think you claim too much on your github. "This project exists to show that training your own language model is not magic." and then just posts a few command line statements to execute. Yeah, running a mail server is not magic, just apt-get install exim4. So, code. Looking at train_guppylm.ipynb and, oh, it's PyTorch again. I'm better off reading [2] if I'm looking into that (I know, it is a published book, but I maintain my point). So, in short, it does not help the initiated or the uninitiated. For the initiated it needs more detail for it to be useful, the uninitiated more context for it to be understood. Still a fun project, even if oversold. [1] https://spreadsheets-are-all-you-need.ai/
[2] https://github.com/rasbt/LLMs-from-scratch The constraint-driven approach here is what makes it actually useful as a learning tool. When you're working with ~130 lines of PyTorch, you can't hide behind abstractions — every design choice has to be explicit and intentional. Curious: did implementing attention from scratch change how you think about the "key/query/value" intuition that gets used in most explanations? That's usually where the hand-waving happens in tutorials.
fg137 - 2 hours ago
armanified - an hour ago
totetsu - 3 hours ago
maverickxone - an hour ago
algoth1 - an hour ago
ordinarily - 10 hours ago
mudkipdev - 7 hours ago
functional_dev - 6 hours ago
hackerman70000 - 4 hours ago
zkmon - an hour ago
amelius - 33 minutes ago
drincanngao - an hour ago
zwaps - 7 hours ago
armanified - an hour ago
bblb - 4 hours ago
amelius - an hour ago
armanified - an hour ago
amelius - a minute ago
cbdevidal - 9 hours ago
vunderba - 9 hours ago
ankitsanghi - 7 hours ago
brcmthrowaway - 7 hours ago
59nadir - 4 hours ago
siva7 - an hour ago
wiseowise - 2 hours ago
toyg - 2 hours ago
loveparade - 5 hours ago
ananandreas - 2 hours ago
kaipereira - 7 hours ago
kubrador - 7 hours ago
gdzie-jest-sol - 4 hours ago
gdzie-jest-sol - 4 hours ago
Exception: No such file or directory (os error 2)
``` File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/home/user/gupik/guppylm/guppylm/__main__.py", line 48, in <module>
main()
File "/home/user/gupik/guppylm/guppylm/__main__.py", line 29, in main
engine = GuppyInference("checkpoints/best_model.pt", "data/tokenizer.json")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/gupik/guppylm/guppylm/inference.py", line 17, in __init__
self.tokenizer = Tokenizer.from_file(tokenizer_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gdzie-jest-sol - 3 hours ago
NyxVox - 9 hours ago
rclkrtrzckr - 6 hours ago
SilentM68 - 10 hours ago
AndrewKemendo - 11 hours ago
dvt - 10 hours ago
Terr_ - 6 hours ago
andoando - 6 hours ago
AndrewKemendo - 10 hours ago
LeonTing1010 - 9 hours ago
secabeen - 9 hours ago
dinkumthinkum - 8 hours ago
martmulx - 9 hours ago
Propelloni - 4 hours ago
areys - 17 minutes ago
