The annotated PyTorch training loop
idlemachines.co.uk76 points by smaddrellmander 4 days ago
76 points by smaddrellmander 4 days ago
PyTorch already does a huge job at keeping DL not verbose. It captures the complexity to allow you to make a whole GPT model with it, running on your GPU, and keep the code readable. Yes, there are a lot of breaking points, but at some point, DL is supposed to be non trivial, it handles math concepts most people don’t understand. That is why Data Scientists have a job ;)
I mean partial derivatives aren't that complicated if you know normal derivatives, which most people do. And backpropagation isn't too difficult either.
The value of PyTorch lies more in utilizing accelerators like GPUs while offering a nice abstraction. But you can build your own (inefficient) tensor library without too much effort as e.g. Andrej Karpathy has shown in his "NN zero to hero" youtube series.
I hope you don't actually believe that most people know derivatives.
I would hope most people do. It's in pretty much every high school curriculum, isn't it?
I attended one of the better (read: top ~200) public high schools in the US, and you could graduate without even taking trigonometry
Even if most people had calculus in high school (which is not a given), I'd expect pretty much anyone who did not go into STEM to promptly forget it, just like I forgot much from chemistry and biology after studying Computer Science.
The host website seems extremely interesting in general. Anyone here used it before?
I think the methodology is interesting, but you can probably design a similar, and probably better loop for "diving" into a topic and going through examples than is presented.
I have to imagine they just asked Claude to dive into a topic and generate example problems along the way.
As for the content... a lot feels like knowledge overload and concepts are introduced without explanation or "why". It basically says "here is a training loop" and never answers my immediate question of "...for what?" It also introduces random concepts like setting the seed that don't look like they're even in the final training loop it provides.
As a comparison, Pytorch itself has docs that go over the entire training loop as well. And it explains why you're loading the data its loading. With examples that run instead of being scattered on.
The design is similar to what Claude generates. Makes me skeptical that all the content is AI-generated.
This looks better than 90% of websites I visited. The palette is pleasant, content has the right density and spacing is very relaxed. If this is AI-generated without some clever prompt tuning, frontend as a profession is done.
I am 95% sure Claude came up with the design. It has all the tell-tale signs like the all-caps bold letter-spaced microcopy (See the orange "Sharpen your AI fundamentals" on the landing page.) Many card-like elements with a colored border on one side only. The italic serif font as subtitle. Etc. That being said I'm not judging. It's competently done.
I also put the content of a workshop into an LLM detector and it said 100% of the text was likely AI generated.
I'd just like the author to disclose how much of the content is entirely LLM-generated.
It's surprising to say people complimenting the color palette on this site. It's almost unreadable on my (very nice) monitor but clearly it works for many readers.
Most people here seem to think the content is high quality.
Do you disagree?
Uhhh apart from OP I don’t see anyone commenting on the content quality?
Also OP didn’t even praise the content. They said it “seems interesting.”
https://idlemachines.co.uk/courses/foundations doesn't render correctly on mobile
even the submitted page doesn't, huge margin if you zoom out on mobile (Firefox)
I've seen that huge margin when zooming out issue before. I'm using Safari on iOS, but have also experienced this issue in Chrome on Android. It's probably some sort of weird compatibility thing for desktop websites. You start zoomed correctly, but for some reason, there's horizontal scroll and you can also zoom out. For seemingly no reason, as I doubt any of the websites where this happens intended it.
Note that I can't reproduce it myself on this submission, but I've seen it so many times and in so many different places that I don't think it's a website issue.
Does Anthropic/OpenAI use PyTorch to train?