I were 17, I'd learn how to build LLMs from scratch

372 points by bilsbie 20 hours ago

https://xcancel.com/paulg/status/2091544343589060625

There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities.

The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM company, and still then it's a struggle. Those few that do train, they spend most of their budget on compute and have relatively small teams.

Getting experience in this field requires having access to very expensive hardware to begin with. And the skills will be quite hard to convert into any real value for someone, leading to a decent income, unless you have a ton of funding from patient investors, or you have decent contacts in Bay Area networks to get hired at the right place.

With all due respect, paulg is in somewhat of a bubble, this is not congruent with the global situation.

femto - 5 hours ago

In that regard, it's not too different from mobile telephony. Mobile phones drove the electronics industry 20 years ago, but there is limited demand for people who really know how to build a phone (ie. build the hardware and write all the signal processing from scratch), as there aren't that many companies that do phones at the lowest level. A few of the engineers got rich (eg. Viterbi), but most 'just' made a good living. Most people who got rich off phones didn't do it by knowing how phones work.
Incidentally, the skills for the lowest levels of LLMs aren't that far removed from those needed for mobile telephony, in that both are based on maths, computation and information theory.
- Aissen - 2 hours ago
  
  I was there at the start of the smartphone boom. I built a demo Android device that was capable of telephony/data, 3D rendering, etc. all before Google open-sourced the OS, for a SoC vendor that wasn't in Google's inner circle. Yet the industry was not interested in my junior profile during the subprime crisis.
  - storywatch - an hour ago
    
    I wonder if the modern incarnation of Android flagships would have taken off at all if not for Apple. When Android first showed up, the dominant "smartphones" were running Symbian and they had a very different view of the world when it comes to UX and hardware.
    
    Izkata - 16 minutes ago
    
    Phones were already going in that direction before the iPhone was released: https://phys.org/news/2007-02-samsung-unveils-revolutionary-...
    But it was generally seen as a gimmick instead of desired before Apple made it look good. Even when the iPhone came out, one of the jokes was how the grid of icons looks like how a Windows user's desktop would look like when they didn't understand the filesystem.
  - rfgplk - an hour ago
    
    Ridiculous frankly.
  - nomoreusernames - an hour ago
    
    [dead]
- __MatrixMan__ - 2 hours ago
  
  Good points, but I think we can expect the AI space to be more tumultuous.
  What most people want from mobile technology is for it to work, not too expensively, and for it to get out of their way.
  What most people want out of AI is for no leader to emerge and wield supremacy against the rest of us. People are afraid of it in ways they weren't afraid of mobile, so they're more willing to work together against whoever is in the lead.
  Its more like an arms race and less like a utility. The disadvantage I face when my competition has better mobile coverage and bandwidth is minor. The disadvantage I face when my competition has better intelligence on tap is much more significant.
- armcat - 5 hours ago
  
  This is super interesting because I moved from mobile telephony into ML and data science, and information theory and working with data in statistically correct way was what helped me! This was 10 years ago though.
- mnicky - 3 hours ago
  
  Also, in a few years, LLMs will be building the next generation of LLMs anyway, probably autonomously to a high degree.
yobbo - 8 hours ago

Yes. It's like looking at the (Apollo) moon rocket launch and then suggesting teenagers should learn to build rockets in their garages for the coming space age.
It is viable as a toy project, but there are vanishingly few career opportunities.
- bmacho - 7 hours ago
  
  It's like looking at the early internet and then suggesting teenagers should write browsers as their projects instead of webpages.
  - mixmastamyk - an hour ago
    
    That may have worked at the time, but no companies are interested in learning projects today. If you didn’t do the reqs list for the last 5-10 years with the same title, forget about it. Because there are dozens of folks who have, lined up. No one is indulging career changers (and most fresh grads) for the time being.
  - nextaccountic - 5 hours ago
    
    If someone writes a browser in their teens, they will probably learn more about the web than if they were just writing web pages
    
    paulryanrogers - 4 hours ago
    
    They'll learn so much more that won't transfer to as many job opportunities. For ex, say more about C++ and less about cutting edge CSS (because modern browser tech is an ocean). I suppose they might luck into other adjacent or unrelated roles with the same skills.
    
    simonjgreen - 3 minutes ago
    
    This is like saying writing a compiler doesn't give you transferable skills. The surrounding competencies required to do this grant a pretty large amount of broad domain awareness
    
    hnlmorg - an hour ago
    
    CSS wasn’t around in the early web.
    Building a browser in the early web was actually a very achievable goal for exactly the reasons it isn’t now. There was not JS. No CSS. No SVG. In fact very few widely supported image formats (and graphical browsers weren’t around in the earliest days of the web anyway). TLS didn’t exist. HTML only had a subset of methods. And even POST was usually just managed by CGI/BIN calling an external process, often written in C++ or Perl.
    It was a simpler time.
    
    dylanowen - an hour ago
    
    I have a hard time imagining anything where writing a browser wouldn't be an excellent transferable foundation. Even your example: writing css will never teach you as much as writing a css engine.
    
    switchbak - an hour ago
    
    They will also know the fundamental underpinnings of the web, you know - the thing that those html devs are actually using.
    That’s like saying someone who fabricates cars doesn’t have the skills to drive them. Perhaps not, but they’re very well placed to pick it up quickly. They’ve also shown they can do something far more challenging, which is actually better than hiring for narrow immediate skills.
    In other words: I’d hire that candidate in a heartbeat.
    
    StilesCrisis - 39 minutes ago
    
    You'd hire an auto designer to be a limo driver? I think it's just two different skill sets. The auto designer is clearly smart but a limo driver also needs to have customer service skills, etc.
    
    bigfishrunning - 2 hours ago
    
    Are you asserting that C++ is not a marketable skill?
    If anyone reading this has this un-marketable skill, Carnegie robotics in Pittsburgh is hiring
    carnegie-robotics.breezy.hr/p/2d85f5321cc7-software-engineer
    
    ryandrake - 2 hours ago
    
    I think he's saying that building a browser is not a transferrable skill, like making a generic web page is. Employers don't like specialists. I used to build display drivers for graphics cards. Wonderful learning opportunity but other employers not in the graphics card manufacturing business didn't give a shit--those three years were essentially treated as an employment gap. "Well that's nice, but we really wish you had general experience writing CRUD apps..."
    
    hnlmorg - 13 minutes ago
    
    CRUD didn’t exist in the early days of the web.
    People keep talking about a browser in the modern context but the GP specifically said “early days of the internet” (which, in fairness, would mean pre-web. But I think it’s safe to assume they meant “web” not Internet).
    In those days, it was actually a much simpler exercise to write a browser than it is today. I even wrote one! And writing a browser absolutely teaches you how HTTP and HTML worked. Plus a lot of backed development was forms data sent to CGI and thus written in languages we wouldn’t even dream of using for web development nowadays, including C++.
    So in the early web, writing a browser absolutely was a transferable skill. It might not be now, but in the context defined by the GP, it was.
    
    bigfishrunning - an hour ago
    
    I would argue that in a post-LLM world, becoming as specialized as possible is the only way to survive. If you have the serious systems programming skills required to write graphics drivers, your talent would be wasted writing CRUD apps anyway. I hope you eventually found/will find something more appropriate for your skillset!
    
    QuercusMax - an hour ago
    
    But building a web browser isn't a niche skill - it requires a whole bunch of them. You need to be a generalist to write a web browser.
    
    ignoramous - 5 hours ago
    
    A few did.
    [Blake Ross] worked as an intern at Netscape at the age of 16 ... Ross became disenchanted with the browser he was working on and the direction given to it by America Online, which had recently purchased Netscape. Ross and Hyatt envisioned a smaller, easy-to-use browser that could have mass appeal, and Firefox was born from that idea ... in 2003 all of Mozilla's resources were devoted to the Firefox and Thunderbird projects. Released in November 2004, when Ross was 19, Firefox quickly grabbed market share ... with 100 million downloads in less than a year
    https://en.wikipedia.org/wiki/Blake_Ross
    
    nwatson - 3 hours ago
    
    In the meantime Mozilla was mocked on slashdot.org and elsewhere relentlessly after the first two years of the project when nobody believed there would be any value in the effort. Hats off to the team that took around five years to get to Firefox 1.0 (and released Mozilla browser in the interim). It took a lot of conviction to see it through.
    
    topham - 3 hours ago
    
    Sure, you had to pick the browser I despise worse than Netscape 4.
  - serbuvlad - 7 hours ago
    
    If you learnt to build a browser in 2000 you're probably doing well for yourself lol.
    Not a lot of demand, but also probably not a lot of supply.
    
    zorked - 5 hours ago
    
    There is an entire graveyward of browsers, almost all that didn't die are now forgotten.
    
    serbuvlad - 3 hours ago
    
    Sure, but I'm sure most of their lead devs are well paid now.
    In general C++ work and similar, if not in Chrome development.
    
    aswegs8 - 2 hours ago
    
    cause != effect
    
    XRG - an hour ago
    
    There is a confounding variable, however -- the person is most likely a good dev. It stands to reason that they've had a decent career at least since then.
    
    serbuvlad - an hour ago
    
    Yes but as a dev you are shaped by the projects you work on.
    
    sampullman - 5 hours ago
    
    I think the point isn't that you'll necessarily build a great browser (or LLM), but the experience will benefit you in other ways.
    
    antupis - 42 minutes ago
    
    Yup like you go far enought you will pick lots of transferable skills like data cleaning in the case of LLM, DOM parsing case of web browser or welding case of rocket.
    
    paulcole - 5 hours ago
    
    If any were built by teenagers, I’d imagine those teenagers ended up with pretty good careers in technology?
    
    paulryanrogers - 4 hours ago
    
    For every wunderkind that has a long career there's also plenty who are overlooked or peak early.
    
    spolitry - 4 hours ago
    
    Citation desperately needed.
    
    fifilura - 3 hours ago
    
    https://en.wikipedia.org/wiki/Survivorship_bias
    
    paulryanrogers - 3 hours ago
    
    Ken Silverman