Fiber lasts decades, GPUs last three years: the AI boom isn't the dot-com boom [video]
youtube.com36 points by frag 4 hours ago
36 points by frag 4 hours ago
Shameless self promotion: I put together an AI infra built-out financials data-vis project/page recently. It includes graphs and tracks updates to stuff like GPU depreciation schedules like this.
I love the presentation of this. Phenomenal work, thank you.
Bears are gonna eat.
Oops, I read it as bullish. 98.6% utilization? Seems like demand for more capacity.
I agree, and it can’t be right - is this based on “projected utilization” or something?
My thinking is there wouldn’t be downward pricing pressure on data center compute if utilization was actually this high - it would mean providers could jack up prices and walk with larger margins rather than burn money for acquisition, right?
Plus the market wouldn’t increase the price of borrowed money if utilization was that high; they’d see the same signal. But the cost to borrow money for datacenters has roughly doubled year on year (might just be for riskier ventures? Buildings? Is this aggregate?) - those two signals wouldn’t disagree, it’s nonsensical (I mean, right?)
Edit: it’s just “the buildings”, and it makes sense: maximal utility comes from installing compute in a building already setup to house compute (cooling, infra, power). I actually take this as a negative signal: we’re seeing downward pricing pressure already, AND new projects aren’t fully booked, AND money is now twice as expensive. Somebody’s going to lose their shirt unless OpenAI’s Omnissiah comes online and converts us all.
Thanks for sharing. Counterpoint: H100's MSRP for around $30k. For 6 months at $12/hr, that's $51,840 revenue per card. So they're hitting break even in 3 months, everything after that is profit (minus electricity, etc), and the cards are still selling for around MSRP used.
The real question IMO is building capex. Those depreciate over a longer timeline, but if demand were to drop in 5 years and we have an excess of datacenter space, that's going to hurt. We're approaching the physical limits of what's possible with silicon and fiber optics. Unless there are new architectures or break throughs in algorithms, the only way to make AI more productive today is to build larger clusters, or much longer training/RL runs.
- GPUs last a lot longer than 3 years.
- The factories don't stop working after 3 years. They can keep making GPUs.
- The AI boom isn't just about GPUs. There have been massive investments in CPUs, RAM, networking and routing components, physical data centers, power plants, cooling, and even fiber (the thing you yourself claim lasts decades). None of this is going to disappear in a couple years.
I recommend looking at any data regarding A100 rents (6 year old GPU) and then save 50 minutes of your time.
Same with the consumer variants. RTX 3090 launched at $1499 and used ones on ebay are over $1500. GPUs large enough and new enough to do inference are not losing value.
Volta generation prices are collapsed, so GPU depreciation is currently around 8 or 9 years
What are we looking for? What’s your time-saving insight?
Thats the weird thing, scarcity drives value up so long as there is scarcity.
Once there is enough, or even a glut, it's over.
And I think the counterpoint here would be: Will there ever be enough intelligence or will we just want and use as much as we can possibly build?
Is intelligence like for example food where if I get my daily calories I’m good? Or is there no limit to the demand for intelligence. If there’s more available, will we just keep using more? Forever? It can’t be limitless right?
But where’s the limit because right now we’re consuming every GPU hour we can.
There is a use for older GPUs, but the chips do have reliability issues. Turns out running chips hard 24/7 causes them to burn out. So it’s less will they be “useful” a while from now but more will they still work.
Folks are leaning it’s like these LED light bulbs that say they’ll last 15 years but after 2 years they start flaking out at a high rate.
Separately, the broader point is still directionally correct. We’re still using fiber from the .com network infra implosion today and will for a long time. The deprecation cycle is incredibly long. The increasing reports of GPUs sitting on shelves because there aren’t data center to power them is very concerning considering all the high interest debt being used to buy what is ultimately a rapidly depreciating asset. Financially, that’s a disaster waiting to happen.
Turns out running chips hard 24/7 causes them to burn out
Hopefully somebody with direct experience can correct me if I'm wrong...
Fluctuations. If the use of the GPU is steady state, I think you're right; the thermals would be consistent. However, the reality is oftentimes loads come and go, and the rise+fall+rise of temps can lead to structural problems due to thermal expansion. Micro fractures in solder, thermal paste degradation, sagging PCBs, etc.
No, take a look into people who bought used GPUs that were from crypto farms. It's not just thermal paste, the chips become unreliable because of degradation of the circuits etched on the chips themselves.
Which is almost exactly the same story that applied to the flaky LEDs - bad thermal designs leads to the LEDs or their driver circuitry breaking.
If you use opencode for your AI work or becomes affordable. To the point where you would reasonably build applications with intelligence the LLM version. I think in the near future the intelligence will be free and there is no business in free. I know it is a big statement but if you look around there is already a large amount of free models of gradually increasing strength. For most applications you don't need a gpt6
Ed Zitron is going to become the Simpsons of ai critiques...
Lots of respect for him for staying unwaveringly principled while knowing when and how to adjust his expectations and predictions. He is really well informed and it's something else to watch him speaking live on the podcasts and be so specific with numbers. He really knows his stuff.
Go read "Some Zitron predictions" here: https://danluu.com/zitron/
His track record of prediction on this topic is stunningly bad, and he's made a job out of being counter to the AI hype, so he seems likely unable to honestly update his beliefs to adapt to changing evidence without jeopardizing the attention people pay to him.
Predicting global events is next to impossible. Don't look at the predictions, as his prediction game has slowed and he has recently admitted that he was wrong on his predictions. Look at his exposing of financial shenanigans which let the bubble grow.
The specifics are tough, but the rough shape isn't impossible. But it's hard, which is part of why most people try not to do it publicly with much confidence, lest they look like they're not worth listening to on the topic. Like, why are we listening to his opinions on this when he so clearly isn't an expert, and so many of his past opinions on it have been so far off?
And I think a big part of why you shouldn't listen to him is that he's said repeatedly that he doesn't really use generative AI outside of occasional experiments - he hasn't seen its potential enough to bother using it himself beyond some initial tests. Which is fine, but I think it also means that he likely doesn't have a good intuition of the likely future demand, which is pretty key in figuring out whether debt loads are likely to be sustainable or not. Investing is fundamentally about trying to predict the future and putting your money into the things that will likely be successful there. And he's demonstrably been terrible at predicting the future of this stuff.
"A foolish consistency is the hobgoblin of little minds, adored by little statesmen and philosophers and divines. With consistency a great soul has simply nothing to do. He may as well concern himself with his shadow on the wall. Speak what you think now in hard words, and to-morrow speak what to-morrow thinks in hard words again, though it contradict every thing you said to-day" Ralph Waldo Emerson.
It would be one thing if he once admitted that he was wrong and adapted his world view to account for new information, but he just doubles and triples down.
GPUs may last 6 yrs or more. Fiber vs GPUs is however not the right comparison. Fiber vs Open models are. Open models have mostly caught up with frontier models. Now, with a reasonable harness and tools they can easily be useful for vast majority of users for a decade or more.
There are multiple YT channels proclaiming Russia is finally collapsing/losing war with Ukraine, huge "It is happening" in thumbnail; few of them put out a variation of such video every week, not even bothering to excuse their previous, wrong prediction. And any reporting related to AI bubble is exactly as terrible. I might as well just check Bloomberg/FT/QZ, news there will be devoid of technical details and still more useful than those endless armchair-experts' speculations.
GPUs might be 3 years but the rest of the hardware: processor cores, ram, storage can power other cloud applications for a decade or more no?
Except the GPUs last a lot longer than 3 years - A100s are 6 years old at this point and still in heavy use.
Some GPUs may easily last more than 10 years of light use, but I am not aware of public data about the reliability of datacenter GPus, which would give the percentage of A100s that die during the first 3 years of 24/7 use and during the first 6 years of 24/7 use.
I doubt that many of the 6-year old A100s have been in use 24/7.
In this paper, they examined a cluster with 448 A100s for 2.5 years at ~50% utilization, 4 were replaced over that time, about 0.36%/year:
Another thing to pay attention to is the margins of hardware companies like nvidia or micron. Hyperscalers are spending huge amount of money but the money mostly goes to the profits of hardware companies.
[flagged]
[dead]
[dead]
I don't want to argue here, but once models are trained, using them is quite cheap. Also in the last few months most improvements I've seen are in the smaller sized models, which might make using models even cheaper.
If the models are cheap to run, while good for customers, isn't that a problem for hyperscalers and all the investment poured into the AI data center build out?
No. If models become better, the natural expectation is for hardware to get more valuable, not less. (The hardware that was previously able to achieve X can now do 2X. Why would that make the hardware worth less?)
Likewise if models become commoditized, it's raising the value of the existing hardware (you'd expect the hardware owner to keep a bigger slice of the surplus when that's the scarce component).
For model improvements to be bad for the value of hardware, what you'd need is for demand for intelligence to be inelastic. But everything we've seen so far suggests that is not the case: the demand for intelligence (whether human or artificial) is highly elastic both in price and in quality.
its a false equivalence.... the dotcom boom wasnt just a fibre rollout, it was all the internet company startups that were overpriced and crashed. no doubt some of the hyperscalars will get busted out, but the models and LLMs are here to stay. just like dot com websites didnt disappear after the dot com crash, neither will AI transformer models
All of this spending only has a chance of paying off if video generation takes off.
> using them is quite cheap.
until the other guys comes out with a slightly better model, then using yours wont be cheap enough.
There's this weird tension where small models are getting a lot better, but the gap between the frontier and "what I can run on my gaming GPU" at home isn't necessarily getting smaller. That said, Qwen-3.8-27B-Q3 runs on my 9700 XT and performs well enough for tool calling in agents, which is a critical floor in local model performance imo.
Do you use qwen for any coding? Do you make a frontier model orchestrate it? Are you running with llama.cpp? Sorry for the barrage of questions, but I've been interesting in local AI for a while, yet I've struggled to set up anything usable.
>Do you use qwen for any coding?
No.
>Do you make a frontier model orchestrate it?
Also no.
>Are you running with llama.cpp?
Yes, on Arch Linux.
>Sorry for the barrage of questions
No worries.
My Qwen testing has been with Hermes trying to do mundane (to me) things that hosted models understandably refuse to do, even open-weight ones like Deepseek V4.1 Flash. Example: I tried to get it to find a torrent for $tvshow and add it to qbittorrent (connected to hermes via MCP already). It certainly tried and I think frontier models would have done better. My biggest issue was context space more than anything else. I had to drop down a quant level to increase it to 128k. On 64k it was constantly compacting.
Just go download and install lemonade server. Open web ui for front end. There are mobile apps to connect to it.