LLMs are not superhuman at reading and writing neural network weights

seangoedecke.com

151 points by agrm 5 days ago


StilesCrisis - 4 days ago

"If someone showed you that heatmap without the game labels, I think you’d be able to immediately identify at least the first three games" -- absolutely not. Pac-Man in particular seems like a huge stretch to me.

solyin - 4 days ago

> the best way to work with binaries is still to write source code and compile it. Likewise, the best way to update the weights of a model is to train it normally.

This is why the distinction between open weights and open source is important.

ChrisArchitect - 5 days ago

Discussion on source:

https://news.ycombinator.com/item?id=50341174

nagi_builds - 5 days ago

"I think a smart human ML researcher could perform all of these tasks."

And do that for how long?

aaroninsf - 4 days ago

False, in a way that genuinely matters.

True: they are superhuman at every one of these tasks, in the boring way they are better—according to a wall clock and the coupled standard of actual results—at other things, just not in some mystical way.

snoof - 3 days ago

I disagree with the last paragraph. Training is costly because data is costly. Reduced reliance on training is a significant development.

nfw2 - 5 days ago

One could say they might be "superhuman" here in the same way that LLMs might be said to be superhuman at math. Perhaps they do this work faster and more effectively than any person could manually. But they are certainly still doing it the "human" way ie reasoning with legible and coherent thoughts, rather than communing through some Matrix-like symbology which is what people who skimmed the article seem to be assuming.

lnenad - 4 days ago

We keep moving the goalposts, I love it. A few years ago LLMs writing simple functions was downplayed and now this. I lovehate the future this will bring.

jiwidi - 4 days ago

always one in the room

ahmedk29 - 4 days ago

[flagged]

braiain - 5 days ago

[flagged]

npip6969 - 5 days ago

[dead]