I struggle to believe that AI R&D is ingenious
lesswrong.com211 points by porridgeraisin 2 days ago
211 points by porridgeraisin 2 days ago
Llms are far from being ingenious or innovative. The main way they can surprise you today is pulling in information that’s new TO YOU - I impressed with the algorithms they pop out sometimes simply because never formally studied compsci. But when it treads on my home turf I find it very basic, and for bigger picture stuff like system architecture its terrible, by its nature it doesn’t have a “vision” of how the whole system fits together
> The main way they can surprise you today is pulling in information that’s new TO YOU
Many comments here are speaking in absolutes and being myopic about what's "novel". The truth is yes, LLMs output is essentially derivative, but it's also true that a lot of what we consider novel is also derivative. The two aren't mutually exclusive at all.
Had a novel idea? You didn't get it from thin air, everyone is standing on the shoulders of everyone who came before. LLMs most of all, but that doesn't mean it can't bring disparate bits of word-encoded knowledge together and burp out something we'd consider "novel" if a person had made the same connections.
That was actually the basis of some of the mathematics proofs. It made connections from existing material. Nothing a human couldn't do looking at the same material, but we're talking speed and scale here - same as all computational processes.
A calculator does math faster but we don't call it sentient or feel inferior or dismiss the results. The only difference is LLMs are intentionally nondeterministic, as that behaviour can be usefully harnessed, pun intended.
So there's no need to argue about what's novel or not. LLMs can make novel connections via raw speed and breadth of disparate data. The human brain is more complex and "imaginative" for whatever your definition of that is - I mean fundamentally one either believes we are just physical neurons, or that there's a spiritual aspect on top, the end result is the same.
A six sided dice can also be novel if it happens to unexpectedly roll your phone number. It's a novel occurrence, same as an idea coming to mind that, TO YOU, seems novel... but we can't ask the other 8 billion people on the planet so who are we to judge what's a novel idea anyway?
This is a common rebuttal. But it falls flat, in my opinion, because LLMs appear to be a compression of language-based knowledge, and that compression is lossy. The result is that "knowledge" retrieved from the language-based storage model is biased towards the "average" idea.
As a concrete example, when you ask Claude Sonnet 5.5 to generate a small-to-medium-sized one-shot Python project, it will always fall into the same code patterns. Same structures of functions. Same file layout. Same package layout. They're not even good patterns, because if you throw a few common linters at it (e.g. flake8, isort, mypy) there are a ton of linting errors to clean up.
The only way that it escapes this mediocrity is through context. If you give it enough guardrails or sample data (e.g. Python code to emulate), it will start to go off in a "novel" direction that is defined by that context.
Now, what's the most likely way that "novelty" will be generated in an automated system like this? Randomly pulling in context and going for a random walk through the compressed language-based knowledge storage. It remains to be seen whether that's an effective way of producing more than just parlor tricks for investors. Human+LLM is overwhelmingly more novel and effective than only LLM.
This doesn't even address the direct point. The question is how any of this is fundamentally different (difference in type and not just of "scale") from how human cognition works.
Well, I've addressed it implicitly. Every human collects different context by living one of 8.3 billion+ unique lives. We don't have 8.3 billion+ different LLMs. The only way to simulate that is to perform a random walk through the compressed language-based knowledge storage by injecting random context.
Also, those 8.3 billion+ unique lives already produce contexts that are highly optimized for solving human problems that modern humans experience in the physical universe most relevant to humans.
All knowledge storage and retrieval is lossy. Oral histories are lossy. Lectures are lossy. On on one mentorships are lossy, writing or reading a textbook is lossy.
What's your point?
Agree with this as language use is inherently an attempt to compress observations into a shared semantics
The syntax of math is overloaded so you need less syntax to write out something that would be verbose in English ...no one wants to make sense of arithmetic expressed in German syntax. May as well write all software in brainfuck.
Written literacy on a core language like we focus on today is not the historical norm. Creoles and local dialects have vanished in the US as we hyper-normalize everyone into Children of the Corn like carbon copy zombies of character traits
If you want to argue, as I think you do, that there are no new ideas other than new combinations of old ideas, then you have to explain where the old ideas came from. It can't just be combinations all the way down. At some point, some initial set of ideas must have been created where none existed before otherwise we would have nothing to combine. After all the combinatoric closue of ∅ is ∅.
Ultimately any claim that there exist only combinations of old ideas must tackle this chicken-and-egg problem convincingly. For example- explain art. There was no art until it appeared fully formed and out of nowhere in early anatomically modern humans' dwellings (Neanderthals also had art but it was simpler and it is subject to the same chicken-and-eggness). So where did art come from? What pre-existing ideas were combined to bring it into existence from non-existence?
Btw, since I've had this conversation before- if the answer is that we looked around us in the world and saw things; yes, but we didn't see art. Where did art come from? Where did the idea come from of abstracting shapes and colours so that they resemble the shapes and colours of existing things, without actually being those things, or being composed of pieces of those things? How about numbers? How about writing? How about all our technology and science? Where did those things come from? What did we combine to create them?
Obviously I think that the argument that there are only new combinations is wrong: we are a creative species. We can create new things. Somehow. It would be very interesting to understand how. We have not yet created machines that can match us in that and we can't understand the "how" by studying the machines we have now. By denying the mystery of this our ability, by denying that it is a mystery, by dismissing it and accepting any facile explanation that presents itself in our current era -it's god, it's nature, we're like LLMs- we are making ourselves a great epistemic disservice, denying ourselves the chance to find out what it all means, to understand what kind of creature we truly are.
> then you have to explain where the old ideas came from
Its fairly plausible that the original ideas could be sensory gradient-triggered impulses in simpler creatures' proto-minds that (physical and cultural) evolution has gradually built up through more complex neural machinery supporting more complex representational language.
> where did art come from
Similarly, its not a stretch to see art as built from earlier attempts at representation in physical space - starting with mundane attempts at information exchange by colony insects or pack creatures. The breadth and depth of expression has evolved alongside more complex neural machinery and greater variety in media through technology. There's a hidden lineage in creatures' expressive capabilities that may not be evident when we look at a Caravaggio today, sure.
There's nothing terribly chicken-and-egg about these issues (though the resolution is the same - "chicken" and "egg" co-evolved through iterative selection pressures, one didn't precede the other).
All that said, I concur that its a fantastic and mysterious topic to unpack, but it doesn't follow for me that humans are anointed as uniquely "creative" or that there's necessarily something metaphysical at play.
Truly new ideas are, say, 0.01% of all 'novel' ideas, with the rest being rehashing or recombinations. And even then, the truly new kernel of the truly new idea is some small fraction of the whole thing. Like Einstein's theories of relativity, they were truly fresh, but mostly had to recombine existing work in 'novel' ways.
So it's not there are zero new ideas, but so few compared to the whole of the intellectual domain, that we can say 'there are no new ideas under the sun' and be basically correct to within a small rounding error. And AI is sufficiently ingenious to serve the needs of the vast majority of humanity, even though we'll still need to find/cultivate/inspire some new genius humans if we want to have more truly new ideas in the future.
>> A calculator does math faster but we don't call it sentient or feel inferior or dismiss the results.
Actually, we do. We do dismiss calculator results. We are all generally aware of a calculator's abilities. We know it has limits. So when a calculator outputs 9999e999999 we know to distrust that result. It was something out of the calculators concept of understanding. So too a calculator's version of Pi. "3.14159265359" is not Pi and every user knows it is an approximation not be used for anything too important or precise. The same approach should apply to LLMs... but their limitations remain nebulous and ever-changing.
Most calculators use 14-15 digits of pi even if they only show 10. It would be hard to come up with a case where for single calculations this was not “sufficient”.
Radians? Do any math where Pi is a base number and the errors can cumulate quickly, depending on where you insert the pi approximation.
I agree with your opinion. The definition of novelty largely depends on subjectivity. Results produced by combinations of LLMs can also be novel. Combining multiple insights, imitating them, and then organizing them into one can also be considered sufficiently novel. LLMs have a kind of bias due to RLHF, and that may make them uninteresting to humans, but when a "human" uses them, they can enable the human to do more, and that greater amount of work can create new novelty.
Strong agree. My 30 years of experience building computers and breaking them has made me better because I don’t trust my knowledge or myself as a first principle. If I didn’t have that and I could easily see myself in an alternate reality, coming in late to the game to leverage ai to build a car out of seatbelts; harnessing all the power of 1000 suns seeing that idea to fruition.
Gell-Mann Amnesia Effect:
> Briefly stated, the Gell-Mann Amnesia effect works as follows. You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward-reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them. In any case, you read with exasperation or amusement the multiple errors in a story-and then turn the page to national or international affairs, and read with renewed interest as if the rest of the newspaper was somehow more accurate about far-off Palestine than it was about the story you just read. You turn the page, and forget what you know. —Michael Crichton
Edit: sibling comment explains it much more plainly.
I have noticed that several times with a newspaper I had at the time. It seemed to be a generally good and reliable newspaper, and one of the biggest in the country, but every time an article was about something that I actually knew about, it was wrong, which made me doubt the validity of everything else.
I now have a newspaper that doesn't give me that feeling.
I've come to the conclusion that they know a lot, but they're not very smart. Not even the best ones.
They know far more than any human ever could: they've read all the books. Their knowledge is enormous. And that often includes how to apply it in standard ways, and sometimes in slightly non-standard ways. But whenever something gets complex, detailed and nuanced, they can't keep up with a human. It becomes a shallow illusion that's easily punctured by just a bit of critical thought.
You mention system architecture, but even for writing code, the code it writes is often horrible. It makes it work because it has a feedback loop that rapidly tests it and ensures it will work, but the actual code it writes down is often not something I would feel comfortable showing to another programmer.
Strong agree, but with a major caveat: First since we humans don't have a memory the size of the internet, new TO YOU is extremely easy to find. And second: a lot of discoveries can be made by trying to match two different domains and generate new insights at the seams.
A lot of math theorems are discovered that way by humans btw: find a difficult problem in a domain, map it to another domain, apply existing theorems there, go back to the original domain. The only difference is an LLM would have to search for an interesting problem, but that can be done by brute-forcing search at the seams and validating the problem does not previously exists. And there is no new TO YOU for the LLM: it can get help from a vast amount of helper domains.
IMHO there's a shot at having category theory really unifying a large chunk of maths in the next decade. It'll be a big ugly ball of Lean, but a queryable one at that.
And then there's all the math problems if solved where the information is new to every human on earth.
This is not novel. Computers have been generating mathematical information that would be resource-prohibitive to produce by hand since the dawn of computing.
The ingenious aspect seems to be that we've found simpler ways to apply computing power to a broader range of problems, which it's important to note is as much a testament to the developers of proof assistants like Lean as it is to LLM developers, to say nothing of the countless mathematicians over many centuries that contributed training data.
Whether this will lead to many revolutions, or a handful of impressive results followed by a long dark spell remains to be seen.
I, for one, wouldn't place large bets on either.
LLMs are like an athletic child - they can do some basic moves, but in general they're uncoordinated.
Occasionally they do something that surprises you. If you put them inside a search loop and spend huge sums of money you'll get some nuggets of value, but for most hard problems the cost isn't worth it.
And they work best with a lot of steering.
From that POV they're exactly like human culture. Most scientific papers are trash, a few are gamechangers. Same with the arts, etc.
But we've gone from 'Can write sentences' to 'Can do frontier math problems' in maybe five years.
We absolutely do have AGI now. It's not very good AGI, but it's probably better than a lot of people were expecting.
Being better than humans at everything isn't AGI, it's ASI. A lot of the comments here seem to be lamenting that we don't have that yet, which is an interesting data point about how far the goal posts have moved.
Edit: also it makes things like this possible. So...
if you search papers with it, and from there search related papers and have it do literature reviews, it comes up with very useful white areas uncovered, and from there problem statements and experimental designs to uncover the white spots.
it is nothing but combining known knowledge that begets new.
> by its nature
Declarations about what LLMs can't do "by their nature" have a terrible shelf life, historically.
Unless LLMs drastically shift incredibly quickly, said declaration is likely to hold, because of how LLMs work. I mean, they've worked the same way at their core for several years now.
Their nature hasn't changed but their capabilities have. What we thought was impossible due to their "nature" keeps shifting.
They predict what comes next and are very good at it. That's all intelligence is. What comes next in a pattern? What will my senses perceive next when I drop this ball? What is the next hypothesis to test after I made these measurements in the previous experiment? What is the best next move in this game after the current trajectory?
Maybe. And while this may be a useful superpower, it's hard to predict whether it would be an enjoyable one.
> by its nature it doesn’t have a “vision” of how the whole system fits together
I think a harness for building hierarchical summaries with tooling for expanding and searching history is what's missing for this. We're getting there with Recursive Language Models and continuous compaction systems like the one NVidia just published:
This stokes my suspicion that effective LLM input for software development will ultimately resemble more of a higher level programming language than the natural language in common use today, and that the design of such languages and related tooling will play at least as large a role as the models themselves.
which is actually great, because LLM so far does not show true AGI ability, they are not innovative. Humans are still needed so everyone still have a job. And you can delegate the stuff that you need to be careful with, or attention to detail stuff, to LLMs
Most human jobs don't require innovation.
On the other hand, there's no evidence that work will dry up merely because it becomes more efficient, or even if it does, that this cannot lead to a better world. The trick is finding a way to prevent rapid changes in the nature of work from destroying too many lives, and that's a very human problem machines alone can't solve.
I'd argue it is the other way around - LLMs are great at pure creativity, creating something that never existed before, but not always at "pulling in information that’s new TO YOU". I use them for entertainment - jokes, short flash fiction etc. and they are excellent at it.
What?
They literally generate stuff that's averages of other stuff.
It's not interesting in the least
They're terrible at jokes. I bet any good joke they've given you is simply new TO YOU.
I tested this. Out of 200 attempts I got 2 novel jokes that got good amount of upvotes on r/jokes.
Usually commentators say that they have heard so and so version of the joke, or that it's a repeat, but they did not
Both from opus. Sol/astra could not be funny. The versions that worked I had fed top posts from r/jokes as references and gave abstract directions.
However, the core premises were not new even if the construction was. And it's a lot of work to get something useful out.
I can believe this is true. It's great at going out there, connecting ransom dots in very clever ways, sometimes stumbling on something actually funny.
I'm curious, did you test with the 200 jokes if it could guess which were the beat ones according to a human audience?
In images, music, games, it usually can't. So the process of validation is offloaded to the audience. I'm very interested in seeing where this takes different platforms. I bet Spotify can pull off such a validation process. Music is passive, tolerating a bad song doesn't require much from you. And they have a massive audience. They can try out AI generated mess with a very low success percentage without saturating their playlists and alienating their users. But can Steam tolerate it? Trying out a game is a much bigger effort than listening to a song picked for you. Customers are already reacting to the number of demos in Steam Next Fest. The saturation point is much lower.
And what about jokes? What happens to r/jokes if even 10 standup comedians are validating 200 jokes per day?
And we see it in software too. QA is already being offloaded to the customer.
I want to see where this goes, even if I don't like it.
But if you find a media where AI can also validate the output instead of having to offload it, that's going to be very valuable!