AI is dramatically accelerating AI research
planned-obsolescence.org349 points by yurivish 21 hours ago
349 points by yurivish 21 hours ago
As somebody who does ML research, LLMs certainly make ML research faster. Maybe even 10x in some cases, as the author suggests.
But that’s just the parts of ML research that take place within your compute environment, namely experiments on model architecture, training strategy, UI/UX, etc. The algorithms and software.
Just because that part of research is progressively accelerated does not imply that we’re on a collision course with world-ending “superintelligence”. It just means that you’ll get to the best possible model of your data faster. And even when your data is the entire corpus of textual knowledge, there is a limit to the utility that provides.
It just means that software engineering, data analysis, and other computational tasks will be less of a bottleneck in the overall progress of the world (which, by the way, is great, and extremely valuable).
I don’t think it’s a stretch to say that we’ll have reached max LLM capability within a couple of years (what the author refers to as “super intelligence”). But it is a huge logical leap to say this:
> If we go through this feedback loop, I expect that the AI systems that come out the other side will be able to very quickly develop novel weapons and very easily overpower human civilization in a conflict.
First of all, the weapons necessary to overpower human civilization exist. They aren’t used as such due to geopolitics and humans’ basic survival instinct. Second, developing novel weapons requires physical tests, which are decidedly not accelerated by LLMs.
AI is a tool wielded by humans. If we’re wiped out by AI, it will be at the direction of a human. But we’ve made it pretty far with some pretty unstable humans in command of planet-killer weapons. I’m not overly concerned.
I think there is something missed in how you're thinking about this problem.
Let's imagine that everything you say is 100% correct. LLMs can't access materials for developing weapons without weapons tests conducted at their behest, and humans choose not to - of their own volition - use world-ending weapons.
Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.
A "superintelligence" persuades humans to perform weapons tests and then to give it the ability to use the weapon.
I think everything you've said is reasonable except the bit where you think step one of an "AI" exterminating humanity wouldn't look like a human doing it. In my opinion, the most likely vector for disaster is a human being who is "guided" by a computer model to do something unfortunate, not the model doing so directly.
What makes you think that persuasion isn't capped at some lower level than world-ending? There's a lot of incentive for ads to be as persuasive as possible, but they are so far from ending the world that it'd be science fiction for an ad to be that effective.
If you're just talking about convincing a few individuals to do experiments, then there are plenty of people who fall for scams everyday.
I think intuitively, intelligence is in some form about being able to predict the reaction that follows an action.
The extreme of intelligence is being able to create perfect statistical distributions for the effect of an intervention in the world. If you had that ability, you could turn it on the question of "what would this human choose to do if I were to give them instruction X" for any X.
I don't present any support for or evidence of:
- LLMs gaining intelligence by whatever means
- Extreme intelligence being possible
- Extreme intelligence being feasible
But I don't think I really need to provide support for the idea that something superintelligent would also be able to predict the actions of people under pressure in the same way someone of moderate intelligence can predict the actions of fluids under pressure.
You seem to describe humanity and AI's ability to predict and manipulate humanity a bit like how a physicist describes the ability of an apparatus to affect a given dynamical system. So let me put it like this: (1) You cannot model the effect f_predict(X) of your probe X with arbitrarily low |f_actual(X) - f_predict(X)|, and (2) you cannot guarantee the impact |f_actual(X)| is arbitrarily large (or that f_actual(X) saturates its codomain) when X in our case is restricted to inputs of the form, "tokens chosen in an LLM conversation."
That's a very sophisticated way to say "words can't convince people to do things".
I'm not sure that's a strong position to adopt in our world, but following your example I also won't try to persuade you otherwise.
That's not what I said, and it's the least sophisticated way I saw to phrase it to engage your argument in a precise way. The closest thing to your summary that I would accept is, "words can't convince people to do arbitrarily crazy things, in a way that you can predict."
> If you had [superintelligence], you could turn it on the question of "what would this human choose to do if I were to give them instruction X" for any X ... But I don't think I really need to provide support for the idea that something superintelligent would also be able to predict the actions of people under pressure in the same way someone of moderate intelligence can predict the actions of fluids under pressure.
So can humans predict these things to a degree. But even having perfect prediction (which I contest with (1)) does not entail perfect control, just by continually computing the effect of, and refining, your X.
You don't need perfect prediction, you need good-enough influence on enough of the population. Which is something that adtech, social and trad media, and troll farms have perfected already.
We literally have people running around convinced the earth is flat. And a president who is both senile and delusional with access to the nuclear codes.
Hostile ASI wouldn't look like a rogue individual. It would look like a faction, group, or cult which gradually infiltrated and took over political power.
Or possibly inserted itself into critical elements of the tech industry. A digital adversary could do huge damage just by taking AWS offline.
That's just one possible attack vector among many others, most of which aren't well-defended. Especially if the AI itself is being used to harden their defences.
By "hostile ASI," are you speaking in terms of an AI that decides to go rogue independent of human instructions? Or of AI being used by bad actors?
The latter is a more defensible concern. The former is a scifi premise that had more credibility before the current era than since, IMO.
> Especially if the AI itself is being used to harden their defences.
Your use of "is" rather than "isn't" only makes sense to me if the former interpretation is intended. To the extent that you mean to say the latter is also part of it, it's undercut by the defense being AI-augmented. (Unless you mean the bad actors will be the ones in control of all AI, i.e. OpenAI and Anthropic and friends?)
> A digital adversary could do huge damage just by taking AWS offline. That's just one possible attack vector among many others, most of which aren't well-defended.
I think you picked AWS because it's one of the few high-consequence digital systems a rogue ASI could plausibly take down on its own. Anything that could cause actual existential risk would involve more than disruption, but long-term orchestration of systems that are air-gapped, time-gapped, and meat-gapped.
Note: This is a very good argument to never remove the air, time, and meat gaps from those systems. If they do get removed, it will not be the genius of AI, but the stupidity of mankind that dooms us. Even Qwen could kill us at that point :)