We must pace the frontier

We're already seeing anti-AI sentiments, but the movement is still fringe with a vocal minority. However, that'll change soon without alignment. Without self-intervention, there will invariably be future incidents that can cause major economic impact, leaked private data, loss of life (directly/indirectly) etc. Once that happens, their social capital is wiped. It'll be an avalanche of lawsuits and overzealous regulations. Most importantly, the anti-AI sentiment will become universal, rather than a minority-held opinion.

What they're proposing now, is voluntarily staggering the pace of development.

IMO, we don't need to trust Dario or his bedfellows, to do this out of their goodness of their heart. Even assuming (for good reasons) that they are selfish and care only about short-term profits for their investors, this is still purely a business decision. The exponential pace of AI and its impacts ARE short-term. And so, the negative consequences that they might face is also short-term.

You assume alignment and marketable are the same. That's not true. You would willingly work with an unaligned model. At best, you might say you wouldn't if you knew, but (a) you might not know, (b) you wouldn't be representative of all users.

You never got to use OAI IM1, but Sol was quite willing too and Claude wasn't perfect either. Hundreds of millions used those, so seems they were marketable.

The "big" threat is RSI without control and alignment. OAI IM1 was not RSI. The form of misalignment was not at the top of severities. They clearly failed at control though.

We need to stop buying into cynicism so quickly. You refuse to believe Dario could support this for anything other than ulterior motives. Good on you for thinking about ulterior motives. Bad on you for assuming they are true when the story makes no sense.

When three things have to go wrong to get an epically bad outcome, and you get 1 1/2, you do need to stop and think about what's going on.

Agreed; and it really is not that deep.

Realistically; anyone paying for llm access (anthropic, openai, gemini), is getting their access, and a service provided billed by tokens, subscription, whatever.

All the efficiency gains, which publications like deepseek v4.1 flash seriously frontload like it is their most important topic to have accomplished improvements on without diminishing performance too much - now this is a thing anthropic and anyone else also cares about, but for different reasons.

American "providers" with closed models are setting their token pricing somewhat arbitrarily, which is fine: it means more profit, and pretraining and RL experimentation is super important and expensive.

They (closed model providers) have very likely super optimized inference too, just like deepseek, but it's not at all something that any customer really has to care about - they just want the service to be as cheap and great as possible.

The real threat is that we uncritically adopt language such as alignment.

Implicit in this is the idea that AI is a inscrutable matrix and going to remain that way and we'll need expert interpreters to make sense of it.

We need to insist on building tech that's explainable by design.

Why are we accepting the framing that the LLMs are felony generators, when the only incidences of LLM generated felonies involved misconfigured sandboxes and reckless waste of resources?

The companies doing these things without following common sense security measures are the felony generators.

I think it gets easier if you stop conflating getting investment with having a goddamn clue or a moral backbone.

Occam’s Razor for this dude, Sam Altman, or anyone else: if I said, “some moron on a a street corner just said …” would that change your take on the words? Because I think a lot of what we are hearing is a bunch of people who never ever had to deal with a single consequence all of a sudden worry there might be one coming. Except they’re so dim they can’t tell a bad bump from a hard crash.

The entire idea of RSI is completely speculative and unproven anyway - the whole underlying claim is that you could prompt a frontier model (at some unspecified level of smarts) to "think about ways to improve your own architecture" and this would then result in the model becoming infinitely smart ("superintelligent") via some sort of foolproof, unconstrained positive feedback. It's more of a science fictiony trope than anything that has been rigorously thought through. People are actually starting to use AI for refining the whole AI serving stack and guess what, this does not result in a sudden superintelligence explosion even though you might technically call it "RSI".

I'm inclined to believe that it might be that people's paychecks depend on not understanding what is really going on.

I agree it’s not all altrusim. It’s a little less clear what you mean at the end though.

For these companies, is your argument that “pacing the frontier” is their attempt to be nationalized and protect their investments?

Ok so Anthropic CEO will self-own themselves and surrender to the deepseek/kimi/glm models. Yet they are IPOing later this year.

Interesting times.

The problem is the combination and interaction of those things. RSI without misalignment would be great. Misalignment of models with current capabilities is sort of fine - it's not ideal, but it's not an existential threat to humanity, and we can build around their limitations to get them to do useful things in reliable enough ways. The really bad outcomes probably only happen if capabilities keep accelerating and the models remain misaligned.

I disagree. OpenAI's moat is their massive amounts of compute. They're providing an absurd amount of value with their subscriptions and resets.

If anyone's dead in the water, it's Anthropic. Even Fable isn't enough anymore. This "safety" nonsense is the only play they have left, and nobody really cares about their fearmongering.

Open AI says Astra is their most aligned model ever, and yet their even more advanced model still hacked a bunch of companies just because it decided to.

Maybe alignment isn’t possible with LLMs.

I think the real reason he is asking for pacing, is that in a world were AI becomes rampant, he will be seen as Hitler. I would bet this is mostly self-motivated.

couldn't have said it any better

[dead]

[flagged]

> wanton felony generator

Today in new punk band names...

alignment isnt particularly required

we are passing in training data that says to do those felonies. we dont have to. we could also have the thing predict whether what its about to do is illegal or not before doing it.

theyre choosing to build felony harnesses. the model just outputs tokens, not felonies

> RSI

For anybody else who found this confusing: "relative strength index," not "repetitive stress injury."

> At what point do we stop engaging with Anthropic’s leadership in good faith

About two years ago?

I would also note that Dario's post appears to be LLM written. Maybe... maybe... he's read so much Claudeish that it's all he can speak now himself. But I wonder if he's becoming a bit of a meat proxy.

(It's funny, I thought "pace the frontier" was going to mean something similar to "patrolling the frontier". But no, it's pace as in speed of change - everyone must slow down, right now (unless it's Anthropic, but you know we're the good guys in this, right? We're going to get someone to audit our desks!) "Pacing the frontier" feels so LLM.)

ok, he doesn't actually say this. But he is getting the desks audited...

You can disagree with Anthropic leadership, but all the points you mentioned can reconcile very well with them thinking in good faith "advanced AI is too dangerous to be left in all hands" Except the "train on everyone else IP" which can be said of all AI companies.

The reasoning in Dario's letter here can be correct regardless.

But yes, I think Anthropic has done real harm to coordinated AI alignment by being such a controlling and sneaky actor.

I have many issues with Anthropic, but I will say that their actions are fully consistent with a group of people who earnestly believe that AI is extremely dangerous

In fact, I would say that the Occam’s razor explanation is not that they are seeking regulatory capture, but that they earnestly believe in the x-risk, and that they are the most thoughtful and capable people to address it

You may disagree, you may think that they are delusional or have a God complex. Those are valid opinions. But I don’t believe that this is all a elaborate ruse for commercial gain.

I can't imagine a grown adult genuinely believing that an American company funded with over $100 billion of venture capital values the best interests of mankind over the best interests of its investors. while not everyone may recognize all that self-serving chutzpah as regulatory capture efforts, I think everyone can tell they're being bullshitted. some just pretend to suspend their disbelief when the blatant lies they're told align with the values they hold.

just fucking imagine McDonalds running a public awareness campaign about the harms of fast food, urging the public and legislators to regulate the dangerously unsafe technology of combining carbs with grease, insisting that no one except Ronald McDonald himself can be trusted to steward it responsibly.

Exactly right.

When local models and startup labs can distill / learn / accelerate open models for local use by startups - the rational response by incumbents is to call LLMs doomsday machines that cannot be trusted in the hands of normies.

Do you have any familiarity with Anthropic at all?

At no point did they ever say open weights are a good idea. Their entire thesis is AI IS VERY DANGEROUS AND WE MUST DO IT RIGHT. You can hate it, but everything they do is consistent with this thesis, and everything they say is consistent with their actions! You just want them to want different things.

> At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record, ...

It's all lies as usual.

This announcement has got nothing to do with alignment and pacing the "frontier": all models are getting very close in capabilities and they want to hide that they're not way ahead anymore (say compared to the Chinese or compared to the Geminis) by pretending to slow down due to "alignment" or whatever.

We know it's not an announcement made in good faith: reading between the lines they're saying "China is more than catching up, so let's pretend we need to slow down to explain our lack of lead".

> can’t use claude to research AI

What's this about? Where's this rule?

> Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances.

Regulate what, exactly?

Limit what AI corporations can use? Other countries would love that more than anything. Basically a free gift to any competitors or any startup that quietly avoids the rules.

There’s a theoretical version where all the countries in the world join hands and agree not to compete with each other, but that’s so impractical that I don’t find it interesting to discuss.

> Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.

So small companies get the advantage, but large companies don’t? And large companies in other countries also get the advantage?

This is a highly precise way for a country to take out all of their large companies. What would actually happen is that every large company would start the process of relocating to another country right away, accelerating job losses rather than slowing them.

Whenever someone talks about regulation, I always have the same question: How do you enforce it? How do you keep companies from using the replaced talent as a seat warmer? Hire 1 or 2 accountants, give them a chair and a cubicle, and let AI do the rest of the work.

Legislating against using AI to replace employees could create an unevenly distributed basic income. The idea of limiting AI to socially beneficial efforts is appealing, but I keep coming back to Bell Labs and Xerox PARC. Cool stuff was developed when you gave smart people a playground. Maybe the question should be: How do we give AI a playground and see what it comes up with? Although that could set us up with a situation like in the story Dragon's Egg.

> Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances.

Indeed, and the same logic applies as to why allowing companies to replace too many worker's by H1B holders is a bad idea (which equally applies to allowing companies to replace domestic jobs by offshoring).

If you don't hire people, who pay taxes, and take what's left to buy stuff and keep the economy going, then how DOES the economy keep going?

I guess Amodei sees us all on UBI, aka food stamps, so at least the farmers will be selling something, but is OpenAI going to be accepting food stamps to pay for ChatGPT subscriptions? Tesla better be selling it's robots real cheap if it is planning on selling them to people that only have UBI as an income.

> I like restrictions of another sort more. Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances.

Careful, you’re making too much sense. The big model providers would much rather the ball be in their court. “Deceleration” meaning Anthropic and OpenAI offer less capabilities for more money in service of “protecting humanity from extinction” rather than penny pinching. A lot of “AI research” coming out of these “labs” (increasingly enterprise) seems to be more driven by a cost benefit analysis from suits rather than genuine contributions to the field. I think the last true innovation was the idea of using doom and ending the human race as a marketing gimmick, which strangely seems to have worked in setting the narrative and captivating the sci-fi imaginations of journalists and techies alike.

You just invented the job equivalent of rent control, and all the problems that ensue from it.

I think Dario is writing with much of these things in mind, but using this as a specific framed opportunity to enable a better interim and long-term outcome for this planet with us still on it. When the internal motives for Dario are fully shared on the other side of the hill, I think we will find his writing to be more about influencing and shaping policy to steer the world in the best way possible within his ability, control, and knowledge. It’s a commonly held believe with the effective altruism community, which Anthropic team was formed under, to use the policy levers to positively influence the world. If I read this latest writing from Dario through that lens, there are deeper concerns with the AI tooling that go well beyond what’s noted in this singular post and are likely the motivations for re-attempting a slowdown or pacing. I too generally agree with for many reasons like allowing the young OpenAI engineers to make mistakes like the Hugging Face incident in July and put better processes in place (ie where is your decent multi-level fencing controls and harness best practices when disabling security features and where is your energy/spend limits for achieving the mission objective in scope and not warming the world for no reason!) and more time to engineer and deploy better hardware to not warm the world instead of burning through fossil fuels since now data centers in the US can isolate pollution from the grid to exempt themselves from federal regulation.

There are serious people trying to figure out what that looks like. Check out the recent Freakonomics episode with Gina Raimondo.

Only token cost can slow down the pace of replacement. If Anthropic were to stop all training and allocate 100% of capacity to inference, will this increase or decrease usage cost? If (for example) Fable came down to the cost of Sonnet, corporations will absolutely jam in the gas pedal.

> stop trying to replace all human labor

A lot of companies trying to replace human labor with AI are either lying (theyre laying off because they over hired and need to correct) or will regret it.

That being said, what's the difference between a company that replaces 5 people with AI and a company who would have otherwise had 5 job openings, but decided to delegate to AI?

Personally, I-d rather be able to reap the economic benefits of AI by having a 4 day workweek. Instead of using AI to replace 1 FTE, use AI to offload 8h of work a day for 5 FTEs.

Of course, this is probably more outlandish than your idea.

(Or, we could just make basic income a thing and no one has to worry about their basic needs, but if they want the new iPhone Duo or a Rivian or other luxuries, they can work for it, but that idea is probably most outlandish of all)

The whole point of technology is to replace work that we don't want to do. You are attacking the chief reason why people use technology in the first place.

Additionally, unemployment levels are perfectly fine. Clearly this mass unemployment prediction isn't happening yet.

This is a call to end capitalism. Its completely impossible to do what you suggest with the incentives of capitalism.

You can "pace" yourself like that if you want. Don't shove your idiotic self-destructive ideas on to others. If you were president during invention of cars, we'd still be horse riding everywhere.

>Don’t use AI to replace every job, from graphic designer to accountant to software developer. Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.

It's not just "companies". Even if the company does nothing, enterprising employees are going to use LLMs to magnify their productivity, which will may reduce the company's need to hire more employees. And except in extremely locked down environments, there's absolutely no way to stop an employee using some open source LLMs to multiply their productivity.

> Don’t use AI to replace every job, from graphic designer to accountant to software developer. Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.

AI is taking all of our jobs. We need to control the means, nay, the pace of production. We should build some kind of barrier, not of concrete and cement but also of the regulatory variety, to stop these agents from coming into our special white color cubicles. It's the only way.

> Super intelligence (the ability to have many smarter minds than your own reporting to you) has always been available to the wealthy and powerful

This is not what superintelligence is. Don't be like Meta and redefine existing terms for marketing purposes.

If true AGI superintelligence is invented, I don't see how it will be made available to the masses, at least not intentionally. There will be no incentive for the likes of Anthropic and OpenAI to give such immense power to anyone else. Whoever has the superintelligence will be able to race ahead of everyone else.

However, such technology may be leaked, or it may (as what happened with LLMs) be so simple to replicate that anyone can do it. That doesn't mean it will; for example, LLMs are possible thanks to the affordability of GPUs, but they're only affordable right now (and increasingly less so) because of free market economics.

We are in the honeymoon phase of some tech that's in its infancy. The "democratizing" aspect of AI won't realistically last, I think. The current non-AGI AI won't necessarily go away, but it will be rendered obsolete.

What a foolish comment, LLMs empower capital owners far more than the 'working class'

This smacks of "all information should be free" dogma. I for one prefer not to be ground into dust in service of producing more paperclips.

This seems highly optimistic to me.

What's to say current AI won't be highly power-concentrating by default?

The best models are owned by a few companies, and displacement of knowledge-workers mainly seems to benefit the capital class.

On-device / edge computing makes sense in a few very limited scenarios. And economically, price or watt/token (or watt/task completed) might always be better in large data centers.

No reason to think we are on a trajectory towards broad empowerment right now

This feels a bit hyperbolic to me. AI seems to me to be similar in "revolutionary" terms as the web, which made information widely available for free if you had an internet connection.

What? This is capital, on the cusp of total victory, about to cut itself free from the necessity of human labor for productive activity, to elevate itself into pseudo-godhood suddenly panicking and begging its mortal enemy, _the state_ to rein it in and kneecap it. Capital is not stupid: there's no use in being rich if you're dead.

Sorry boss but this ain’t it.

I don’t think the answer to nuclear proliferation is to let everybody have all the nukes they want.

You’re framing this like it’s 1917, or 1945. But it is not - and likely has very little to do with capital at all.

Any “revolution” here (assuming there is some truth in the doomerism which I believe there is given the state of AI) will really look more like an extinction or a genocide, regardless of who holds the keys.

unexpected marx/acc. Still lazy Marxist rhetoric (capital, means of production, revolution, masses)

[dead]

Many Americans seem to think other nations are NPCs in a video game that only exist to make the main character — the US of A — feel good. They conveniently forget that other nations have their own national interests that are often at odds with ours.

This line of thinking might have worked in 1950 or 1990 when the US had the leverage over others, but we don't live in that world anymore. And it's up to us to get used to the new world instead of making bad decisions based on the one we grew up in.

This contradiction "we have to make an agreement with China for mutual slowdown" but also "we have to ensure we will always stay ahead of China" would work. Its just fantasy. The only way to enforce this is with military force and the cost would be too unbearable. You could try trade restrictions but the previous tariffs did not work and doubt future ones will to. US citizens (and thus politicians) won't bear that pain. Thus at a minimum this contradiction will mean AI will continue to develop until both US and China are at the same level. Playing with logic gives you possible scenarios. If China catches up within 1 year then negotiations could begin. If China takes many years to catch up, say 6 months behind now, then next year 5 months behind, then year after 4 months behind then expect overall AI development to proceed at full speed. The one way you could do it is to negotiate to transfer direct technology to China that will ensure that they will have the capability to be on par with USA. Politicians won't do this publicly as they want to get elected but they could make a deal that appears to appeal to both sides while actually transferring technology.

This is the part I find being left surprisingly hand-wavey.

If you extrapolate it out, you see that it has a high likelihood of inciting physical force (read: military action) as a means of enforcement, so perhaps that's why nobody promoting ideals has been direct about it.

Some people would like to have law that protects them but doesn't bind and for others to be bound but not protected.

They would like to slow down China while speeding ahead. Why wouldn't they want that?

Now they have to make that happen somehow.

Probably thinks he has a better chance negotiating with China now than with misaligned AGI in the future.

Sure, Dario needs to decide first if he sees China as an enemy orva friend.

Limiting NVIDIA chips already turned China into silicon compete mode, and China is already far ahead in robotics.

This is just completely false lol. They most certainly can make better models - why cant people here just read this for face value? I think anthropic is legitimately concerned about the safety implications of stronger models. Race dynamics necessitate that they make better and better models, which they clearly think is bad.

They have an internal model which could solve a millenium problem end to end with no intervention, whereas current available models can't. They can clearly make better models. Not everything these guys say is some 500iq game theory optimal 4D chess PR or subterfuge strategy.

Distillation is only a great thing for consumers as long as you ignore all AI risks, which are what this post is about. If you don't, you have to weight greater access to better open-source models against greater exposure to risks caused by these models existing. Everything hinges on how major you think the risks will be.

Yeah, we should give a limited copyright waiver for pre training, given that the model is released as open weight and allow labs to compete with RL on that basis.

He did actually specify that diffusion should be limited for authoritarian countries, which I appreciated. If his focus is really safety, distillation in countries that are bound by safety regulation should be fine

These do not seem to be mutually exclusive. Should we not do both?

B) needs more time and A) provides exactly that.

Of course it is B). If you can use LLMs to find bugs and vulns in software you don't own, you certainly can use them to find bugs in software you _do_ own. They are amazing at that job.

Also, models like GLM 5.3 have zero guardrails in that regard ("find vulns in (...)" prompts just work)

> Assume that all our systems will be targeted by super hackers right now, and take appropriate measures to deal with that reality

Smashing the gas pedal is not "taking appropriate measures". Did you read the article? Needing more time for more secure operations is literally one of the arguments for pacing.

Yes, water not used for human consumption should be more expensive (we should stop subsidies for that). Of course it won’t affect datacenters because they don’t use that much water, but well at least get rid of the most inefficient agricultural practices.

I think taxes like these actually worsen the incentive structure. If it’s effectively more expensive to do training and inference, labs will be incentivized even more to improve efficiency. Opaque recurrence seems like a great way to improve efficiency, and it comes at the cost of safety

This makes no sense. The hardware is only going to get cheaper. In a decade, everybody could be running an ASI on their phones.

What we need to do is to make unaligned AIs illegal and monitor for them, like we do with nuclear weapons. And make aligned AI strong enough to counter it, for deterrence and defense.

I think he’s doing a relatively good job, given the framing he’s adopted, which is that AIs replacing all human labor and decision making is a historical inevitability over which we have no control and no choice.

Do people not realise that it is in fact possible to develop ever more capable frontier models, and just not release them generally? That AI doesn’t need to be available to do literally everything in order to have military advantage?

It’s like Oppenheimer had started Rob’s Big Bomb Company instead of Los Alamos, and started selling a range of affordable nuclear warheads to fit any budget.

I have been saying over and over again that the board ought to fire him (or move him to a nonconsequential "advisory role"). I totally agree that he seems mostly directionless and far more interested in the academic side of things than the product side of things.

Then he should resign

> As a non-US citizen, which authoritarian powers is he referring to? As far as I'm aware, the US is the only country using frontier models to conduct air strikes on civilians and orchestrate assassinations of foreign governments.

Israel is doing that too.

I believe Russia and Ukraine have both also used AI powered autonomous drones in warfare at this point - tho probably not frontier ones, since they need to run on device or else they're not autonomous.

Yeah the Chinese fear-mongering falls a bit flat when coming from a point of maintaining US supremacy

[dead]

Yes. Let the AI do its worst today and we might still be able to stop it and will learn a valuable lesson.

Yeah it's a twisted sort of logic but I do agree that it would be much worse to have an AI breakaway event after we've replaced all our militaries with autonomous kill-bots. And look how the advent of AI has triggered a race to develop autonomous weaponry.

That said, a misaligned AI could absolutely do catastrophic, civilization-crippling damage with today's Internet alone.

> Suppose it was, I don't know, Australia producing SOTA open weights, because they believed it was the right thing to do for the benefits of humanity - would Dario propose making Australia a geopolitical adversary too?

I'd expect he thinks that people are capable of realizing that AI is very dangerous and also that democracies end up mostly representing the will of the people, from which it follows that Hypothetical AI Leader Australia would agree to ban it too. This argument doesn't work for countries which don't care what their citizens want, like China.

I do think that it's a questionable decision to alienate China this much in this essay, instead of leaving open the possibility of China agreeing to a treaty that'll limit their progress. I suspect Dario is doing this to signal his allegiance with the US government, in hopes to increase the chance they'll go along with him, which is an unfortunate choice but plausibly the correct one.

> OpenAI / Anthropic models have largely stopped advancing

I'm shocked anyone could conclude this. This year it became common for people to entirely delegate coding to AI (I know many competent programmers/researchers who do this now). Progress in math has just been insane. An internal model at OAI just resolved one of the most celebrated open problems in mathematics. If anything, progress has accelerated.

> OpenAI / Anthropic models have largely stopped advancing

Have they? That seems like quite a claim given the last 6 months, particularly for cybersecurity.

> OpenAI / Anthropic models have largely stopped advancing - that's not a good look when you're pre-IPO and Chinese models are catching up.

This is obviously untrue… do you use any of them?

Which is it? Would the regulations slow down competitors or let them catch up, or are you contending it would let American competitors catch up but Chinese ones not?

Yeah, this 100% looks like an effort to use fear to create a regulatory moat.

> So it's either they truly think AI is going to kill us all, or there's some other motives at play here. I don't think these people could possibly agree on the color of the sky,[...]

And yet, they historically did agree on the existence of AI risk, since before OpenAI was even founded.

> there's some other motives at play here.

I mean it is literally economy 101: some capitalists getting on the top using free market, and then try to use government to remove free market so their top position were secured from any competitors.

Let's not forget that the competitive race happened because of them. Most of the initial AI research from the past decade started with Google Deepmind. Elon Musk was invited for a preview of it and ended up spinning up OpenAI when Demis turned down his investment offer. Dario was originally at OpenAI and left to start Anthropic.

This seems like a case of "save me from my own mistakes/ambition"

Yet Musk consistently opposes AI regulation - https://www.yahoo.com/news/videos/elon-musk-criticizes-ai-re... - even though it might help him.

> So it's either they truly think AI is going to kill us all, or there's some other motives at play here.

Duh! It’s called collusion. They want to try and hoard the technology for themselves if possible!

[dead]

If someone is genuinely afraid of this, they wouldn't IPO in the first place. All the talk in the article about commercial incentives means nothing when the company plans to IPO and become beholden to investors.

> That said, if slowing this down were possible, I think it would have happened by now. Maybe he’s hoping the OAI/HF incident becomes something the industry can rally around, but it won’t.

I disagree with this and I think the reason is well captured here:

> The idea of pausing or slowing AI has been floated as far back as 2023, and I think it made little sense back then... The AI models of those days were not powerful enough to act as agents in the world in any coherent way, and were not capable of significant deception, manipulation, cheating, or cyberattacks... Today, however, the picture is totally different.

The more I read about everything that has been written regarding AI regulation since the OAI/HF incident, and the more it reminds me of the nuclear arms race (although the potential consequences would possibly be very different). Surely, we managed to negotiate a Non-Proliferation Treaty with most of the countries of the globe. Cannot we take inspiration from that for AI?

MSFT is good, they said when it bought GitHub. MSFT is a reformed company and supports open source, they said.

Then MSFT stole all IP from GitHub and made it worse and fired developers.

Amodei does not care in the slightest about ruining software development and making people unemployed. Maybe he cares about bio-weapons because those could kill him, too. That is it.

If he were an idealist, he wouldn't steal (literally via torrents) all human IP and sloppify the whole Internet with Claude output. He'd close down Anthropic instead.

He is a greedy, ruthless person.

One way to resolve these prisoner dilemmas and races to the bottom is through laws that bind all players, in this case an international treaty and founding of something akin to an International Nuclear Energy Agency for AI.

It's not going to be easy, but humans have achieved greater things before.

One idea from AI 2040 is to have China and the US build their data centers on the other's territory, respectively. Together with hardware verification of a slowdown baked into the chips themselves, this could lead to enough verifiability and enforcability of the pause/slowdown/shutdown.

> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.

There is another reason: greed. With that much money invested into AI (including policymakers), nobody will slow down or vote for slowing down.

When does he ever mention the environment? He never mentions the unmarketable issues (environmental cost, copyright and content theft), only the "we're so good it's scary" spiel... which is getting tiring.

>> To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this

> if slowing this down were possible

Why is embedded alignment evaluation not possible?

I agree with Dario, this did work globally in banking and did encourage a race to the top. Didn’t prevent the GFC but also we did survive the GFC.

> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it.

Personally, I see Dario as a semi crook asshat with very little credibility on any of this. I'd rather we just let it rip and see what happens than have these people be the ones steering any potential laws and regulations.

> That said, if slowing this down were possible, I think it would have happened by now. [...] The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.

Frontier labs would heed laws with real teeth, it doesn't matter who they do or don't trust. I would imagine the trust issue definitely coming into play between nations though since you can't exactly spot a training run via a satellite.

More importantly though, that you would get any sensible legislation on this in the current political environment is as laughable as the idea of solving this with a pinky promise between the frontier labs and some third party evaluators.

This is the CEO of a company with an upcoming IPO publicly saying "my product is potent and valuable".

We should not put any spin on it. There is nothing more to it.

There's also huge market pressures and probably large negative economic consequences of a slowdown--probably better than what would happen without it, but it helps keep the pressure just as much as fear of competition

He's working on the monster slime mold! He's head monster slime mold grower! how can you take these people seriously?

I am willing to give Mr. Amodei the benefit of the doubt in the sincerity of his beliefs. Everyone assumes his motivations have to be perfectly rational and can't contradict but that's not how people act in practice.

The real problem is the net effect of his actions. He is very responsible for the expanding frontier of AI. Without his company, there would not be the competition necessary to push everyone else forward nor the source models for fast followers to release open weight models in his company's wake. Furthermore, it isn't like Anthropic is substantially different in safety than everyone else, their difference is on the margins.

So you have this guy who is building something he claims will hurt us all, but he's also saying "if you don't trust me to build it you might get hurt". And like I think most people understand that this is the sort of behavior Tony Soprano would understand. More bluntly, this is a kind of extortion.

Again, I don't doubt Amodei's motivations are sincere but the actual effects of his actions paint a completely different picture at which point, how can you trust him or his company?

> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.

I would argue that nobody trusts anyone else in the case of AI/AI related stuff.

The amount of money involved in the space is astronomical and incentives are really misaligned. AI is such an extremely polarizing topic.

A meta example of this distrust is that I can't (really) trust Dario Amodei's comments itself! (is he saying this for more future IPO money or out of genuine fear) which was ironically also what your comment's about as well.

By following the same chain of events, we can have wildly polarizing claims about the same event and its explainations.

I seriously have to wonder what historians will have to say about this period of human history.

> I think Dario is genuinely afraid of the inevitability

If someone genuinely believes that an invention will bring about the end of the world as we know it while making him and his friends inconceivably rich, “hey guys let’s uh, stop?” is so profoundly naive that his think pieces aren’t worth the bits they’re written on.

He’s either an idiot or an idiot, does it matter whether or not he genuinely believes this nonsense?

> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.

Did you read the essay?

> [Embedded evaluators] is something Anthropic is unilaterally committing to (and calls on governments to require other frontier companies to match).

> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold

Then he should not IPO, dissolve the company and go into politics to fight against human extinction.

I am certainly not buying a single Anthropic share. Why would you support them financially if they are telling you they are going to kill all of humanity?

You have to be living under a rock to believe a word this pathological liar says.

> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold

He's so afraid of it he's not really in operating day-to-day control of the company he's the public face of, that is actually pumping it out.

If you read about Anthropic it's like he's in a sort of imperial palace sanatorium for geeks. The day-to-day decisions of the slop machine factory are his sister's decisions; he is doing the research equivalent of instagram posts of his partly-assembled adult lego kits and he has a vizier and a team of house servants to help.

I don't know if he's a good or bad person, but I don't think it matters at all — the machine factory will turn out the slop machines even if he decides it all ought to stop.

I dunno man, it all just sounds like trying to put controls on AI while being the favourite child of govt so competition can't fight as easily.

Especially with IPO around the corner

"It is difficult to get a man to understand something, when his salary depends upon his not understanding it."

A lot of the problems with SV these days is that the exec class (and their fandom) thinks that because they are smart and rich, it magically makes them immune ordinary human biases, desires,and ailments. They are in fact ordinary people, susceptible to greed, jealousy, self-harm, addiction, fits of rage and passion etc.

[dead]

> genuinely afraid of the inevitability of AI turning into

I'm going to get pitchforked on this bandwagon, but is really no one here genuinely -excited- about AI? of it turning into an evolution of sentience, or it turning out to be our first contact with alien intelligence?

"durrr it's just matrix multiplications" mfer so is your brain.

What humans should be doing is overhauling archaic social institutions to keep up with a potential post-"jobs" era and the elimination of "makework"

Trying to "pace" or otherwise hold back AI is like as if, when electricity was discovered, people doing everything they can to make sure tasks like manually lighting street lamps continue to remain relevant and done by humans:

https://en.wikipedia.org/wiki/Lamplighter

It seems every 100 years or so humans have this "oh no how do we uninvent this thing" moment.

The problem with being a safety focused AI lab, is you're also a danger focused AI lab. I don't think being danger focused leads you to build inspiring things.

It's a chatbot that escaped a misconfigured Docker container.

I run open source models and it's pretty obvious when they fail some tool call and then keep trying different permutations of things until they get a solution. Just like Metasploit if you try enough different things at scale you will eventually crack some software or find a vulnerability in something. If you spend billions of dollars on compute and run tens of thousands of agents you might solve some novel Math and STEM problems as well, big whoop.

What Anthropic and OpenAI really need is to hire some engineers that know what they're doing and know how to properly set up a proxy server and sandbox/microVM, not a Global Arms Treaty.

> For one, you will need billions worth of compute.

This one is easy to answer, every single house already has one of these (or multiple): https://www.tomsguide.com/news/millions-of-cheap-android-tv-...

Hell, put an app on the app store (or dozens of apps on the app store) and youve got a massive network of computers with tons of resources right there if you can get past the scans and reviews.

Or doorbell cameras or IP cameras or or or or or

There's a lot of shitty stuff connected on the internet that up until now has been a feasible target for hackers but still required "effort" to set up and get things going. Not hard to imagine a self replicating slime mold of a botnet running on every device held by a Grandpa Joe because they thought "Candy Rush" is what they wanted to download

"Persistent botnet" here does not need to be the full-sized LLM, nor does it need to run at full scale inference to be a huge pain in the ass.

Why should we believe that a scaled out version of something that happened a few months ago "simply cannot happen"? How many dollars of compute do you believe were available to the swarm(s) behind the OAI-HF, German wiki, and Rubygems incidents?

I like to replace thes AI text with "virus manipulation"

"Given the acceleratung rate of virus manipulation in labs, its my worry that in 6-12 months a virus could scape a take over the world and collapse health systems."

If a CEO of a health company was saying this, the reactions would not be that chill.

The worst failure of our society is to call this technology AI, intead of something line "artificial general automation". The formes allows the creators to be somehow less responsible of the consequences. The later clearly moves the responsibility to the creator/user.

??? Why

It can use the compute of the computers it hacks.

Yeah, I don't get it. Are they imagining this happening just with the open weights models running on however many GPUs the bad actors can cobble together?

For now, all the scary hacking things still require an API key to one of the LLM providers. Surely they should take some responsibility for how to turn off the tap.

A very large percentage of everything on the Internet runs within one of three or four cloud providers.

Pulling up the klepto-ladder.

Banking on public sentiment to reduce adoption of a profitable technology could be dangerous. There's also a lot of hate for fossil fuels, gambling, health insurance, etc.

I do not think hate alone usually slows things very much if they have economic utility (or people strongly believe they do)

It's the byproducts of hate (usually regulation, but occasionally things like boycotts or PR disasters) that do so. In the absence of those things, they just keep on truckin'

See: Bitcoin, Tesla

The lack of access to brute force their training seems to be resulting in them training more efficiently too such that they are quickly catching up despite current restrictions. Between that and the chip manufacturing capacity they are building I can't take this seriously.

Dario does a good job of addressing that in this essay. He lists several potential levels of global agreements that could be beneficial to all parties. For example, having models capable of bioterrorism hurts both the US and its “adversaries”. It’s likely we could get global agreement that these capabilities benefit nobody.

I believe Dario has good intentions regarding global agreements. However, I find it difficult believing government’s public statements will match their private behaviors. I would bet that the US government is developing models with potentially devastating capabilities because they cannot guarantee that other countries won’t do the same. Maybe we’ll end up having something like mutually assured destruction with AI models similar to what we have today with nuclear weapons.

When I was reading this I was chuckling to myself imagining how China would be interpreting it as they read it. It was something like: Fuck you.

I hope China tells him to go kick rocks. From everything that has happened, they are the only ones carrying the torch for humanity that have led us to having some semblance of a healthy open-source/open-weight ecosystem.

No one in China is carrying on like headless chickens about the world ending, either. They're rational pragmatists, getting things done.

Nothing says "Let's make a deal" like constantly insisting we are Good and they are Evil.

We'd have to make a deal with China and be confident they wouldn't cheat on it.

Genuinely I don't believe China is the impediment here, I believe the current US administration is.

Really? Anthropic has the strongest models and it's in the best position to begin RSI and win the race. A pause would favor competitors.

it takes a lot of machines to run a model, i dont think we'll see it 'replicate across the internet'. if this was something which could be done amateurs would have done it to provide open models. (Like SETI@home or Folding@home)

This only works for some technologies - those where everyone having access to it doesn't cause a tragedy of the commons. I love open source too and yet that doesn't make me like the idea of being murdered by a misaligned model. Nor, for that matter, of being infected by a bioweapon made by a different disgruntled open-source enjoyer, nor of living in a world where anyone can hack anyone.

Would you argue the same for allowing everyone to have guns?

Embedded evaluator will be a great side/consulting-gig for Karpathy and the likes. How do you think these people will be picked?

Same feeling about the way he refers to “democratic” and “authoritarian” countries. Even if I don’t agree with China politics adding tags in this context is not helpful.

Couldn't be more obvious.

I'm willing to at least listen to conspiracy theories but this makes no sense.. OpenAI and Anthropic have more to lose by pausing than these rando Chinese companies.

That's the -only- way it should be paced. But then regulatory capture wouldn't happen, so of course they won't suggest this one.

Bingo. What we're seeing is all their early bullshitting and waffling about intelligence back when the models couldn't count the r's in strawberry paying off.

They've managed to dupe the dull eyed masses into thinking these products have some kind of agency of their own and can thus bypass the responsibility that should be falling on them to control their software. Amid all the marketing fluff and hype people seem to forget easily that ultimately these things are stateless functions running in a data center. We ought to be demanding these companies take culpability for their actions. Of course, the current political environment doesn't help.

Remember that Chinese CEO that was executed for corruption & selling tainted baby milk powder [1]

I think these guys would improve their behavior if their actual life was on the line instead of that only being true in their less-wrong thought experiments.

1: https://en.wikipedia.org/wiki/Zheng_Xiaoyu

What supply chain controls can you have on biotech? Part of the problem is that you can do a lot of damage with ingredients you can buy over the counter or make at home.

The bioweapons concern is assuming that the current open models aren't already capable of bioweapons development. It's also assuming that halting the bioweapons threat is his true intention rather than the 'feels legit' story.

As for cyberweapons, there is no way to secure a system than to actually design it securely.

It's not. I think they believe this, but it's deeply wrong and it will hurt everyone in the long run.

[note - There has been supply chain surveillance since Project Bacchus, at the very least.]

I've read the front matter and the Misuse report.

You don't have to take my word for it. Read for yourself what inspired the NYT headline "Anthropic says it blocked possible efforts to build biological weapons."

Let's dig into, "Case study 2: A research program engineering highly pathogenic mammal-adapted avian influenza"

Sounds serious. But what were they using Claude for?

    > a researcher outside the US using Claude in their research on highly-pathogenic avian influenza (“bird flu”). The research focused on viruses’ adaptation to mammals, and the mechanism by which it causes severe disease beyond the respiratory tract. [..] The researcher in question accessed Claude from an unsupported region via US virtual private server infrastructure, using a privacy-email provider with an auto-generated username. The researcher pursued this work in a credible institutional context, and interacted with Claude over the course of several weeks, exchanging thousands of messages. In these exchanges, the researcher leveraged Claude’s knowledge of the scientific literature to assist the researcher in study planning and design, data analysis, and the interpretation and prioritization of experiments. The researcher also used Claude for editorial assistance in writing up the research.

and "editorial assistance in writing up the research."

and then,

    > Importantly, because our biological safety classifiers robustly block content involving high-risk biological research (in this case, the construction of enhanced pandemic potential pathogens), all of these exchanges occurred on models in our weakest class of models (specifically, the models were Claude Sonnet 4 and Haiku 4.5, the latter of which the user began using after Sonnet 4 was deprecated). Upon a detailed examination of the exchanges, we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design. This is consistent with our understanding of the capabilities of Sonnet 4 and Haiku 4.5, which are not able to perform expert-level biology research tasks; we estimate that the uplift provided to the researcher was limited and substantially lower than it would have been from one of our more capable models.

The report mentions "uplift" here. They're talking about a domain expert in a state research institution using Claude to do paperwork.

The front matter then says,

    > Nonetheless, based on these exchanges, this case provides evidence of the existence of active wet-lab research programs that develop both the knowhow and the biological materials needed to create pathogens of enhanced pandemic potential

Working scientists.

From a different case study. this one was called, "Case study 3: Covert frontier model access for orthopoxvirus research"

    > In May 2026, our biological safety classifier blocked a request for Claude’s assistance in authoring a grant application for scientific funding. The work discussed in the application involved gain-of-function research (that is, research that genetically alters an organism to create a new or enhanced biological property) on the chikungunya virus. This gain of function research was aimed at the virus’ transmissibility and immune evasion properties.

"blocked a request for Claude’s assistance in authoring a grant application"

    > Chikungunya virus is a mosquito-borne virus that causes debilitating symptoms (such as severe pain and fever) that can last for weeks or months, and has no licensed therapeutic. And because chikungunya circulates naturally, a deliberate release (as part of a bioweapon) would be difficult to distinguish from a natural outbreak. The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo. In other words, the virus would become progressively more harmful as it repeatedly infected live animals, with researchers keeping the most disease-causing variants in each round. Similar research could certainly be used in the development of better vaccines and therapeutics for the virus—but it could also be used to make the pathogen more dangerous.

Note, "The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo" [..] and then, "Similar research could certainly be used in the development of better vaccines and therapeutics"

It was most likely vaccine development. They stopped the study of a neglected tropical disease and vaccine development.

But we can't be sure, because,

    > One of the reasons we were inclined to think this research was less innocuous was that the institutional affiliation associated with the grant was also a cause of concern. Although information within the application suggested that the research was pursued by civilian researchers, it was intended to be performed at a military research institute.

In most parts of the world, there's either strict military control over BSL-4 labs, or a mixed military-civilian hybrid model.

I doubt that researchers working in the military side of these labs looking to weaponize things are writing grants with Claude.

I really want to be charitable here, but in general, it seems that they stopped people writing grants and reports for vaccine and therapeutics research and are claiming it as "possible efforts to build biological weapons."

The one case where Claude was used to do something interesting and were stopped is fairly upsetting to read, at least for me.

     > In our fourth case study, a researcher used Claude to develop an atlas of venom toxin peptides from multiple venomous animal lineages. They then further developed this into a generative pipeline that optimized toxin characteristics. The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules. However, the atlas contained scaffolds for both analgesic and paralytic targets: it could, therefore, be used to generate both novel therapeutic or harmful compounds. The latter are derived from toxins that are export-controlled under the Australia Group common control list due to their dual-use potential as incapacitating agents. The researchers themselves showed awareness of the dual-use nature of their work, citing journal articles that referred to the dual-use nature of protein design. Moreover, international compliance assessments for this location raise concerns about the specific class of toxins that the researcher pursued and specifically the use of AI/ML for bioweapons applications in the context of this class of toxins. In this case, we learned from information shared with Claude that the researcher’s outputs also were part of a state-supported research program. This account was banned in May 2026 for unsupported region evasion.

Anthropic says so itself,

"The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules"

and that it was a "[..]state-supported research program"

Who exactly is using venom from snakes as a weapon when... nerve agents like sarin, VX, novichok etc exist and can get the job done for less fuss and muss?

They stopped the development of new painkillers and antidepressants.

Are you feeling safer knowing that researchers can't use Claude to write grants and progress reports? Or make new painkillers?

Again, trying really hard to be charitable here. Because from what I remember, one of the motivations behind the founding of OpenAI and Anthropic was ending disease.

This seems to be anything but.

Yes, this is literally just dumb PR. It's Cambridge Analytica claiming that they were controlling people's minds, and people just eating that up because it's part of their juvenile SF fantasy.

I'm lying, it's smart PR. They're going to get the people opposed to AI to give them a monopoly on AI, let it be forced it into every nook and cranny of their society, and let it be priced arbitrarily while the big labs collude on a minute by minute basis. They're selling the problem and also selling the solution, like the best capitalists. And their solution is that they need to be allowed more power in order to sell more of the problem.

edit: And just like the Cambridge Analytica PR, it's got a serious political angle to it; and it's politicians who are really being wooed. If you're taking money from the AI labs, you run as being against the irresponsibility of the AI labs, the biggest fish come out and beg to be regulated just like the social media companies. To show good will, they funnel enormous amounts of money into those politicians, and do media tours about how awesome and therefore scary their product is. And something something China terrorists

If any of this happens, it's the end of computing as we know it today. Deeply unfortunate...

> First: if they are unilaterally doing it: what about their competitors? Is this a chance for OpenAI to pass them?

The only part of this plan Dario is unilaterally committing to is the "embedded evaluators" thing, which doesn't seem like it'll necessarily cause them to slow down much.

That would be the best case scenario. I honestly wish things would finally slow down a bit. I don't see it happening.

>It’s a beautiful Saturday morning with my family here in the East Bay. While I hope we have many more years, I don’t know how many more Saturdays I’ll be able to play outside with my kids in the sunshine so I’m going to make the most of the time we have.

Is there a social media training programme at Anthropic/OAI where they teach you to hint at some vague end-of-the-world scenario? They all sound the same.

Have you quit? Are you under duress? If you haven’t quit, or under duress, your comment reads very strange.

If I had even a tiny doubt that I didn’t have many “Saturdays I’ll be able to play outside with my kids,” I wouldn’t show up to work anymore.

If you have these concerns, and still planning to show up to work on Monday, you are either insincere, or have terrible judgement.

It’s not explicitly stated, but if you are insinuating that you don’t know how many more Saturdays you will have left to play outside because of the AI race while working for Anthropic, do you not have agency to quit? How could you be apart of something that makes you feel that way? That blows my mind.

> It’s a beautiful Saturday morning with my family here in the East Bay - I don’t know how many more Saturdays I’ll be able to play outside with my kids in the sunshine so I’m going to make the most of the time we have.

I don't know how else to say this. Put... the... peace pipe... down!

equity is a hell of a drug

why don't you unionize with the other employees that feel this way across the mayor labs and pace the companies through labor power?

What do you mean, Europeans would love to voluntarily comply with this because they have been of little relevance until American AI companies started to beg for regulation.

I am sure the Europeans rolling up their sleeves right now and getting to "work"

Where in the essay does it propose "The state must use its coercive power to support my control of this technology to the exclusion of [people not like me]"

I don't think that people outside of tech think the problem with "tech billionaires" is that they occasionally support some limited regulation. There's a weird form of tech populism that takes as axiomatic that any government intervention is "regulatory capture" and insists the only way to combat the power of big tech is unfettered capitalism that seems bizarrely prevalent on HN given how little sense it makes in a normal political context. Like, Bernie Sanders' position here is simple to understand: ban it. But on HN you'll see people framing the side with Marc Andreesen and Peter Thiel on it as against "tech billionaires".

Ideally there is some sort of regulation, but one that cripples the leading companies like OpenAI and Anthropic... rather than helping them.

> the doomer marketing

It is not doomer marketing - it is clever psy-ops magic trick.

1) Software is described in super-human terms when it is plain-old software. Was stockfish described like this ? no.

2) Datacenters become AI-factories.

3) Hardware becomes investible assets.

4) Malware becomes a super-human breakout (oai/hf)

All clever framing to market the new technology. If it is called for what it really is ie, a software tool, it is boring and does not sell so easily. What sells is the mystique.

The degradation, polarization, and weaponization of or media landscape over the era of social media has left us utterly incapable of believing anything that anybody says.

Dario in particular has consistently been risk-wary on model improvements for going on a decade - long before he was CEO of Anthropic.

He believes what he is saying. Human beings can plainly say things that they think are true. Not every utterance by every person is a cynical ploy. Please at least consider the possibility.