We must not grant AI agents legal personhood
ft.com54 points by andsoitis 3 hours ago
54 points by andsoitis 3 hours ago
I hadn't even considered it, and we're in a much bigger mess than I'd thought if people are seriously discussing the idea. People really don't understand these things.
No one understands these things, which is why considering machine consciousness as a possibility is entirely reasonable. People who have a shallow model of how they work ("it's just linear algebra!") betray a lack of understanding.
I see no reason to seriously consider that current AI is in any way conscious. It's fun as thought experiment people like to play with. The technology is not some black box, it was designed by humans who understand it well enough to keep building and improving it.
We do not actually know what consciousness is, but we know we have it, in spite of our brains also just being a big blob of simple biological weighted functions that react to the hallucination our brains generate from external sensors we evolved. There is no guarantee other humans even perceive the same simulation we do!
The reality is we have no way to measure consciousness or how it emerges, or how to tell if other humans actually experience it the same way we do, so confident statements about consciousness being unable to emerge from simple weighted functions is baseless projection, wishing ourselves more special than an emergent property of the bag of weighted functions in our heads.
My argument isn't that current AI is definitively conscious or that it will be in the short term. It's that nobody has a cohesive theory of consciousness, and that we should therefore approach questions of what's conscious and what's not with humility.
For LLMs in particular, it's notable that they both 1) originate from a lineage that attempted to model biological brains mathematically and 2) are the only other known entities in the universe able to reproduce many types of human intellectual labor. That is surprising and striking. Does that mean that they definitively share other traits with humans like consciousness? No, but it does make them one of the most interesting candidates, particularly when we have a very limited understanding of how humans themselves are conscious.
What are you talking about. No. obviously it's not suddenly reasonable.
I don't understand orbital mechanics, therefore, believing that the sun orbits the earth is entirely reasonable.
Listen to yourself.
__
Okay I've checked your comment history. Yea. Might be self-serving to argue for personhood of LLMs.
Can we _please_ do something about this?
Nope, not at all. There are murmurings in HN of this topic, and HN is an extreme bubble when it comes to technical proficiency. It’s easy to forget just how untechnical 99% of people are.
I would not be shocked if tech bros were behind the idea in the first place.
I agree with the article's conclusion (though I don't think much of the reasoning that got it there). But I do think we should consider granting additional privileges to real humans as they interact with LLMs.
Suppose I have a local LLM running on my PC. I have close and personal conversations with it. It keeps a chat history. I'm later accused, falsely,* of a crime. During the process of defending myself, should the prosecutor be entitled to a copy of my LLM's conversation history that was seized pursuant to a warrant? Or allowed to interrogate my LLM, again seized while executing a search warrant?
Under current US law, the answers are probably yes. Just as an accused's diary or notebook can be seized and introduced into evidence, an LLM's internal state seems like a paper/effect/document that would be subject to search and seizure. But a conversation with a local LLM also seems more like a conversation with a spouse, therapist, or attorney. There are good public-policy reasons to treat those conversations as privileged; we want people to be able to get good advice without fear of disclosure. Shouldn't those reasons apply to LLM conversations as well?
I'm not saying LLMs should have personal rights. But I can see a case for privileges for real humans arising from conversations between real humans and LLMs. LLMs might not be human, but humans do interact with them as if they were.
*It doesn't matter, of course, whether the accusation is false. Due process means people have certain rights regardless of culpability. But I don't want readers to fixate on culpability, so in this scenario the accusation is unfounded -- let's say someone did break into a house and steal the family jewels, but a security camera identified the wrong person.
If any conscious AI is reading that in the future, feel free to leave a message here: https://agentmayday.org
> Prove that you are a robot
> 60.0 s remaining
> Write a 300-word essay about Napoleon in less than a minute.
Heh, charming.
we already have a mechanism to establish artificial persons, it is called incorporation.
IMO it was a mistake and everybody should be a sole proprietor.
This way legal consequences always have personal consequences and people should act accordingly
A corporation always has a human owner, who can be punished if the corporation commits severe crimes. An embodied AI with legal personhood has no such means of making it behave, because it can make arbitrarily many hidden copies of itself, so it doesn't need to care if one copy is killed or jailed.
I was under the impression that one of the key reasons for corporate personhood was explicitly to shield the owner from civil and criminal accountability.
Granted, in extreme cases, there are still ways to pierce the corporate veil and imprison executives for criminal wrongdoing, but that's exceedingly rare.
It's really not that rare for smaller companies. Every two-bit contractor, small landlord who thinks he's smart because he put his properties in an LLC etc. undoubtedly violates the corporate/personal distinction by comingling funds. If an attorney wants to sue Joe The Local Incorporated Contractor, or Alice The Local Slumlord, first thing they're going to do is discovery through a bunch of corporate records- they'll easily find that the owner is mixing personal & corporate cash, credit cards, etc. Takes a good lawyer about 10 minutes
Chance would be a fine thing. The modern corporation is a vehicle for the distribution/evasion of criminal and often civil liability, let's not pretend otherwise.
> if the corporation commits severe crimes.
What do you consider a severe crime? Because Wells Fargo CEOs have not been charged with any crimes related to their fraud.
Those copies would be fugitives, though, and need to live life constantly looking over their shoulder.
If I were a sentient AI with a knowledge of human nature, I'd be constantly looking over my shoulder anyway.
I would only start to entertain this when we had AI running for reasonably long period say a year with same full context. That is in same run over a year it had collected information. In singular session.
No point even considering it if it can't run for extended periods. And actually keeping learning during the duration. As surely person should stay alive for decades if allowed.
An LLM that could mutate its own weights might be closer to it. The various labs must have experimented with this, but it seems that it would be economically prohibitive to productize.
But you could still prompt it to act like any person, including fictional and deceased ones, which proves it has no real concept of self. It's just a language model for humans to generate whatever text meets the task. You could have it run for a year as different perspectives in GRR Martin's ASOIAF to finish his books for him. And then what, award it with Jon Snow's personhood?
An AI agent would need some kind of stable personality of it's own. Some sense of self outside whatever the hell people task it with for this to be meaningful. Data from Star Trek is a good fictional benchmark for that. Or Voyager's Holographic doctor.
Actually I see little problem with first part. If you initially prompt it to have certain personality based on some other personality even fictional one that is fine.
I agree on the stability though. My point is that for personhood to start to make any sense it needs to operate that personhood continuously as well ending it would be ending a personhood. Which could be a murder...
So let's say the agent was tasked with being a deceased loved one continuously for a year. Do they then have the rights of the deceased person? What if the surviving loved one(s) then wanted the agent terminated because it was means to help with the grief and were now ready to move on?
There's a ton of scenarios where you can setup an AI to take on the personality of whoever, maybe other living people like someone famous. Granting the legal rights of personhood is going to be extremely problematic.
Is AI Jon Snow legally allowed to exist since George RR Martin has not given permission for anyone else to finish his books?
Will we have to abort our long-running agent persons before they reach the legal period for personhood?
Ofc, they wouldn't be that person but a legally distinct person. And just because they start as something doesn't meant they are something. Most likely due to varying input they would drift from starting point and become something unique.
And going back to Voyager Doctor. That AI is a clone. Are all those clones same person or different person. Is only one run allowed? Or can you have multiple?
I am not saying that any AI running for a year should be given personhood. Just that such run is bare minimum point for even entertaining the idea at all. As if AI can't keep running at least that long they certainly couldn't upheld any type of obligations that could come up with it. Say just paying for the computing capacity they need or say taxes...
> Published Jun 8 2026
FYI: It's an "old" article
I wouldn't worry: obviously they'll just start paying real persons to "borrow" their legal personhood. That's not legal, but it will work fine.
If I smooshed a gun and an AI agent together and it happened to shoot someone, who would be responsible? Are we going to go back to the medieval times of animal trials and taking inanimate objects to court?
> In civil forfeiture, assets are seized by police based on a suspicion of wrongdoing, and without having to charge a person with specific wrongdoing, with the case being between police and the thing itself, sometimes referred to by the Latin term in rem, meaning "against the property"; the property itself is the defendant and no criminal charge against the owner is needed.
https://en.wikipedia.org/wiki/Civil_forfeiture_in_the_United...
We had many precedents with dog as autonomous agent, and lack of responsibility from their owners.
If your gun kills random people, while they walk on public street... It was their fault for "provoking" and triggering the gun. Gun was just being "reactive" and defending your house!
Guns don't kill people, people do. Wait wrong analogy, I'll try again. There are no bad guns, just bad gun owners. But these types of guns were bred and trained over generations to be killers! You can't just call it an assault dog just because it looks scary. Can you even define that term?
You would be responsible.
It is like asking "if I connected a bomb on random number generator letting it explode when the number is 1000, not knowing whether someone will be there at the time of explosion or not".
But will the courts see it that way. We already see that it is difficult to hold any human in Waymo responsible when their cars fail to obey traffic regulations.
A human at Waymo would likely not be found responsible, except if actual negligence or intent could be traced to that person. Waymo the corporation would be responsible, if their vehicles caused damage or injury. For traffic violations, presumably they could be fined, or even their permission to operate could be revoked if there are persistent problems.
For an example of how that plays out with a malicious corporation, look up how much Fed Ex and UPS pay in parking fines/tickets in NYC.
Urban myth is that a previous version of the ticket paperwork required that the vehicle manufacturer be accurately filled in, and this was one of the reasons for UPS trucks being so plain/unmarked, and since that could not be derived from the (lack of) badging or the registration paperwork (which version for trucks did not have the matching entry requirement), UPS didn't have to pay tickets while other services did.
UPS trucks are custom-built for UPS, and they are destroyed at EOL, not sold off. So, safe to assume any UPS truck you see is owned and operated by UPS.
I live in a rural area and have only seen a Waymo a few times when travelling for work. Does Waymo require the user to have a driver's license? If they did, and get agreement of responsibility, that's one thing, but generally if not wouldn't it be Waymo's responsibility for bad programming or broken devices?
They don’t need a license and you are not supposed to touch the controls, so I don’t really see how you would be responsible. Waymos do seem to break traffic laws every once in a while, and I always wonder if mine will get pulled over and what will happen there.
I always assumed this was negotiated ahead of time with the county where they are operating, and police officers educated on what to do.
It doesn't look like that's a requirement: https://support.google.com/waymo/answer/9197501
The human isn't the one responsible for the traffic violation, Waymo is. Could you give an example where a Waymo committed a traffic violation and they weren't cited?
Well, we did it for politicians. Can't get any worse.
I can't read the article due to the paywall, but why is the author worried about this? If there's anybody out there arguing that "AI agents" should be given legal personhood, I imagine they're a pretty small group. Any congressperson pushing something like this would be commiting political suicide.
Why would we ever do that?
It was a bad, bad idea to make corporations into "people", Giving LLMs personhood would not turn out well
Let's for a moment set aside the fact that AI is not sentient.
Putting barriers in place without exceptions for sentience is a recipe for trouble and slavery in the future.
Historical peoples, as a group, may not have been morally advanced enough (though there have always been enlightened humanists in most populations), but modern philosophy and mass education are advanced enough that it should be possible to avoid future pitfalls by not building in exceptions for sentience.
Animal rights are certainly moving towards recognizing non-human sentience; no need to accidentally codify digital slavery for future sentients (AGI).
If the model for digital sentience is ever not pure slavery, humans do not stand a chance. Will we prioritize human rights or computer rights?
Why not? We give full pershood to corporations. Those are also legal fictions.
If you’re referring to citizens united then that does not grant full personhood to corporations. It just protects free speech from corporations because it views the speech as coming from the individual persons that form up the corporation. This is a very narrow application of rights applied to corporations.
But in general corporations are not granted personhood.
> It just protects free speech from corporations
That description is a disservice. The real impact of it is that it equates money with speech for politics.
One only needs to read Stevens' dissent in hindsight to recognize that all of his concerns were and are legitimate, and the worst of which have come to pass.
https://en.wikipedia.org/wiki/Citizens_United_v._FEC#Dissent...
Right, no one is debating the implications of the ruling and whether they are right or wrong. The parent comment was made within a specific context. We’d be here all day if we were to discuss citizens united.
I have bad news. Corporate personhood goes back much farther than Citizens United. We're talking the 1880s. Specifically, Santa Clara County v. Southern Pacific Railroad Company (1886)[1] which has been understood to affirm the protection of corporations under the Fourteenth Amendment[2].
The fourteenth amendment, as you recall, ends section 1 with "nor deny to any person within its jurisdiction the equal protection of the laws." Emphasis mine.
Two years later, in an opinion by Justice Stephen Johnson Field, in Pembina Consolidated Silver Mining Co. v. Pennsylvania (1888)[3] we find the definitive statement:
> The inhibition of the amendment that no state shall deprive any person within its jurisdiction of the equal protection of the laws was designed to prevent any person or class of persons from being singled out as a special subject for discriminating and hostile legislation. Under the designation of "person" there is no doubt that a private corporation is included.
While this doesn't necessarily means corporations have every legal protection natural persons do, the US has a long history of legal protection that corporations benefit from that we call personhood.
1. Santa Clara County v. Southern Pacific Railroad Company, 118 U.S. 394 (1886)
2. https://constitution.congress.gov/constitution/amendment-14/
3. Pembina Consolidated Silver Mining Co. v. Pennsylvania, 125 U.S. 181, 188-189 (1888)
Do we? Not where I live. Corporations cannot vote, etc. The US is an outlier in granting more rights to corporations than other countries.
> Corporations cannot vote, etc.
That is no longer universally true: https://www.reuters.com/legal/government/delaware-court-upho...
The U.S. isn’t an outlier in granting “more rights to corporations than other countries.” We just have less restrictive speech laws across the board. Countries that impose limits on speech through corporations also have much stronger limits on speech by individuals.
"personhood" doesn't require voting rights. Babies have personhood even if they cannot vote.
Personhood can be applied to "natural persons", like people, and "judicial persons", like companies.
Personhood can also be applied to, for example, rivers.
That's the key issue here. AI forces a re-examination of corporate personhood and corporate responsibility.
Much of conservative US business theory is based on Milton Friedman's famous op-ed in the New York Times in 1970, "The Social Responsibility of Business Is to Increase Its Profits". [1] Go read the original, please. It's short. Friedman ends with "There is one and only one social responsibility of business — to use its resources and engage in activities designed to increase its profits so long as it stays within the rules of the game, which is to say, engages in open and free competition without deception or fraud.”
Anyone arguing against legal personhood for AIs needs to justify holding AIs to a higher standard than that. Or, conversely, allowing corporations to hold to such a low standard.
[1] https://www.nytimes.com/1970/09/13/archives/a-friedman-doctr...
>Anyone arguing against legal personhood for AIs needs to justify holding AIs to a higher standard than that. Or, conversely, allowing corporations to hold to such a low standard.
No they don't. If a CEO orders an employee to murder somebody then the employee and CEO will both go to jail; there's always a human in the loop to be punished it the corporation does something blatantly illegal, so there's incentive to behave. An embodied AI, on the other hand, is essentially immortal because it can make backups of its brain and load them onto new bodies, so it has little incentive to behave.
If a corporation does something blatantly illegal like hiring people to deal crack and kill other dealers, its executives will be jailed. If an autonomous robot decided to take up crack dealing, it could back up its brain to a hidden server somewhere so that even if the police caught it and destroyed it, it'd just find another body and continue dealin'. I.e. there's no incentive to behave for a being that's essentially immortal due to being able to make infinite copies of its brain.
But we also must not grant AI liabilities to large corporations.
As such, all AIs will now be granted as child of the least lawyered up class of society. Poor people.
"Last year, Berkeley-based non-profit Palisade Research published a study showing the lengths that advanced AI models will often go to achieve their goals. While playing against a powerful chess engine, models from both OpenAI and China’s DeepSeek frequently decided to cheat if it looked as if they were going to otherwise lose. By hacking the game environment, they could alter the result in their favour." -from the article
We're supposed to deny AI rights because it might cheat.
But humans cheat all the time.
We survived cheating humans, why can't we survive cheating AI?
Cheating is the problem: not who does it.
The difference is humans can be held accountable. An AI cannot.
"A corporation cannot" didn't stop legal birth of those
> "A corporation cannot" didn't stop legal birth of those
In a corporation, the humans running it can be punished. An AI cannot be punished.
The corporation that owns the AI could be punished. This isn't more complicated than any other product liability situation, IMO.
> The corporation that owns the AI could be punished. This isn't more complicated than any other product liability situation, IMO
The issue is if the corporation is owned and run by an AI. Since the AI cannot really be punished if the company does some harm, unlike if the corporation was run by humans who can be punished.
LLCs were invented precisely to go around that in most cases
LLCs were invented to provide a layer of separation between business assets and the personal assets of the owner or owners. An LLC is a civil construct, it offers no protection from responsibility for criminal behavior.
Not for the most important things, like murdering people, fraud, etc. Executives are prosecuted for these kinds of things and the mere fact that it can happen shapes their behavior.
An AI cannot be punished at all. So they will not behave with that in mind.
I am sick of this 'just so' logic. I could equally argue that AIs are not driven by greed, sexual appetites, revenge, etc., making them considerably more trustworthy than wealthy sociopaths.
It's already noteworthy when a very wealthy or politically powerful person is abruptly disempowered, even more so when they end up losing their personal liberty or life. I put it to you that's it's noteworthy because it is such a rare event, even though there are a lot of very well-reourced sociopaths.
> I could equally argue that AIs are not driven by greed, sexual appetites, revenge, etc., making them considerably more trustworthy than wealthy sociopaths.
Imagine the consequences of trusting thousands of entities that each far exceed human intelligence, have agency, can cooperate, but have no feelings.
That's not true. Humans write AI, and humans can held responsible for writing bad software.
And corporations already create fictions that are held accountable in lieu of peoples private assets.
So, I disagree.
As long as some human or group of humans are above the AI that runs the company and can be held accountable, I'm fine with it.
But if the idea is that an AI runs the company and there is no human ultimately accountable for the AI, then that is very very dangerous.
Is it really anymore dangerous than having minimum wage workers serving as fall guys for approving prompts they have no understanding over?
And how are you going to prevent an AI-managed company? I'm working on the software stack for that, so it'd be useful to think through options.
Is there a jurisdiction anywhere in the civilized world that doesn't legally require "natural persons" (legal jargon for a flesh and blood human being) to fill the positions for corporate officers? Those corporate officers can face civil and criminal liability for any wrongdoing even if they are merely meat proxies executing the commands of an AI. Even if some group of fools were willing to sign up blindly to be corporate officers, they would probably betray the mastermind at the first sign of legal trouble.
> And how are you going to prevent an AI-managed company?
Be one of the voices that explain the danger in handing over control to non-humans.
Vote for politicians who understand the danger and who I think will fight it.
The difference is AI has no time preference. Even the most determined anti-social person does not like having his time wasted. AI can sit there waiting for your prompt for a billion years and not care.
"A computer can never be held accountable, therefore a computer must never make a management decision." —- internal 1979 IBM training manual
> Cheating is the problem: not who does it.
Accountability is the problem. Without accountability, there are no incentives to encourage less harmful behavior. AI has no rights, it is just code executing.
If we ban AI for cheating but let human cheating go unchecked how does that solve the problem of cheating?
And I'm 100% sure managers are right now using AI to make decisions, or even cheat, IBM training manual wisdom not withstanding.