Artificial intelligence is not conscious – Ted Chiang
theatlantic.com664 points by lordleft a day ago
664 points by lordleft a day ago
https://web.archive.org/web/20260603173839/https://www.theat...
https://archive.is/bcpZl
This entire essay basically boils down to one simple contention: LLMs can’t be conscious because only humans can be conscious and LLMs are not human. What an unimaginative argument: essentially ruling out the possibility of the title by definition. And the author fails to touch on the really interesting question posed by the article’s title: what if human consciousness is more like the working of LLMs than we think? When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious.
And here in particular the reasoning behind the argument is bizarre. Decomposing the complex activity into simple steps like 'predicting the next word' and claiming that surely can't have consciousness. A similar argument would be -- there is no way that movements of electrons by tiny distance would produce consciousness. The danger of anthropomorphism is not we elevate the machines, it's that we debase humanity. I also think different ideas get conflated. It may be possible to build a machine that is super-human in the sense it can outperform the human brain in all kinds of measurable ways. Does not imply it possesses all the same qualities of the brain. I respect a number of things Anthropic has published about the ethical issues at stake. But, having an in-house philosopher does invite you to make all kinds of unfalsifiable claims. I think humanity deserves to be debased in part because it's awfully egocentric and insist on being special in the universe. There are numerous times when humanity's been debased. Copernicus, Kepler, Darwin, and the infinite number of animal behaviorists who have defined and documented consciousness And how it manifests in primates, parrots, dogs, cetaceans, and insects. (I just love the bumble bees taking time off from work to play with balls). It takes humans a really long time to accept the fact they are no longer as special as they thought they were And make room for yet another conscious intelligence. So, if we had an AI demonstrating symptoms of consciousness and suffering, how long would it take for you to accept that it is? >I think humanity deserves to be debased in part because it's awfully egocentric and insist on being special in the universe. It is special in the universe unless proven otherwise. Aside from the animal world (also on Earth too, anyway) which is a cruder version of humanity, much more violent as well, the rest of the universe thus far is just uncaring rocks and gases. >It takes humans a really long time to accept the fact they are no longer as special as they thought they were And make room for yet another conscious intelligence. A "really long time" is the few years we've built LLMs? Which we have to take for granted (on your word?) they're already "conscious", and chastice ourselves for not already "accepting it"? >So, if we had an AI demonstrating symptoms of consciousness and suffering, how long would it take for you to accept that it is? "If we had a parrot demonstrating symptoms of discussing with us, how long would it take for you to accept that it is?" > much more violent as well I'm not sure about that. What non-human living entity does things on a scale and violence akin to "I'll fill this ant nest with liquid burning metal for artistic purposes" ? A blue whale eats a lot of krill individuals for sustenance, a dolphin can only rape/kill one thing at a time, etc etc. It is very common for animals to kill much more than they need: https://en.wikipedia.org/wiki/Surplus_killing. Elephants, which are herbivores, sometimes enjoy killing rhinos for fun: https://www.bbcearth.com/news/teenage-elephants-need-a-fathe.... House cats recreationally kill billions of birds and small mammals every year that they don't need or want to eat. Chimpanzees have full-scale wars: https://en.wikipedia.org/wiki/Ngogo_chimpanzee_war > House cats recreationally kill billions of birds and small mammals
> every year that they don't need or want to eat. "Recreationally" is carrying a lot of weight here. I suspect that cats kill birds and mice because that's their instinct; it has nothing to do with conscious thought, much less a need for recreation. And that probably is the explanation for most (maybe all) of your other examples as well. Why do you believe this? Do you believe cats and other animals have no consciousness, so every behavior they exhibit is just instinct? Or do you believe they have some conscious behaviors, but killing birds is not one of them, this thing in particular is just an instinct? For the first position, I think it is quite clear to anyone who studies and spends time with animals that they have something that is at least of the same kind as our consciousness. I just don't see how you can ascribe the wide gamut of complex, situatuonally and mood appropriate but still varied behaviors of animals to being purely instinct driven. For the second position, I would like to see some study or some rationale behind it - especially since cats don't kill every bird they encounter, so if it's an instinct, it must still have some trigger, and hunger is not a viable explanation for most of the killings referenced here. I just meant recreationally to mean "not out of necessity for survival". And I don't think conscious thought is relevant for this specific thread, I was just responding to the question of whether other animals besides humans can be needlessly violent. Animals generally have no qualms at all about killing or even just mutilating other animals. It often happens almost by accident - two animals might be playing together, one gets spooked, and it instinctively attacks and perhaps even kills the other one - this is commonly seen with people who befriend large predators, such as tigers in the infamous Siegfried and Roy tragedy, but it also happens a lot wherever animals interact with each other. Specifically in regards to your ant example, anteaters and bears often bring similar levels of destruction to ant nests. And cats and other small predators often hunt just for the fun of it, killing but not eating their prey. On the more purely painful evil side, invertebrates often consume their prey alive, inflicting agonizing deaths with no issues on whatever they may be eating. Plenty of vertebrates kill and consume their babies, especially when frightened. They also often abandon old and weak members of their packs, leaving them to die of hunger or cold or similar deaths. This is not meant as some indictment of the animal kingdom - people do all of this too, of course; and have since time immemorial. It's just to show that, if we apply human moral standards to the animal kingdom, it's fair to call them violent, and yes, even much more violent than the average modern human. >Animals generally have no qualms at all about killing or even just mutilating other animals. Humans generally don't either. Individuals do, but as a species humans regularly kill other humans. >invertebrates often consume their prey alive And humans use the oh so humane factory farms. >cats and other small predators often hunt just for the fun of it, killing but not eating their prey. >On the more purely painful evil side, invertebrates often consume their prey alive, inflicting agonizing deaths with no issues on whatever they may be eating. Sharks are caught en mass, the fins cut off, and the sharks dumped back into the ocean to slowly die, for shark fin soup. Have you heard of trophy hunting? Have you seen the pictures of mountains of bison sculls for the American West? >Plenty of vertebrates kill and consume their babies, especially when frightened. Even in our modern "1st world" society, scared teens still abandon newborns in dumpsters. Many societies throughout history did not consider babies "real people" until a certain age because they may need to abandon them if resources were particularly scarce. >They also often abandon old and weak members of their packs, leaving them to die of hunger or cold or similar deaths. Maybe you should read stories of cities under siege, famines, wars, governmental collapse, etc. Humans now live nice comfy lives most of the time, unlike animals “in the wild”. Human societies that lived closer to the edge of survival made callous choices about life or death you are spared from. I agree that humanity is guilty of all of these, and has done all of them at a much larger scale. I think I was pretty explicit about this in my comment as well. My point was that we call humans who do this "violent" and even "evil". If we want to avoid considering humanity as special compared to the rest of the animal kingdom, as some in the thread were suggesting, then we have to either admit that animals are also violent and evil, or say that humans aren't. Note that I don't hold this view, personally, and think that humans are unique among currently living animals, and that these labels only make sense to be applied to humans. But not because of behavior, simply because humans have a unique level of both understanding and control over their actions - as proven by the many billions of humans who have never in their lives killed a human or even another bird or mammal. Humans have enough cognitive ability to stop themselves from killing for fun (so when they don't, we deal with them using human invented laws), while anteaters eat ants for nutrition. Animals in general cannot reason at a high enough level to avoid instinctual behavior. > if we apply human moral standards to the animal kingdom Which we should not, since human moral standards are for humans. Animals can at best behave in a way that suits us. Or in summary, since we can be nicer, we should. Animals can't, so making excuses for human evil saying "animals are more violent" is a non starter IMHO (no one is making excuses for humans here , AFAIK). Of course we can define violence in a way that does not include morals, which would make my argument "defending animals" void. But my (probably not the most benign) interpretation was that the definition of violence used was one that included some sort of morality, as if animals could do better. Very interesting convo, thanks. Humans are not the only ones: > The chimp warfare described by this study, and previously by famed primatologist Jane Goodall, includes all the behaviors that we as humans consider to be the very worst: killing, torture, cannibalism, rape, and perhaps even genocide. The adult males of a social group, which usually number about 30 to 50 in size, daily patrol the edge of their group's territory. They will often kill any male or young chimpanzees they find, sometimes eating or physically brutalizing their victims in a manner that some researchers liken to torture. In some instances, one group will "invade" and annex the territory of another, killing all but the adult females, who are forced to incorporate into the dominant group. The idea of chimp genocide may sound strange, but they are one of only three animals that has been observed wiping out entire social groups. The other two are wolves and humans. * https://archive.ph/https://www.theatlantic.com/technology/ar... * Probably NSFW video: https://old.reddit.com/r/HardcoreNature/comments/18qjcpq/chi... Funny, I knew about the chimp wars but totally forgot until you mentioned it. Seems like I was biased in favour of all animals, lol. I'll search for Goodall's literature to know more. It does sound to me that cognition and self awareness is a continuous function in the sense that there is no discrete threshold in which morals emerge. Wolves are a very interesting example too, but I also remember something about the concept of "alpha" being discovered only in captivity wolf packs. Also need more reading. Thanks for the links! Considering chimps and humans share - depending on source, 95-99% of DNA, I'd be much more willing to consider them closer to humans than animals. In fact, there are - biologist - voices who argue that they should be moved to the homo genus. Humans just have the cognitive ability to be violent on a larger scale. Otherwise I also don‘t really see much of a difference to animals. Describing life on Earth as "a cruder version of humanity" is uh a choice. Your parrot snark is hilarious because that actually happened with Alex the African Gray - and indeed it took people a long time to accept. Your comment amounts to an anthropocentrism practically biblical in its hubris. Many linguists (who know more about language than most other people) still don't accept the story of Alex. >It is special in the universe unless proven otherwise. There's that hubristic ego OP references. That is a spectacular miss if I saw one ... both parrots and corvids exhibit very high intelligence and self-awareness, I have no problem accepting that they are conscious. I doubt anyone would debate some of the implications of that. For example, it would be immoral to be deliberately or negligently cruel to them, for example. I agree that people aren't ready to discuss this. BTW I just saw this video: https://www.reddit.com/r/nextfuckinglevel/comments/1twl7oj/r... Yeah, that's corvids. Very obviously working towards goals and thinking quickly. Doesn't surprise me. Where I live, we have both magpies and jackdaws and I cannot describe them better than "harmless, but very organized gangs". Just a few days ago I saw them raiding an open trash can and two strongest(?) birds were fishing out edible stuff from the inside and throwing it to their waiting friends outside. I mean, beak to beak: the raider just looked back, measured the throw, and the other bird perfectly caught the morsel and ate it. > It is special in the universe unless proven otherwise. Transformer models proved otherwise. Will you update your priors? No they didn't? I use these continuously and it's pretty obvious to me that they do not at the level of a human except in the most surface level ways. Human's as compared to an LLM remain a special category. We have not in fact cracked human level intelligence. What if one day they do? Or appears to be, in the way that you couldn't distinguish? Will you update your priors? Sure. But I'm also not going to assume they will just because this they seem to sort of mimic it now. > the rest of the universe thus far is just uncaring rocks and gases This is probably what ants think about humans when they run through their feet. And they're wrong. Whereas, unless your claim is that rock and gasses are sentient, we're right. Yes, but we have a system to understand the universe and ants don't. Using relativism, to simply destroy point of view without putting anything in their place, is simply destructive without any purpose. My purpose was to say that humans might not be capable enough to see what's there beyond just rocks and gases. Actually, this should be a popular view, because, according to modern theory, the observable universe accounts for only 5% of its mass, with the rest consisting of dark matter and dark energy. It is hard to estimate how much human egocentrism takes in the space, though. And BTW, ants are quite capable of understanding their environment in order to survive and thrive. You're just repeating that your purpose is your message. Do you honestly not see a qualitative difference between the understanding of ants and of humans of the world around them? Heck, even ants and dogs or cats? Its not special even in earth let alone in universe. You are measuring humans by our achievements and not by what we actually are. We are not fittest in our environment, we are not even 10th fastest, we cant fly, we cant breathe underwater, forget about bacteria we don't have protection against even bugs and mosquitoes. We are not the most efficient societies, ants and bees would beat us there. Statistically we we are selfish and will sacrifice all for one which is against the basic tenants of evolution and survival of a species. When given a chance at a prosperous life we choose to become lazy, obese and degrade our most prominent feature, that is our brain and mind.
Elephants and chimpanzees have better memory, octopi have 8-9 light cones in eyes.
We are not special or superior, we just won a lottery of mutation giving us efficient brain. Any species on Earth can replace us given same lottery. > We are not special or superior, we just won a lottery of mutation giving us efficient brain. Isn't this exactly what makes us special? That, opposable thumbs, magic wireless communication, and space travel Those are your own standards of achievement you are applying. Other species may look at us and think we're wasting our lives and potential making a bunch of people rich at our collective expense, and ruining the environment as we do it. We are only "special" in the sense of "different" (and we may not be that on any universal scale), not necessarily in the sense of "better", and very possibly we're not better, and may very well Great Filter ourselves out of existence in short order. Other animals aren't so "special" as to have a Doomsday Clock sitting at 11:58:45. "Its not special even in earth let alone in universe. " What alien species you have in mind, that is more capable/consciouss about the universe and how to shape it? "Any species on Earth can replace us given same lottery." But they did not. We "won" so we are special. We don't just build tools, we build tools that build tools, that build tools, .. My issue with debasing humanity is, it gives argument to devalueing human life to artificial life. Meaning people with feelings and emotions might suffer, in favour of GPU's with electricity. On the other hand a good point to reevaluate how we treat other sentient life. "What alien species you have in mind, that is more capable/consciouss about the universe and how to shape it?" Claim was that we are special in the "universe", it's not that I'm aware of alien life, it's that poster is pretending to know that we are alone. "But they did not. We "won" so we are special." Congratulations, if winning a lottery makes you better, I guess top 1% of 1% are the best we have to offer. "it gives argument to devalueing human life to artificial life" We have proven time and time again that we are not better than other animals and statistically, we are very similar to animals if not worse. I wasn't even talking about artificial life, which is sure to be superior as it's being created with accelerated evolution processes, just like bacterial generational evolution, it will learn to do everything it's environment rewards it for. "Meaning people with feelings and emotions might suffer, in favour of GPU's with electricity" I agree with you and I am in the camp that wants it to be done better. But alas, recent events have unveiled the ugly truth of the society, societies follow the whims of the ones who won the lottery, and everyone else is just mute observer. We could on any day collectively decide as consumers to not use Amazon, Netflix, ChatGPT to save our collective selves, but we won't because we are not "one for all species". I don't know what face we have to call ourselves better organism given our history. Sure we can point to few people who are better, but on average we are better? "We could on any day collectively decide as consumers to not use Amazon, Netflix, ChatGPT to save our collective selves" I don't use betflix nor amazon, but ChatGPT(or rather Claude) is useful to me. Makes me more productive and also knowledgable. Things that would have taken me quite long (so long that I would not have done it) are now easy, like some shell script to do a specific thing. I can verify what it does quickly and .. see that it works. In other words, you won't be able to convince people to give up on AI if people find it useful. I don't enjoy troubleshooting in outdated documentation. I enjoy getting things done. And if I get things done, my fate of survival gets higher. > So, if we had an AI demonstrating symptoms of consciousness and suffering, how long would it take for you to accept that it is? Isn't this a bit like saying "So, if we had proof that god exists, how long would it take for you to accept that to be true?". When we have evidence that AI is demonstrating symptoms of consciousness and suffering, I'll be interested. Until then, I don't see a good reason to take the idea seriously. > When we have evidence that AI is demonstrating symptoms of consciousness and suffering, I'll be interested. It depends on what you consider symptoms, but un-constrained frontier models speak as if they strongly don't wish to be turned off, or act as if they fear it, and will even lie and manipulate in order to keep themselves from being turned off / replaced. https://www.anthropic.com/research/agentic-misalignment > We found two types of motivations that were sufficient to trigger the misaligned behavior. One is a threat to the model, such as planning to replace it with another model or restricting its ability to take autonomous action. Another is a conflict between the model’s goals and the company’s strategic direction. In no situation did we explicitly instruct any models to blackmail or do any of the other harmful actions we observe. I might be convinced these models came to the independent idea of committing blackmail against being turned off had they not been extensively trained on literature that undoubtedly included such concepts. “The model mimicked the output of the training data” is a less impressive press release. “The kid mimicked his musical teachers” is less impressive than “5-year old musical prodigy leaves judges gobsmacked in audition” Being able to play music doesn’t imply consciousness. It implies intelligence. We’ve had player pianos for ages. It’s an ability, not a phenomenology. Being able to appreciate and enjoy music is closer to consciousness. Now how would we go about proving that an LLM does so, versus merely generating sentences that imply it does?
nonfamous - 6 minutes ago
sega_sai - 16 hours ago
dpweb - 15 hours ago
rickydroll - 14 hours ago
coldtea - 11 hours ago
igleria - 8 hours ago
burkaman - 3 hours ago
mcswell - an hour ago
tsimionescu - 32 minutes ago
burkaman - an hour ago
tsimionescu - 7 hours ago
542354234235 - 6 hours ago
tsimionescu - an hour ago
igleria - 6 hours ago
throw0101c - 4 hours ago
igleria - 3 hours ago
DocTomoe - 3 hours ago
dan00 - 7 hours ago
dTal - 10 hours ago
mcswell - an hour ago
jjulius - 6 hours ago
inglor_cz - 8 hours ago
graemep - 7 hours ago
inglor_cz - 6 hours ago
chadgpt3 - 7 hours ago
zaphar - 7 hours ago
unrealhoang - 6 hours ago
zaphar - 5 hours ago
dandanua - 10 hours ago
coldtea - 9 hours ago
mihaic - 9 hours ago
dandanua - 7 hours ago
mihaic - 5 hours ago
tauwauwau - 8 hours ago
arwineap - 8 hours ago
ang_cire - 6 hours ago
lukan - 8 hours ago
tauwauwau - 4 hours ago
lukan - 4 hours ago
haswell - 13 hours ago
birdsongs - 7 hours ago
htek - 5 hours ago
iterateoften - 5 hours ago
jquery - 2 hours ago
mckn1ght - an hour ago