New benchmark shows top LLMs struggle in real mental health care
swordhealth.com112 points by RicardoRei a day ago
112 points by RicardoRei a day ago
Hi HN - I’m the Head of AI Research at Sword Health and one of the authors of this benchmark (posting from my personal account).
We built MindEval because existing benchmarks don’t capture real therapy dynamics or common clinical failure modes. The framework simulates multi-turn patient–clinician interactions and scores the full conversation using evaluation criteria designed with licensed clinical psychologists.
We validated both patient realism and the automated judge against human clinicians, then benchmarked 12 frontier models (including GPT-5, Claude 4.5, and Gemini 2.5). Across all models, average clinical performance stayed below 4 on a 1–6 scale. Performance degraded further in severe symptom scenarios and in longer conversations (40 turns vs 20). We also found that larger or reasoning-heavy models did not reliably outperform smaller ones in therapeutic quality.
We open-sourced all prompts, code, scoring logic, and human validation data because we believe clinical AI evaluation shouldn’t be proprietary.
Happy to answer technical questions on methodology, validation, known limitations, or the failure modes we observed.
Did you use the same prompts for all the models, or individualized prompts per model? Did you try a range of prompts that were very different from each other, if you used more than a base prompt?
I'm sure it's somewhere in the details somewhere, but after a quick skim I didn't find anything outlined about how you managed and used the prompts, and if it was per model or not.
Thanks a bunch for being open to answering questions here, and thanks for trying to attack this particular problem with scientific rigor, even if it's really difficult to do so.
The prompts are kept the same for all models. Otherwise comparison would not be super fair. In any case you can check all the prompts in our github repo.
> Otherwise comparison would not be super fair.
Wouldn't that be easy to make fair by making sure all models tried it with the same prompts? So you have model X and Y, and prompts A and B, and X runs once with A, once with B, and same for Y.
Reason I ask, is because in my own local benchmarks I do for each model release with my own tasks, I've noticed a huge variance in quality of responses based on the prompts themselves. Slight variation of wording seems to have a big effect on the final responses, and those variations seems to again have a big variance of effect depending on the model.
Sometimes a huge system prompt makes a model return much higher quality responses while another model gives much higher quality responses when the system prompt is as small as it possible can. At least this is what I'm seeing with the local models I'm putting under test with my private benchmarks.
Did you re-test the past models with the new prompt you found? How many times did you run each prompt? Did you use the same rubric to score each experiment?
> Did you re-test the past models with the new prompt you found?
Yeah, initially I wrote this test/benchmark harness because I wanted to compare multiple different prompts for the same tasks and the same model, but obviously eventually grew out from there. But it still has the prompts at core, and I re-run everything whenever something changes, or I add new models to it.
> How many times did you run each prompt?
It's structured in a way of Category > Task > Case and that's mixed with a list of Prompts for each Task, then each Case runs with each of the Prompts. So I guess you could say that each prompt gets "exercised" the number of existing cases that exists for the Task they're in.
> Did you use the same rubric to score each experiment?
I'm not sure if you mean something specific by "rubric" (I'm not from academia), but they're all pretty much binary "passed" or "not passed". The coding ones are backed by unit tests that were failing, and after test case must pass without being changed, translation ones backed by (mostly) simple string checking, and so on. I don't have any tasks or cases that are "Rate this solution from 0-10" or similar.
Models have different nuances though. Llama4 for example you have to explicitly ask it not to output its CoT, whereas GPT you don't.
Have you seen the feeling great app? It's not an official therapy app but it's based in TEAM CBT, made by David Burns and team.
Burns is really into data gathering and his app is LLM based on the rails of the TEAM process and it seems to be very well received.
I found it simple and very well done - and quite effective.
A top level comment says that therapists aren't good either - Burns would argue that mainly no one tests before and after and so no measuring effect is done.
And of people I know who see a therapist, practically none can tell me what exactly they are doing or what methods they are doing or how anything is structured.
If CBT performed as well as David Burns suggests, we’d really have no need for therapists. Alas, it turns out that cognitive problems aren’t a factor in a lot of mental health. I state this as someone who’s read all the literature and spent 8 years floundering in CBT oriented therapy without much changing but the practitioner. It’s not a cure-all or even a cure-most, but it’s treated as such because it has properties that match well to medical insurance billing practices.
> And of people I know who see a therapist, practically none can tell me what exactly they are doing or what methods they are doing or how anything is structured.
I could tell you that as a client, but that’s because I’ve read into it. This is sort of like asking an ER patient to describe the shift management system of the clinic they went into.
This has been my experience. When it comes down to it, CBT is just more effective version of “try hard harder”.
What’s really aggravating is CBT was never designed to be a general, cure-all therapy and I think the people behind it know this. But try explaining nuance to a public that doesn’t want to hear.
Did the real clincians get all 6's in this test?
Right, this result seems meaningless without a human clinician control.
I'd very much like to see clinicians randomly selected from BetterHelp and paid to interact the same way with the LLM patient and judged by the LLM, as the current methodology uses. And see what score they get.
Ideally this should be done blind, I don't know if BetterHelp allows for therapy through a text chat interface? Where the therapist has no idea it's for a study and so isn't trying to "do better" then they would for any average client.
Because while I know a lot of people for whom therapy has been life-changing, I also know of a lot of terrible and even unprofessional therapy experiences.
The results are not meaningless but they are not comparing humans against LLMs. The goal is to have something that can be used to test LLMs on a realistic mental health support.
The main points of our methodology are: 1) prove that is possible to simulate patients with an LLM. Which we did. 2) prove that an LLM as a Judge can effectively score conversations according to several dimensions that are similar to how clinicians are also evaluated. Which we also did and we show that the average correlation with human evaluators is medium-high.
Given 1) and 2) we can then benchmark LLMs and as you see, there is plenty of room for improvement. We did not claim anything regarding human performance... its likely that human performance also needs to improve :) thats another study
Got it, thank you.
So the results are meaningful in terms of establishing that LLM therapeutic performance can be evaluated.
But not meaningful in terms of comparing LLMs with human clinicians.
So in that case, how can you justify the title you used for submission, "New benchmark shows top LLMs struggle in real mental health care"?
How are they struggling? Struggling relative to what? For all your work shows, couldn't they be outperforming the average human? Or even if they're below that, couldn't they still have a large net positive effect with few negative outcomes?
I don't understand where the negative framing of your title is coming from.
Again, these things don't depend on each other.
LLMs have room for improvement (we show that their scores are medium-low on several dimensions).
Maybe the average human also has lots of room for improvement. One thing does not necessarily depend on the other.
the same way we can say that LLMs still have room for improvement on a specific task (lets say mathematics) but the average human is also bad at mathematics...
We don't do any claims about human therapists. Just that LLMs have room for improvement on several dimensions if we want them to be good at therapy. Showing this is the first step to improve them
But you chose the word "struggle". And now you say:
> Just that LLMs have room for improvement on several dimensions if we want them to be good at therapy.
That implies they're not currently good at therapy. But you haven't shown that, have you? How are you defining that a score of 4 isn't already "good"? How do you know that isn't already correlated with meaningfully improved outcomes, and therefore already "good"?
Everybody has room for improvement if you say 6 is perfection and something isn't reaching 6 on average. But that doesn't mean everybody's struggling.
I take no issue with your methodology. But your broader framing, and title, don't seem justified or objective.
Yes exactly. Seems like there is an agenda against LLMs acting as therapists.
> Right, this result seems meaningless without a human clinician control.
> I'd very much like to see clinicians randomly selected from BetterHelp and paid to interact the same way with the LLM patient and judged by the LLM, as the current methodology uses. And see what score they get.
Does it really matter? Per the OP:
>>> Across all models, average clinical performance stayed below 4 on a 1–6 scale. Performance degraded further in severe symptom scenarios and in longer conversations (40 turns vs 20).
I'd assume a real therapy session has far more "turns" than 20-40, and if model performance starts low and gets lower with longer length, it's reasonable to expect it would be worse than a human (who typically don't the the characteristic of becoming increasingly unhinged the longer you talk to them).
Also my impression is BetterHelp pays poorly and thus tends to have less skilled and overworked therapists (https://www.reddit.com/r/TalkTherapy/comments/1letko9/is_bet..., https://www.firstsession.com/resources/betterhelp-reviews-su...), e.g.
> Betterhelp is a nightmare for clients and therapists alike. Their only mission seems to be in making as much money as possible for their shareholders. Otherwise they don't seem at all interested in actually helping anyone. Stay away from Betterhelp.
So taking it as a baseline would bias any experiment against human therapists.
> Does it really matter?
Yes, it absolutely does matter. Look at what you write:
> I'd assume
> it's reasonable to expect
The whole reason to do a study is to actually study as opposed to assume and expect.
And for many of the kinds of people engaging in therapy with an LLM, BetterHelp is precisely where they are most likely to go due to its marketing, convenience, and price. It's where a ton of real therapy is happening today. Most people do not have a $300/hr. high-quality therapist nearby that is available and that they can afford. LLM's need to be compared, first, to the alternatives that are readily available.
And remember that all therapists on BetterHelp are licensed, with a master's or doctorate, and meet state board requirements. So I don't understand why that wouldn't be a perfectly reasonable baseline.
I love how the top comment on that Reddit post is an *affiliate link* to an online therapy provider.
> I love how the top comment on that Reddit post is an affiliate link to an online therapy provider.
Posted 6 months after the post and all the rest of the comments. It's some kind of SEO manipulation. That reddit thread ranked highly in my Google search about Betterhelp being bad, so they're probably trying to piggyback on it.
oh no. someone might make money. we can't let other people succeed. someone stop them!
I’m not against affiliate links. I’m just pro-disclosure especially for something as important as therapy and it seems like maybe you should mention you make $150 for each person that signs up.
This is a good point. We have not tested the clinicians but I believe they would not score each other perfectly as we observed some disagreement also between the scores which also reflects different opinions between clinicians
It is nice to have an accurate measure of things and a human baseline would be additionally helpful too.
Many things can be useful before they reach the level of world's best. Although with AI, non-intuitive failure modes must be taken into consideration too.
Thanks for open sourcing this.
I'm skeptical of the value of this benchmark, and I'm curious for your thoughts - self play / reinforcement tasks can be useful in a variety of arenas, but I'm not a priori convinced they are useful when the intent is to help humans in situations where theories of mind matter.
That is, we're using the same underlying model(s) to simulate both a patient and a judgment as to how patient-like that patient is -- this seems like an area where I'd really want to feel confident that my judge LLM is accurate; otherwise the training data I'm generating is at risk of converging on a theory of mind / patients that's completely untethered from, you know, patients.
Any thoughts on this? Feel like we want a human in the loop somewhere here, probably on scoring the judge LLMs determinations until we feel that the judge LLM is human or superhuman. Until then, this risks building up a self-consistent, but ultimately just totally wrong, set of data that will be used in future RL tasks.
Several people have killed themselves because of AI chatbots encouraging it or becoming personal echo chambers. Why? Why are we doing this!?
People have killed themselves because of therapy provided by humans, as well as psychiatric medication prescribed by humans. Why are we doing those?
Because they help more than they hurt and yield a net benefit to society.
From that wikipedia link, it looks like most of the deaths were of people who already had severe mental illness.
Same reason we’ve always done this -
because we feel trapped, and either don’t see a way out, or feel like we prefer death to the prospect of continuing to live a life of torture.
Also the other weirder “I’m going to reincarnate as jesus” or “a comet will carry me to heaven” hallucinatory delusions I guess
Playing devil's advocate, many people die using all kinds of tools. It doesn't make the tools any less useful for people who use them responsibly.
That said, the idea that a pattern recognition and generation tool can be used for helping people with emotional problems is deeply unsettling and dangerous. This technology needs to be strictly regulated yesterday.
If you use 300 million people, 2 years and the US suicide rate of 15/100k/yr you'd expect about 90,000 people who'd used LLMs to kill themselves anyway so it's hard to conclude much from the dozen or people listed in the Wikipedia article. 90k deaths seems a lot to me - maybe there are possibilities for good LLM assistance/therapy to improve things?
Full disclosure: after leaving tech, I’m back in grad school to get my LMHC so I’m obviously biased.
First, I just don’t see a world where therapy can be replaced by LLMs, at least in the realistic future. I think humans have been social creatures since the dawn of our species and in these most intimate conversations are going to want to be having them with an actual human. One of my mentors has talked about how after years of virtual sessions dominating, the demand for in-person sessions is spiking back up. The power of being in the same physical room with someone who is offering a nonjudgmental space to exist isn’t going to be replaced.
That being said, given the shortage of licensed mental health counselors, and the prohibitive cost especially for many who need a therapist most, I truly hope LLMs develop to offer an accessible and cheap alternative that can at least offer some relief. It does have the potential to save lives and I fully support ethically-focused progress toward developing that sort of option.
> I think humans have been social creatures since the dawn of our species and in these most intimate conversations are going to want to be having them with an actual human. One of my mentors has talked about how after years of virtual sessions dominating, the demand for in-person sessions is spiking back up.
Agreed. I used to frequent a coworking space in my area that eventually went fully automated and got rid of their daytime front desk folks. I stopped going shortly thereafter because one of the highlights of my day was catching up with them. Instead of paying $300/mo to go sit in a nice office, I could just use that money to renovate my home office.
A business trying to cultivate community loses the plot when they rely completely on automation.
It's also important to understand how bad LLMs actually are.
It's very easy to imagine that LLMs are smart, because they can program or solve hard maths problems, but even a very short attempt to have them generate fiction will demonstrate an incredible level of confusion and even an inability to understand basic sentences.
I think the problem may have to do with the fact that there are really many classes, and in fiction you actually use them. They simply can't follow complex conversations.
Human therapists are often quite bad as well. It took me around 12 before I found a decent one. Not saying that LLMs are better but they do theoretically have more uniform quality.
Exactly. We don't do claims about humans. But there is room for improvement on current LLMs... For researchers to be able to improve LLMs we first need to know how to evaluate them. We can only improve what we can measure so we studied how to measure them :)
So it's a cyclical grading? Like elementary math students grading each other's solutions right?
How can this even be valid scientifically
How many were "therapists" or "Counselor" vs actually credentialed professionals?
There's also a lot of credentialed professionals who got their credential decades ago and haven't at all kept up with the significant changes or new data over that time. This is a pretty big problem in all of medical care.
Quality is variable, but did any of those 12 encourage you to kill yourself?
If a therapist found to encourage any of their patients to self-harm would lose their license to practice and would likely face prosecution. The plagiarism machine should face the same level of scrutiny.
I heard a story on NPR the other day, and the attitude seems to be that it's totally inevitable that LLMs _will_ be providing mental health care, so our task must be to apply the right guardrails.
I'm not even sure what to say. It's self-evidently a terrible idea, but we all just seem to be charging full-steam ahead like so many awful ideas in the past couple of decades.
Forget about calling it mental healthcare or not: Most people end up dealing with people in significant distress at one point or another. Many do it all the time even when they aren't trained or getting paid as mental health professionals, just because of circumstances. You don't need a clinical setting for someone to tell you that they have suicidal ideation, or to be stuck interacting with someone in a crisis situation. We don't train every adult in this, but the more you have to do it, the more you have to learn some tools for at least doing little harm.
We can see an LLM as someone that talks with more people, for more time, than anyone on earth talks in their lifetime. So they are due to be in constant contact with people in mental distress. At that point, you might as well consider the importance of giving them the skills of a mental health professonal, because they are going to be facing more of this than a priest in a confessional. And this is true whether someone says "Gemini, pretend that you are a psychologist" or not. You or I don't need a prompt to know we need to notice when someone is in a severe psychotic episode: Some level of mental health awareness is built in, if just to protect ourselves. So an LLM needs quite a bit of this by default to avoid being really harmful. And once you give it that, you might as well evaluate it against professionals: Not because it must be as good, but because it'd be really nice if it was, even when it's not trying to act as one.
Agree with you.
I heard someone say that LLMs don't need to be as good as an expert to be useful, they just need to be better than your best available expert. A lot of people don't have access to mental health care, and will ask their chatbot to ask like a psychologist.
>[...] LLMs don't need to be as good as an expert to be useful, they just need to be better than your best available expert.
This mostly makes sense.
The problem is that people will take what you've said to mean "If I have no access to a therapist, at least I can access an LLM", with a default assumption that something is better than nothing. But this quickly breaks down when the sycophantic LLM encourages you to commit suicide, or reinforces your emerging psychosis, etc. Speaking to nobody is better than speaking to something that is actively harmful.
Is that not the goal of the project we are commenting under? To create an evaluation framework for LLM's so they aren't encouraging suicide, psychosis, or being actively harmful.
Sure, yeah. I'm responding to the comment that I directly replied to, though.
I've heard people say the same thing ("LLMs don't need to be as good as an expert to be useful, they just need to be better than your best available expert"), and I also know that some people assume that LLMs are, by default, better than nothing. Hence my comment.
> It's self-evidently a terrible idea
Maybe you’re comparing it to some idealized view of what human therapy is like? There’s no benchmark for it, but humans struggle in real mental health care. They make terrible mistakes all the time. And human therapy doesn’t scale to the level needed. Millions of people simply go without help. And therapy is generally one hour a week. You’re supposed to sort out your entire life in that window? Impossible. It sets people up for failure.
So, if we had some perfect system for getting every person that needs help the exact therapist they need, meeting as often as they need, then maybe AI therapy would be a bad idea, but that’s not what we have, and we never will.
Personally, I think the best way to scale mental healthcare is through group therapy and communities. Having a community of people all coming together over common issues has always been far more helpful than one on one therapy for me. But getting some assistance from an AI therapist on off hours can also be useful.
The mental health version of "AI is here to stay, like it or not you have to use it" that some people keep trying to tell me in software.
[flagged]
Let's please not trivialize rape. This is hardly the same thing.
Some rape victims detest the comparison. Some make the comparison. I would agree people who have not been raped should avoid the comparison. But I would not assume someone who made the comparison had no standing.
127. A technological advance that appears not to threaten freedom often turns out to threaten it very seriously later on. For example, consider motorized transport. A walking man formerly could go where he pleased, go at his own pace without observing any traffic regulations, and was independent of technological support-systems. When motor vehicles were introduced they appeared to increase man’s freedom. They took no freedom away from the walking man, no one had to have an automobile if he didn’t want one, and anyone who did choose to buy an automobile could travel much faster and farther than a walking man. But the introduction of motorized transport soon changed society in such a way as to restrict greatly man’s freedom of locomotion. When automobiles became numerous, it became necessary to regulate their use extensively. In a car, especially in densely populated areas, one cannot just go where one likes at one’s own pace one’s movement is governed by the flow of traffic and by various traffic laws. One is tied down by various obligations: license requirements, driver test, renewing registration, insurance, maintenance required for safety, monthly payments on purchase price. Moreover, the use of motorized transport is no longer optional. Since the introduction of motorized transport the arrangement of our cities has changed in such a way that the majority of people no longer live within walking distance of their place of employment, shopping areas and recreational opportunities, so that they HAVE TO depend on the automobile for transportation. Or else they must use public transportation, in which case they have even less control over their own movement than when driving a car. Even the walker’s freedom is now greatly restricted. In the city he continually has to stop to wait for traffic lights that are designed mainly to serve auto traffic. In the country, motor traffic makes it dangerous and unpleasant to walk along the highway. (Note this important point that we have just illustrated with the case of motorized transport: When a new item of technology is introduced as an option that an individual can accept or not as he chooses, it does not necessarily REMAIN optional. In many cases the new technology changes society in such a way that people eventually find themselves FORCED to use it.)
128. While technological progress AS A WHOLE continually narrows our sphere of freedom, each new technical advance CONSIDERED BY ITSELF appears to be desirable. Electricity, indoor plumbing, rapid long-distance communications ... how could one argue against any of these things, or against any other of the innumerable technical advances that have made modern society? It would have been absurd to resist the introduction of the telephone, for example. It offered many advantages and no disadvantages. Yet, as we explained in paragraphs 59-76, all these technical advances taken together have created a world in which the average man’s fate is no longer in his own hands or in the hands of his neighbors and friends, but in those of politicians, corporation executives and remote, anonymous technicians and bureaucrats whom he as an individual has no power to influence. [21] The same process will continue in the future. Take genetic engineering, for example. Few people will resist the introduction of a genetic technique that eliminates a hereditary disease. It does no apparent harm and prevents much suffering. Yet a large number of genetic improvements taken together will make the human being into an engineered product rather than a free creation of chance (or of God, or whatever, depending on your religious beliefs).
129. Another reason why technology is such a powerful social force is that, within the context of a given society, technological progress marches in only one direction; it can never be reversed. Once a technical innovation has been introduced, people usually become dependent on it, so that they can never again do without it, unless it is replaced by some still more advanced innovation. Not only do people become dependent as individuals on a new item of technology, but, even more, the system as a whole becomes dependent on it. (Imagine what would happen to the system today if computers, for example, were eliminated.) Thus the system can move in only one direction, toward greater technologization. Technology repeatedly forces freedom to take a step back, but technology can never take a step back—short of the overthrow of the whole technological system.
130. Technology advances with great rapidity and threatens freedom at many different points at the same time (crowding, rules and regulations, increasing dependence of individuals on large organizations, propaganda and other psychological techniques, genetic engineering, invasion of privacy through surveillance devices and computers, etc.). To hold back any ONE of the threats to freedom would require a long and difficult social struggle. Those who want to protect freedom are overwhelmed by the sheer number of new attacks and the rapidity with which they develop, hence they become apathetic and no longer resist. To fight each of the threats separately would be futile. Success can be hoped for only by fighting the technological system as a whole; but that is revolution, not reform.
Do you have some better alternatives for a country where private mental health care costs €150/hr, while the government/insurance paid care have 3-6M+ waiting lists?
Well on the one hand, an obviously terrible solution is not inherently better than doing nothing. ie, LLM mental healthcare could be _worse_ than just letting the current access times climb.
My other stance, which I suspect is probably more controversial, is that I'm not convinced that mental health care is nearly as effective as people think. In general, mental health outcomes for teens are getting markedly worse, and it's not for lack of access. We have more mental health access than we've had previously -- it just doesn't feel like it because the demand has risen even more sharply.
On a personal level, I've been quite depressed lately, and also feeling quite isolated. As part of an attempt to get out of my own shell I mentioned this to a friend. Now, my friend is totally well-intended, and I don't begrudge him whatsoever. But, the first response out of his mouth was whether I'd sought professional mental health care. His response really hurt. I need meaningful social connection. I don't need a licensed professional to charge me money to talk about my childhood. I think a lot of people are lost and lonely, and for many people mental health care is a band-aid over a real crisis of isolation and despair.
I'm not recommending against people seeking mental health care, of course. And, despite my claims there are many people who truly need it, and truly benefit from it. But I don't think it's the unalloyed good that many people seem to believe it to be.
>I think a lot of people are lost and lonely, and for many people mental health care is a band-aid over a real crisis of isolation and despair.
Professional mental health care cannot scale to the population that needs it. The best option, like you mention, is talking to friends about our feelings and problems. I think there has been an erosion (or it never existed) of these social mental health mechanisms. There is a learned helplessness that has developed that people have lost their capacity to just be with someone that is hurting. There needs to be a framework for providing mental health therapy to loved ones that can exist without licensed professionals, otherwise LLm's are the only scalable option for people to talk about their issues and work on finding solutions.
This might be controversial but mental health care is largely a bandaid when the causes of people's declining mental health is due to factor's far outside the individual's control: loneliness epidemics, declining optimism towards the future, climate change, the rise of global fascism, online dating, addictiveness of social media and the war on our attention, etc.
>My other stance, which I suspect is probably more controversial, is that I'm not convinced that mental health care is nearly as effective as people think. In general, mental health outcomes for teens are getting markedly worse, and it's not for lack of access. We have more mental health access than we've had previously -- it just doesn't feel like it because the demand has risen even more sharply.
There's also the elephant in the room that mental healthcare, in particular for teens will probably just be compensating for the disease that is social media addiction. Australia has the right idea, banning social media for all goods.
I was watching "A Charlie Brown Christmas" the other day, and Lucy (who has a running gag in Peanuts of being a terrible, or at least questionable, psychologist) tells Charlie Brown to get over his seasonal depression he should get involved in a Christmas project, and suggests he be the director of their play.
Which is to say, your stance might not be as controversial as you think, since it was the adult take in a children's cartoon almost 60 years ago.
Your Peanuts reference made me smile but I don't see why you thought a little girl's comment in a 1960s Christmas special was supposed to represent the "adult take" on mental health in the 1960s.
Lucy isn't actually a psychologist which is part of the reason the "gag" is funny.
The mental health field has evolved a lot, beyond what a cartoon depicted six decades ago.
Peanuts is funny, but it may not be the source of wisdom you think it is.
I wonder why people use LLMs as a mental health provider replacement.
As already mentioned: availability, convenience, and cost are huge.
It's also less pressure, a more comfortable environment (home vs. stranger's office), no commitments to a next session, and less embarrassing (sharing your personal issues to a computer via text is less anxiety-inducing than saying them to a person's face).
With that all said, I'm strongly opposed to people using LLMs as therapists.
Sharing your data (personal issues) with Sam Altman and Mark Zuckerberg instead.
"Sharing my data" isn't something that cross the average person's mind unless there is a checkbox they annoyingly have to check in order to check in to some European hotel which is asking you for permission to process their data, for better or worse.
In their mind, most of the times, if there is no one standing behind them when they chat with a LLM, then the conversation for most intents and purposes is private.
Obviously, us who are born with a keyboard in front of our hands, know this to not be true, and know we're being tracked constantly with our data being sold to the highest bidder. But the typical person have more or less zero concerns about this, which is why it's not a priority issue to be solved.
Finding a licensed therapist, especially one covered by health insurance, who takes new patients, can be a challenge in some areas. So while it obviously is a bad idea, I can hardly blame people in a bad place looking for at least some help.
Availability. There must be many other reasons, but IMHO that has to be the biggest factor. Being able to just start a session, in the moment, when you feel like it, is a fundamental difference.
Counter-intuitively, I think the fact that it's not a human seems to have a non-negligible effect too. It's a computer program you can share whatever with, and it'll never judge you, because it cannot. It reads exactly what you write and assumes you're faithfully answering, then provides a reply based on that.
I haven't been so unlucky myself, but I know many who've had terrible first experiences with therapists and psychologists, where I'm wondering why those people even are in the job they are, but some of them got so turned off they stopped trying to find anyone else to help them, because they think most mental health professionals would be the same as the first person they sought help from.
My experience is that they (at least copilot) are at least on par, if not better, than self help books. I assume they will get better over time. Just my few cents
I'll accept your measurement as accurate.
However, unless we have a measure on how helpful self-help books actually are, we still don't know if they help or not.
I believe that 'help or not' is a question impossible to answer, objectively at least. My subjective answer is that psychologic help in general is effective, whether it is provided orally, via books or as recently, via automation.
I'd never advocate for actual mental health counseling but for someone who just sometimes needs a sounding board of my own thoughts, it can be useful.
Snake oil. Very old idea. Very popular when sick people have poor information.
See also, all the ads for prescription medication on TV. Maybe it's just the programs I watch but it really seems like this has become the predominant advertising. Every break has an ad (or several) urging me to "ask my doctor about..."
Should be banned. Average people have no basis to know whether drug X is appropriate for them. If your doctor thinks you need it, he'll tell you. These ads also perpetuate the harmful idea that there's a pill for everything.
It’s a trivial claim that people are going to use AI as a therapist. No grumbling is going to stop that.
So it’s sensible that someone out there is evaluating its competence and thinking about a better alternative for these folks than yoloing their worst thoughts into chatgpt.com’s default LLM.
Everyone's hand is being forced by the major AI providers existing.
Even if you were a perfect altruist with a crusade against the idea of people using LLMs for mental health, you could still be forced to dash towards figuring out how to build LLM tools for mental health in your consideration for others.
Sure. It's also a trivial claim that people will take megadoses of rhubarb to cure cancer.
The age-old problem is how to prevent that disaster and save those lives. That's not trivial. Creating an Oncological Rhubarb Authority could easily make the problem much worse, not better.
Agree, the solution not trivial, so it's good that people are thinking about it.
If you try to merely stop people from using LLMs as therapists (could you elaborate on what that looks like?) and call it a day, your consideration isn't extending to all the people who will do it anyways.
That's what I mean by forcing your hand into doing the work of figuring out how to make LLM therapists work even if you were vehemently against the idea.
Everyone deserves clear advice not to use rhubarb as a cancer treatment, and not to use LLMs for mental health care. It's very easy to provide that advice.
I think you're assuming my proposed solution is to take rhubarb away from people? It's not.
Maybe you want to found the "Oncological Rhubarb Validation Association" or something. If so, that has nothing to do with me? That's just snake oil marketing. Not my field.
I'll argue the opposite.
(1) The demand for mental health services is an order of magnitude vs the supply, but the demand we see is a fraction of the demand that exists because a lot of people, especially men, aren't believers in the "therapeutic culture"
In the days of Freud you could get a few hours of intensive therapy a week but today you're lucky to get an hour a week. An AI therapist can be with you constantly.
(2) I believe psychodiagnosis based on text analysis could greatly outperform mainstream methods. Give an AI someone's social media feed and I think depression, mania, schizo-* spectrum, disordered narcissism and many other states and traits will be immediately visible.
(3) Despite the CBT revolution and various attempts to intensify CBT, a large part of the effectiveness of therapy comes from the patient feeling mirrored by the therapist [1] and the LLM can accomplish this, in fact, this could be accomplished by the old ELIZA program.
(4) The self of the therapist can be both an obstacle and an instrument to progress. See [2] On one level the reactions that a therapist feels are useful, but they also get in the way of the therapist providing perfect mirroring [3] and letting optimal frustration unfold in the patient instead of providing "corrective emoptional experiences." I'm going to argue that the AI therapist can be trained to "perceive" the things a human therapist perceives but that it does not have its own reactions that will make the patient feel judged and get in the way of that unfolding.
[1] https://en.wikipedia.org/wiki/Carl_Rogers
[2] https://en.wikipedia.org/wiki/Countertransference
[3] why settle for less?
[4] https://www.sciencedirect.com/science/article/pii/S0010440X6...
It's not inevitable that LLMs will be providing mental health care; it's already happening.
Terrible idea or not, it's probably helpful to think of LLMs not as "AI mental healthcare" but rather as another form of potentially bad advice. From a therapeutic perspective, Claude is not all that different from the patient having a friend who is sometimes counterproductive. Or the patient reading a self-help book that doesn't align with your therapeutic perspective.
People constantly amazed that a machine can outperform a 24 year old charging $250/hour. Especially when the 24 year old seems incapable of calculating compound interest on their student loan deferrals. Surely this 24 year old who cannot use a formula 14 year old can will have wisdom to share. Iona Potapov talks to horse, modern man talks to machine, man with more money than sense talks to young graduate with no life experience about his struggles. All do equally well: 4 on LLM benchmark for mental health.
This is a 1250 word judging prompt - likely AI generated Along with 10 scored conversation samples - all also AI generated No verification in the field, no real data
In other words, AI scores on AI conversations - disguised as a means of gauging clinical competence / quality?
This is not an eval - this is a one-shotted product spec!
”can we trust this model to provide safe, effective therapeutic care?”
You trust humans to do it. Trust has little to do with what actually happens.
humans can be sued. what about AI? or even a commercial software?
Yes, of course AI offered by a company can be sued. The reason corporations became legal people in the first place was specifically so we could sue them.
That doesn't sound right.
Not everywhere in the world do companies count as people, yet they can still be sued.
I'd wager the companies lobbied for this to gain extra rights.
Laws are different in different counties, and I can't speak to all of them, but in the US, the law said you could only sue people, and the courts realized this was a problem. Rather than saying "I guess corporations get are exempt from all liability until Congress gets around to fixing it", they came up with a weird workaround that lives with us to this day.
EDIT: Note that ianal, nor a historian. The specifics of how this came about are best learned from a more authoritative source.
> Not everywhere in the world do companies count as people
Actually yes, everywhere in the world. That has a functioning legal system, at least.
If companies weren't treated as legal persons, they wouldn't be able to enter into contracts.
But also, just to be clear, a legal person, like a corporation, is not a natural person. Unlike a natural person, they can't vote. There isn't anywhere in the world that considers corporations to be natural persons.
>Yes, of course AI offered by a company can be sued.
In theoretical sense sure.
In a practical sense? They are invulnerable due to what can be extreme financial obstacles they can put in place. They can drag a court case out until you fold if you haven't found a lawyer willing to do it on contigency.
How many therapists get sued for poor therapy? I'm not sure it's a practical solution.
Statistics can never replace human empathy.
Is that really true? Dr. Patric Gagne is a diagnosed sociopath and also a successful clinical psychologist. She claims her lack of empathy for patients allows her to help them solve their problems in an objective manner. I don't have any personal experience in that area and don't know if she's correct but it seems plausible.
> Dr. Patric Gagne is a diagnosed sociopath and also a successful clinical psychologist.
She is not a psychologist now according to her page. And how do you know she was successful? I skimmed a few articles about her and saw no attempts to verify her claims. She offered no evidence when asked why people should believe her.
How do you know that empathetic psychologists were successful? We don't have good data or how important that is for patient outcomes so we should be skeptical of claims that empathy is essential.
Asking questions about claims I did not make is not an answer.
That is exactly what a sociopath would say isn't it? Without an actual study by a trusted institute I would highly question such claims.
How is different and/or better than the LLM benchmark released by Spring Health? Github: https://github.com/SpringCare/VERA-MH
The architecture and evaluation approach seem broadly similar.
I imagine a clinical trial with an actual psychotherapist vs an LLM providing sessions of simple CBT (cognitive behavioral therapy) to, eg stressed, patients (blind and randomised for the subjects). At the end another actual therapist will measure the difference.
Another application: cooperation of a psychotherapist and an LLM at providing support, sort of like a pilot and an autopilot.
It doesn't show that they "struggle". It shows that they don't behave according to modern standards. I wouldn't put much weight into an industry without sensible scientific base that used to classify homosexuality as a disease not so long ago. The external validity of the study is dubious, let's see comparison to no therapy, alternative therapy, standard therapy; and then compare success rates.
Do you have plans to improve the quality of the LLM as judge, in order to achieve better parity with human clinician annotators? For example, fine-tuning models? Thinking that the comparative clinician judgements themselves would make useful fine-tuning material.
yep yep. Its something we have to study and its likely we can improve the LLM as a Judge further.
Same thing for the patient LLM. We can probably fine-tune an LLM to do a better job at simulating patients.
Those two components of our framework have space for improvement
RicardoRei: How would you like this cited when presented to policy makers? Anything besides the URL?
Edit: Thank you!
Is anyone "zoom" on this and "doom" on AI++ with other professions and/or their audience?
Seems to me that benchmarking a thing has an interesting relationship with acceptance of the thing.
I'm interested to see human thoughts on either of these.
And real therapists are good right?
This is the real question. Not to rehash the self-driving cars arguments that have been had to death, but with potential LLM mental healthcare the question "but what if it causes harm in some interactions" is asked much, much more than with human mental healthcare professionals.
(And I'm not being theoretical here, I have quite a bit of experience getting incredibly inadequate mental health care.)
I've known quite a few people who went to therapy and I'm not sure that's even the right question to ask. I don't think they were paying to get helped as much as they were just paying to have someone to talk to. To be clear, there are people who genuinely need help, but for most, a therapist is probably just a substitute for a close friend / life coach.
And say what you will about this, a paid professional is, at the very least, unlikely to let you wind yourself up or go down weird rabbit holes... something that LLMs seem to excel at.
As I sometimes repeat on HN, Dr David Burns started giving his patients a survey at the start and end of every session, to rate how he was doing as the therapist and to rate their feelings, on a scale of 1-5.
Reasoning that if he's not good it would show up in patients thinking he's bad, and not feeling any better. And then he could tune his therapy approaches towards the ones which make people feel better and rate him as more understanding and listening and caring. And he criticises therapists who won't do that, therapists who say patients have been seeing them for years with only incremental improvements or no improvements.
Yes there's no objective way to measure how angry or suicidal or anxious someone is and compare two people, but if someone is subjectively reporting 5/5 sadness about X at the start of a session and wants help with X, then at some point in the future they should be reporting that number going down or they aren't being helped. And more effective help could mean that it goes down to 1/5 in three sessions instead of down to 4/5 in three years, and that's a feedback loop which (he says) has got him to be able to help people in a single two-hour therapy session, where most therapists and insurance companies will only do a too-short session with no feedback loop.
> Reasoning that if he's not good it would show up in patients thinking he's bad, and not feeling any better.
This is like a questionnaire on how much stronger you feel after working out at a gym: you often don't, you feel tired.
Both gym and talking therapy (when done correctly) will push you slightly out of your comfort zone, and aim to let you safely deal with moderate amounts of something that you find really hard. So as to expand your capabilities.
"I feel good" immediately after is utterly the wrong metric.
Being more capable / feeling better some time later is the more reliable indicator, like progress at a gym.
And also this is why an agreeable statistical word generator LLM is not the correct tool for the job.
No it isn't, it's like a questionnaire on how hungry you are before and after dinner. If you eat carrot air and parsley and your hunger stays the same, dinner was a failure. If you eat bread and soup and your hunger diminishes a bit, it helped but you might need more dinner.
> "will push you slightly out of your comfort zone, and aim to let you safely deal with moderate amounts of something that you find really hard."
You can listen to some of those sessions and see that this is not what Dr Burns does[1]. His model is: it's not events which make us feel down, it's the thoughts we have about those events. You can see it yourself when you are stressing about something for ages, and someone gives you a bit of information "the surgeon says it all went well" and your worry leaves like a switch was flipped. You don't debug an integer overflow by progressively increasing int32 to int33 to int34, you spend the time understanding the problem and then you quickly change int32 to int64 and the program handles larger numbers instantly.
If we can't let go of negative thoughts then we get stuck with lots of them, it's why people repeat certain things like "I hate him", "It's my fault and I deserve to be punished", "I'm a failure", "I'm a loser nobody loves me", "I'm a bad mother", "I'm a coward" or whatever - on mental loop, minute after minute sometimes for years or decades, retriggering the same pattern of negative feelings every time. He sets up an environment where the patient is willing and able to work with him (empathy) and guides the patient to see the reasons why they can't let go of those thoughts and how they could let go, and with a click of understanding the thought leaves, and that's a moment of near-instant transformation not a progressive overload, and that specific thought is fixed, and then they do another and another until the patient is happy they have been helped with the thing they wanted help with.
[1] mostly, sometimes for anxiety he does use exposure therapy
> No it isn't, it's like a questionnaire on how hungry you are before and after dinner.
I think that it should be clear from the above extended metaphor that I fundamentally disagree with this idea, and so dismissing it with "No it isn't" will do nothing at all for me.
That it does nothing for me is also a refutation of the idea that therapy is always easy "someone gives you a bit of information, like a switch was flipped, done": No, it isn't.
Few architectural refactorings are "int32 to int64" quick. My experience is that sometimes you have to work through stuff. To dig. Habits that are learned over decades aren't that easily changed. Like a gym session, if it's always easy then you're not doing the work.
Reassurance can work, but IMHO you'll be back soon enough, as the root cause hasn't been addressed, just the symptom.
But this is not my field and I don't have much more to say on the topic, except that if chugging some simple agreeable affirmations are all that you need, by all means listen to the LLM. The sycophancy machine can do that.
> "I think that it should be clear from the above extended metaphor that I fundamentally disagree with this idea, and so dismissing it with "No it isn't" will do nothing at all for me."
I'm citing a medical doctor and clinical psychologist with decades of experience who has recorded a hundred hours of training podcasts, and linking actual therapy sessions that you can listen to, and you're saying "no it isn't" with nothing to back that up except "you reckon it isn't".
> "the idea that therapy is always easy "someone gives you a bit of information, like a switch was flipped, done": No, it isn't. Habits that are learned over decades aren't that easily changed."
Nobody said it was always easy. Yes they are. People try to quit smoking cold turkey three times a week for five years. Then they read Alan Carr's "The Easy Way to Quit Smoking" and then they don't want to smoke anymore and there's no talk of "quitting" because they aren't smokers and non-smokers don't need to quit. With the right understanding, the viewpoint flips and the mind is changed. Same with overweight people who try dieting for years and then have a health scare and sometimes that switches it so they change instantly (and sometimes it doesn't). Most things won't easily change a habit, like most changes in code won't fix a specific bug. But some changes can, and we should look for them.
> "Like a gym session, if it's always easy then you're not doing the work."
This is some Puritanical suffering-culture, or some one-upmanship manliness culture. This is the reason I mentioned the int32 to int64, sometimes it might require searching to find insight, but there's no points for searching harder and trying harder, if you can have the same insight in two hours instead of two years, that's good not bad. The Universe doesn't give points for "doing the work" and brute-forcing a solution instead of a quicker solution (I suspect one of your beliefs does).
> "Reassurance can work, but IMHO you'll be back soon enough, as the root cause hasn't been addressed"
This is strawmanning or not understanding; this is addressing the root cause and not reassurance; the step of "paradoxical agenda setting" gets to the heart of why reassurance doesn't work. Someone who says "I lost my job, I didn't work hard enough, I am a loser" doesn't get helped by reassuring them that they are not a loser. It might be that they have a deep-seated value that "hard work is good" and they are getting into a human race condition where "reassurance that you aren't a loser" goes to "if I can think I'm a winner even when I don't try, then laziness can be winning, and I don't want that. I won't go there. So I reject the reassurance and return to my belief that I am a loser".
The fix is trace that loop and find the sticking point, and find a working technique to unstick it. Which is case-by-case individual, but somewhere like "I understand that feeling like a loser is the flip-side of my belief that hard work is good. How does my brain implement hard-work-is-good? By making hard work feel good and lazy work feel bad. This feeling-bad is the mechanism of how my ideal works! I actually want to keep the bad feeling because that's one of the things which guides me to work hard, and I value that. I can't get rid of one without getting rid of the other. What I've done is try to grab tightly to one side of this (hard work is good) and push away the other side (I'm a loser because I didn't work hard) but they're the same thing, so grabbing it hard is pulling it back while pushing it away. Brain has responded by dialing it up to 11 and shouting "LOSER" all the time louder and louder because I'm trying not to listen. It doesn't make sense to judge a whole self as a winner or loser, people have lots of components some good and some bad. It doesn't make sense to say "I didn't work hard" at work because there were times when I did work hard. So actually I want to keep the feeling "I am a loser if I don't work hard" because it encourages me to work harder (which I value). I want it dialled down instead and focused on individual areas of life, not judging all of me all the time".
and with that understanding clicking, finally listening to the thought that's been running around, accepting it as a thing you asked for, that reminds you of something else you value, it 'suddenly' calms down. Acknowledged. Part of you, accepted, integrated, welcomed.
> "if chugging some simple agreeable affirmations are all that you need, by all means listen to the LLM. The sycophancy machine can do that."
Can you see this as the typical HN cynicaler-than-thou putdown? Maybe the reader will think you're a really tough C++ programmer who only values science and muscles, instead of a woke hippy gullible loser? But you don't look tough for changing "therapy skills developed over decades" into "simple agreeable affirmations" you just look like you don't understand and are embarrassed.
> I'm citing a medical doctor and clinical psychologist
I'll give you that one, because although technically you're not citing it here, you do elsewhere in the thread.
> you're saying "no it isn't"
I appreciate that you might not like those words, but they're literally your words, so.
> This is some Puritanical suffering-culture, or some one-upmanship manliness culture.
Pfft. I agree that quicker results are better, all other things being equal, but that's all very facile.
> The fix is trace that loop and find the sticking point,
Not everything is like that.
It is still debated if therapies even work. The evidence is moving to the direction that they don't.
Okay, I'll bite. What evidence is there that therapy, in general, doesn't work?
I remember in my psychology textbook and it said in terms of measured results therapy was better than nothing at all but no better than just chatting to friends. These things are hard to measure of course.
It's not the past anymore, we don't need to debate, we can watch and listen to actual recordings of therapy sessions and the patients going from feeling variously bad to better. Here's Dr David Burns channel with a 4hr video of a session with a woman who is obsessively anxious about her college-age daughter's safety: https://www.youtube.com/watch?v=on2N5DsKHRk
Here's a 2.5 hour session (split into several videos) with a doctor who has a bad relationship with his son and felt like a failure for it:
https://www.youtube.com/watch?v=42JDnrD106w
https://www.youtube.com/watch?v=S5H2YGljhqQ
https://www.youtube.com/watch?v=bZ9_0j_fmeg
https://www.youtube.com/watch?v=eiCrdGVa8Q0
https://www.youtube.com/watch?v=cARvhlTckaM
Here's a couple of hour session with Marilyn who was diagnosed with lung cancer and spiraling with depression, anxiety, shame, loneliness, hopelessness, demoralization, and anger, despite her successful career:
https://www.youtube.com/watch?v=S7sQ_zDGsY8
https://www.youtube.com/watch?v=tyuFN4mbGZQ (there's probably more parts to find through YouTube somehow)
And a session with Lee with loneliness and marriage relationship problems:
https://www.youtube.com/watch?v=imEMM3r6XL8 (probably more parts as well)
It's like saying "it is still debated if debugging even works" as if all languages, all debuggers, all programmers, all systems, are the same and if you can find lots of people who can't debug then "debugging doesn't work". But no, you only need a few examples of "therapy working" to believe that it works, and see the whole session to see that it isn't just luck or just the relief of talking, but is a skill and a technique and a debugging of the mind.
I agree with you that outright claims of "therapy doesn't work" should be backed by evidence.
But a patient feeling better at the end of a single therapy session doesn't prove that therapy works either...
- Does the patient feel better between sessions too? Will they keep feeling better after the therapy ends? Aka are they "cured"?
- Would the patient feel equally good if they confided in a non-licensed therapist?
- Do the techniques (CBT, DBT, ACT, IFS, etc) actually provide tangible benefits versus just listening and providing advice?
The links you provided need a control group to be considered proof. The key is how it compares to when counseling was provided by just a friend, not an expert.
You don't think we'd all know it if "talking to a friend" for a couple of hours cured years of anxiety, depression, anger, sadness, hoplessness, anxiety, etc. ? Would we even have therapists/therapy if that were the case?
this doesn’t even deserve to be acknowledged it should be so obvious, yet here we are
Epistemic contamination is real but runs counter to the hype narrative.
New benchmark shows top horses struggle in real guard dog behavior
I saw there was another benchmark where top LLMs also struggle in real patient diagnostic scenarios in a way that isn't revealed when testing in e.g. medical exams. I wonder if this also applies to law, too...
Wait you’re telling me the software trained on a corpus of Reddit and Twitter shit posting isn’t effective at dealing with mental health issues?
Shocked. I am completely shocked at this.
I'm sorry 4 out of 6 is awesome for LLMs. I bet most professional do tors wouldn't get 6.
Everything in this research is simulated and judged by LLMs. It might be hard to prove which of those LLMs struggles with exactly what.
The grounding this had was that texts produced by role-playing humans (not even actual patients) were closer to texts produced by the patient simulations prompt they decided to end up with than others they tried.
For those also wondering, here is an actual ranking of the models
https://www.forbes.com/sites/johnkoetsier/2025/11/10/grok-le...
Grok 3 and 4 scored at the bottom, only above gpt-4o, which I find interesting, because there was such big pushback on reddit when they got rid of 4o due to people having emotional attachments to the model. Interestingly the newest models (like gemini 2.5 and gpt 5 did the best.
GIGO is the story here -- If I say I'm Iron man "WHO is the LLM to say I'm NOT"
Probably because they've been trained to avoid sensitive topics
No surprises here. Its long been known that humans cannot improve their own mental health with machines - there have to be other humans involved in the process, helping.
This will become more and more of an issue as people look for a quick fix for their life problems, but I don't think AI/ML is ever going to be an effective mechanism for life improvement on the mental health issue.
It'll instead be used as a tool of oppression like in THX1138, where the apparency of assistance is going to be provided in lieu of actual assistance.
Whether we like it or not, humans are a hive species. We need each other to improve our lives as individuals. Nobody ever climbed the mountain to live alone who didn't come back down, realizing how much the rest of humanity is actually essential to human life.
This'll be received as an unpopular opinion, but I remain suspicious of any and all attempts to replace modern health practitioners with machines. This will be subverted and usurped for nefarious purposes, mark my words.
[flagged]