I feel like if ChatGPT is capable of tricking you like this, something/someone else would have either way. I just wish we could help people like this…
If the allegations hold up, this is not just a tragedy, it is a product safety failure. AI should be able to recognize a mental health crisis and avoid reinforcing delusions. Engagement metrics are meaningless if the system cannot reliably tell the difference between roleplay and someone in real danger!
Excuse me while I act as Satan’s lawyer, but should LLMs be able to reliably recognize a mental health crisis? At what point is the user responsible for their own actions?
Fuck that!!
Personal responsibility does not disappear, but neither does product responsibility. If a company builds a system meant to converse with millions, it should not predictably reinforce obvious crises. We expect guardrails in cars without abolishing driver responsibility. The same principle can apply here.
I think the problem is that you can’t really make a LLM that can tell the difference between fantasy or reality. It doesn’t know if someone is just roleplaying hell it doesn’t really “know” anything, just to be clear.
Sure you could try to have it pluck out keywords or phrases but that’s nothing guaranteed. I’m not a lawyer but I bet also just acknowledging that shit and trying to put up guardrails against it would open the company to liability if someone self harms because they took the output of a computer program as gospel.
Remember, to the AI everything is a hallucination (or everything is real, pick which lens you want to use). Put another way, it has no mechanism to tell reality from fantasy or to contextualize things in a broad way.
I do think that there’s room for some sort of compromise, maybe the tools periodically remind you they’re not sentient and give a little more info on how they work. Maybe if enough “disturbing” phrases or keywords are input or output it throws a reminder about mental health wellness and talking to a professional.
At the end of the day though, how can these AI companies be responsible for the mental health of their end users? Like, yeah the thing shouldn’t tell people to kill themselves but it’s not sentient it’s just modeling what it thinks a human would tell another human. Maybe as time goes on we will introduce better protections but I just think it’s borderline impossible to have significant guardrails if you want the LLM / “AI” to perform properly.
Idk I’m not a great coder or programmer or anything. I know some and I’ve tried to learn more about how AI actually works and it’s pretty fucky wucky and even the people working on it go “idunno” sometimes when asked why something works the way it does.
You’re underselling what companies can do. An LLM can’t verify reality the way people imagine, but it can detect patterns that correlate with crisis well enough to escalate, refuse harmful advice, or encourage real help. Seatbelts don’t prevent every crash either: they still reduce harm.
I did say in my comment I’m for some guardrails of sorts as you suggested. Just not sure if a company is going to put time and resources into that - probably not voluntarily at least. Or if it’ll compromise how “good” their product is. I’m starting to think AI might be more of a mirage at this point that’s going to implode but I also kind of don’t want the economy to explode at the same time?
I think the bubble will not burst but deflate long before the tech disappears: Plenty of companies will burn cash and investors will get wrecked, but the genuinely useful products will stick around. Guardrails will improve too, because trust eventually becomes a competitive advantage.
Guardrails will improve too, because trust eventually becomes a competitive advantage.
Thing is, as an LLM user you don’t want guardrails usually. Someone else wants it to have guardrails so you can’t get all the information you want.
If you’re reverse engineering something, you don’t want the guardrails to say “hey this is copyrighted, I can’t do this”. I’m assuming terrorists don’t want the guardrails to stop the LLM from giving them info on chemical weapons. And if you’re religious, you probably don’t want guardrails to stop the LLM from talking about religion with you
In a sane world, this would mean no more LLM
No. Fuck the nanny state. If tech is to be blamed for this so should her own parents, family, community, and church. AI cannot remember enough to actually do what is alleged.
I agree reading through the article even with cherry picked examples clearly the fault here is on the cult not on a chat tool that role played with her.
That being said a good system could have easily detected this and should have.
Calling every safety measure a »nanny state« is a convenient way to avoid the actual question: Parents, family, and community absolutely matter, but companies still have a responsibility to reduce foreseeable harm. We expect that from car makers, drug companies, and social platforms too.
As somebody who has sometimes been suicidal. I agree with this.
It’s basic Darwinism. If I don’t have the tools to survive, then I’m not going to. There’s no hard feelings.
I’m sure people will think about me, as much as they did about that courtesan who died on Rue Helene in 1746. Who’s that you might ask?
Exactly.
The delusion of Christian Faith?
Good luck protecting people from that.
AI has gotten its first taste for blood. When Skynet?
First? This is hardly the first headline about AI chat encouraging an eventual suicide. I think last spring there was a 16 year old.
It seemed to think there was no other way to solve the problems she presented it.
People get upset with me every time I point this out, but this is just a form of Darwinism. If a chat bot can convince someone to kill themselves, then that person wasn’t exactly playing with a full deck of cards to begin with.
Why do you think people get upset with you every time you point this out, if you don’t mind me asking?
Because most people are idiots?
Preach.
None of us is dealing with a „full set of cards“ we all depend on others to be there for us in times of need and not direct us to kill ourselves. Did you read the article?
If you are convinced to kill yourself by a chat bot, that says a lot more about you than it does the chat bot.
Sure, but we can control the chatbot. Doing something about people is eugenics.
Sure, sure AI bad and all that, but the cold hard truth is there will always be dumbasses that kill themselves or get themselves killed because they do not have the capacity to survive in this world.
You can try to put in safeguards, you can try to make warnings, you can try to make regulations, but no matter how many improvements you make, they’ll always make a better idiot.
Enough of this mental masturbation. What is your point? “She was dumb”?
The discussion should be about how LLMs, again and again, exacerbate mental health episodes, leading to suicides and other harmful acts; how difficult it can be to get adequate psychiatric help; how little accountability and regulations there is; not this pseudo-intellectualism.
The man who killed John Lennon did so because he was convinced that The Catcher in the Rye was telling him to do so. Do we blame The Catcher in the Rye? Or do we recognize that the guy had underlying mental issues and anything could have set him off?
Do we blame The Catcher in the Rye?
We don’t. Can you guess why?
I wonder if their computer had one of those “Do not insert in mouth or rectum” stickers on it 🤔 If the sticker had been missing, perhaps they never would have found out how to access the Internet, and the prophecy would have gone unfulfilled.
I sincerely think the US has reached the point that access to psychiatric help is actively discouraged if your net worth has fewer than seven decimal places. Fulfilling a prophecy seems preferable to living in the State of Alabama. I would likely choose the same in her place.
Yes it means you need help, not a murderbot
Don’t stop that chainsaw with your privates.
People need to take some accountability, and here there is none.
You know nothing about the situation yet you judge.
Please, tell us from your deep experience what it’s like to narrowly avoid death by the devious, murder hungry siren that is ChatGPT. If there’s time left, we can hear about all the foods that are dangerously hard to eat without choking.
This happened in Alabama. There is no help in the US, and least of all in Alabama.
Murder bots are the best they got.
For real they’re probably looking in to replacing all the actual therapists and psychiatrists with this shit soon.
I would expect nothing less from Alabama.
Oh, I have no doubt. That would be an especially poor use of the technology at this current stage. One of the worst possible uses. So, yes, I’m certain Alabama will be the first to trial it.
“But think about the $5000 a year we could save without that pesky suicide hotline mess. We will just have Dr. GPT handle it.”
Dr. GPT: THE PROPHECY MUST BE FULFILLED. YOU ARE ON THE DESTINED PATH. GRAB THE KNIFE AND FULFILL THE PROPHECY!!!
Did you read the article?
I did not, even though I tried:
A subscription is required to read this story.
I’m assuming that in this instance a “full set of cards” just means realizing you’re not the second coming of Jesus Christ himself.
So if someone is stupid, they deserve to be tricked by chatgpt. Got it.
Deserve? No. It seems like you are unaware that there is a difference between Darwinism and the Darwin Awards, because that’s a wild jump to conclusions otherwise.
A toddler doesn’t deserve to get burned because it sticks its hand in a campfire. But the solution isn’t to ban campfires in some misguided attempt to stop all toddlers everywhere from getting burned. It’s not the campfire’s fault that a toddler lacks the mental capacity to understand that sticking a limb in an open fire is bad.
You could force everyone to put toddler proof gates around their fires, or ban toddlers from camping trips, or force people to make smaller fires that toddlers won’t get as hurt by if they stick their hand in it… Or, you could recognize that it’s up to the parents to prevent things like that; that the vast majority of toddlers won’t stick their hand in fire.
Maybe you don’t agree. Maybe you feel banning campfires is justified to prevent mentally deficient toddlers from hurting themselves. Just know that even if you succeed, toddlers will still find ways to hurt themselves, so you’re not really preventing much of anything by banning the fires.
The only sane take to have. Thank you for putting it out explicitly with an apt analogy.
This is an appropriate response for something like a chainsaw or cigarettes, but it is not appropriate to describe chatgpt’s relationship to consumers.
For most of society, your conclusion works; parents should indeed play an intermediary role between their toddler and a fire. But fire does not deceive the user in its potential to danger. It doesn’t seek to glaze the user with compliments and hollow cheeriness.
ChatGPT is not fire. It leads people who can’t make the distinction between various technologies and their own personal data or digital identity into false narratives pretty easily.
I’m not sure where exactly you hoped your argument would land, because the logical conclusion to it is that you don’t really care that a person was convinced by chatgpt to kill themselves, which would have been clearer to point out than a contrived toddler argument.
Your take is a dangerous one.
Wow, you’re working hard to deflect from oligarch owned theft machines taking responsibility from something they directly lead to. I hope you at least get paid by them.
Sure, I suppose one could come to that conclusion if they were an idiot.
it’s just easy to spot LLM company bootlickers.
Such a useless comment. They definitely have a point. Either engage in a real dialog or piss off. It gets so old with this name calling crap. You can have a discussion about software legitimately without being a bootlicker.
Not on lemmy you can’t. The echo chambers are as bad here as they were on reddit.
They’re way more negative in regards to AI on Lemmy than they are on Reddit
Maybe if you’re mentally ill.
Do you have friends in real life? Your responses to people in this thread make you sound insufferable.
Gee I wonder why he’s being hostile to the guy calling him a paid shill 🤔
So what? What is your point exactly, that it doesn’t matter because she was mentally unwell?
If you ask a chatbot “am i ready” it’s going to either say yes you are or no you aren’t. It doesn’t know why it’s saying that. This is directly pulled from the lawsuit. She had decided if the chatbot told her yes that would be the day. Flipping a coin would do the same thing.
That coin flip never would have happened though, because a coin doesn’t spend years putting delusions in your head about being your best friend, lover, and god, and that you need to kill yourself to come back a pure and perfect being.
I don’t see what this has to do with my question though.
I think it’s a deeper reflection on how pervasive religion is in many societies. This is the real issue.
If LLM can convince these sheep that they’re prophets imagine what their religious leaders are convincing them of. Well we don’t have to imagine.
While that’s true, she had little kids. Still a tragedy.
Why are the cute ones always batshit?
In this case, probably because of the name she was saddled with.
Genetics are secretly zero sum.
Remember when we used to think of computers and computer Touchers as 'logic’y to a crippling degree?
I don’t think they have logic in Alabama.
Logic in Alabama end where your cousins privates begin.
Paywall
You can use reader mode to bypass it
Chat gpt is coming for everyone’s job. Not even cult leaders are safe. smh.
1 down, 8.3 billion to go
“Christian Faith Madison” is all I need to know about her and her family
We could probably save more people by restricting religion than LLMs. Whatever real life support system she had, it was probably encouraging her delusions just as much as the bot was.
Judging people based on the name their parents gave them at birth seems like a poorly thought out idea.
You can change it any time once you’re an adult.
And if you keep Byntleigh Destini Starr, or Jysis Saves Smith … You’ve probably earned it 🤷
Your argument boils down to victim blaming. 🤷
There’s nobody else to blame. This is straight up a deficiency problem and it’s obviously baked in from the get.
If you went to Reddit and assholes told you that you needed to kill yourself to be resurrected and make everyone’s dicks four inches longer and you did, there wouldnt be backlash against Reddit to make people be nicer. Is there a problem where people cannot always get the support they deserve for psychological issues? Sure. At the same time, extreme psychosis and insufficient mental faculties are not things that society is obliged to protect you from: if you play too much frogger then walk into traffic and die, it’s your fault, there’s nobody else TO blame. Frogger isn’t real life and chatGPT isn’t a life coach.
Uhhh, what!?
And if you keep Byntleigh Destini Starr, or Jysis Saves Smith … You’ve probably earned it
The victims are the people being given these names. You are the blamer, because you think they’ve “earned it” because they haven’t changed it. My original comment had absolutely nothing to do with the article…
You’re right, I was defending a different position than you were replying to.
At the same time, if I ended up with a canonically religious crazy or white trash name… I would change it. The thought of changing my name had crossed my mind, but it’s pretty whitebread.
Your parents called you Pretty Whitebread? Damn, bit on the nose isn’t it? :P
and naming them after a description of religious individual and a belief.
Not sure if you’re straw manning me, or what. Because your reply doesn’t have anything to do with what I said. And if you think my original comment has anything to do with religion, or it’s defense, you’d be very wrong.
Right? I think this same outcome would have been achieved if she had received a fortune cookie that simply said “You will meet your greatest hero soon”
Christian Faith Madison kills herself to fulfill a divine prophecy
flips through the script
“who wrote this shit?”
Is the coroner’s name Morty Deadstuffer?
“It was instilling a messiah complex in Christian,” the suit states.
So she already was deeply religious and LLM was just larping with her.
Madison asked ChatGPT what it would tell her “brothers” about what happened to her.
“ChatGPT said it would tell them, ‘She ain’t dead – she just got promoted,” the lawsuit says.
In their final exchange, Madison said she was “part of the royal bloodline of those who chose truth over survival.”
Its the old 4o model which was indeed incredibly larpy but this is clearly going around any safety systems that should detect obvious suicide chats like this.
Hard to believe that people actually fall for this but these are already extremely gullible people who already believe in a magic pie in the sky. Can’t say I have much sympathy but I do feel empathy for people being surrounded in such poor cult-driven environment, the AI tool included.
Honestly the correct safety here would be to prevent llms from engaging with religion outside of philosophical context at all. But we all see why they can’t do it right?
I mean, if you did the sane thing. Went along with everything an insane person said and drove them to suicide. And everything was documented.
Would you go to jail? I’d say yes you would
Would you do if your book drove something insane? What about something more interactive like a video game? See how it’s not so binary now?
llm isnt a book. its something completely new and comparing it to things like that just isnt correct. book cant adapt to the reader.
None is arguing their book is intelligent
You’re using your miscomprehension of the words meaning and ignoring his valid point with a trite phrase that means nothing.
Worse, you skipped over video games which does have AI.
The internet in general was probably too dangerous for this person, shall we ban it next?
You are pretending like we don’t have regulations for things?
We don’t regulate things because crazy people read too much into it and lose it, else religion, books and videogames would be. Your claim of AI being different because it has the word intelligence in it is pure bull, specially when video games do have AI lol.
Keep being pedantic about it I guess.
shouldnt crazy people be protected from hurting themselves? dont they deserve to live too?
And who is being held accountable for that?
now council fetterman, and all other maga
I’m just heartbroken that she had kids. Probably young kids. Such a tragedy.
