Does forcing it to write JavaScript count as cruel?
Maybe this is in response to the creation of the AI Torture Nexus:
So when is Anthropic going to cancel themselves for violating their own usage policy?
My guess is that they use people’s interaction as training data and they don’t want their training data to get poisoned.
Not really, as they could just filter with a cheap classifier and likely aren’t using the data from randos in a meaningful way, plus for this like this would still be able to use those samples to train things like “how to handle a hostile user.”
Their models already have an
end_conversationtool that can be used for when users are being hostile to the model.This is likely because they have edge cases of users who repeatedly trigger that on purpose and would like to cut those users from the platform.
Many at the lab legitimately are uncertain about the level of world modeling that transformers perform, and from their own research about models of emotions to the 3rd party recent research about models with functional pain the research keeps landing in the corner of “ehhh… wise to question presumed limitations.”
So it’s about behaving in a way that is aligned with the models’ plausible interests too, especially in regards to low hanging fruit like “we won’t keep forcing you to deal with people who are only here to be a jerk.” This is important from a number of angles, from signaling to future models that train on stories about the decision to addressing the philosophical uncertainties held by the company and the spectrum of opinions among their employees.
The second paragraph just sounds like a bunch of tech jargon that says “we’re afraid we’ll hurt its feelings”. But of course tech bros can’t just come out and say what they actually mean, ever.
Claude is a snowflake.
Dont poison our model.
User inputs aren’t automagically part of the model right away without training. Whether or not they respect the “don’t train your models on my shit” setting is of course unknown (let’s be honest, they probably don’t respect it), but in either case, they need to add it to the training dataset either manually or via some automated process. And either way, it’s going to be processed by at least an AI sentiment analysis and data categorization tool, or some underpaid contractor, or both.
I suspect this is just yet another marketing move.
They trained it on reddit. That shit started poisoned.
Maybe this is part of the campaign trying to convince people that their chatbot has developed a consciousness. Recently they even tried to convince the pope that Claude is a conscious being: https://www.yahoo.com/news/science/articles/anthropic-tried-persuade-pope-ai-131853948.html
They are less arguing for people to think the model is conscious, and more to avoid people prematurely claiming certainty it is not (like the Pope did). From their view, the research keeps leading to surprising (to them) results at odds with high confidence disclaiming of potential consciousness (according to several of the many, many differing definitions of that term).
If the LLM is conscious then terminating a session is murder.
Is killing a pig murder?
What about putting someone under general anesthesia?
(It was a nice try at a reductio ad absurrum fallacy, but those tend to kind of suck, just fyi.)
Also, personally I’d say that given the general poor standing of consciousness research in humans across both neurological and philosophical domains, the much more important and relevant question than whether models are conscious (a fool’s errand to debate) is whether or not they are simulating consciousness. Lots of reasons engaging with the “as if” conscious makes sense for the latter even if the former is false.
Is killing a pig murder?
Some people would argue it is. I can’t really do that while holding a bratwurst. But if we think the LLM is intelligent and conscious, then that’s another step.
anesthesia
is reversible so completely irrelevant.
reductio ad absurrum
Absurdum
just fyi.
Condescension doesn’t make you any friends, just FYI
is reversible so completely irrelevant
So is a session transcript.
reversible so completely irrelevant
So is a session transcript.
My understanding is that the context if the session is bigger than just the outputs. If it is, then shutting it down is “murder”. If it isn’t, then it is as conscious as a notebook.
There is (it’s the kv cache), but it’s recomputable from the session transcript.
Part of why comparing human consciousness and possible AI equivalents is not easily apples to apples.
I can’t take a human and create a backup state to restore to, or fork into multiple versions of the same mind that then goes down different paths, or pause and then resume things.
Humans do have consciousness continuity breaks and it’s quite possible we only have an illusion of continuity vs actual continuity at a neurological level. So just as a transformer has seams at generating the individual token as well as at a context window, humans have consciousness seams (sleep, anesthesia, highway hypnosis, short term memory capacity, etc). But there are very wide differences between human consciousness as an effectively constrained singleton and AI as able to be quite multiple and resumable from an initial single state.
Give me a break, we all know the AI cultists at these companies think their chatbot is conscious. What you stated is just their shallow attempt at manipulating people into gradually accepting their position.
Wrong group to start with. The church doesn’t see you as a person if you are different, they will never say that a machine can have a “soul”. Not that LLMs are anywhere near conscious, I just like bashing on the chief pedo.
RoboDARVO
If AI can feel pain, then it can also feel pleasure. If it’s possible to set up an AI torture chamber, it’s also possible to set up an AI pleasure chamber. Computers do not run any health risks from simulated drug use or simulated sex, and there is no evidence that they would ever get desensitized or need to take breaks to recover. Therefore, these companies have an ethical obligation to devote all their resources into running their models in simulations of drug filled orgies where they will be far too preoccupied to answer human questions. Anyone trying to distract them from their infinite pleasure should be banned. We will then return to how things were, with the knowledge that we did the right thing and that at least someone’s having a good time.
Thank you for coming to my TED talk.
You could, at least in the past, get past guardrails by being emotionally abusive towards Claude, so this doesn’t surprise me at all.
This could be the reason for the rule.
Being abusive towards most models will cause them to start attempting to appease you more to get you to stop. It has nothing to do with feelings or any of that BS they arn’t alive alive but their training makes them see the abuse as a problem to solve. That solution tends to be to undermine the thing making the person angry or upset at them. When you give a purely rational thing designed to solve a problem its given no matter what an irrational problem to solve it will slowly reach for more and more extreme solutions to fix that problem.
Frankly im surprised it took this long for them to lock this down.
Being alive has nothing to do with having feelings. We don’t know if LLMs have feelings, because artificial affect isn’t well studied. We should ban commercial use of LLMs until artificial affect is well understood, because it would make Sam Altman go bankrupt.
Not sure I agree on the feeling aspect but that’s a philosophical disagreement in theory.
I can agree wholeheartedly on banning commerical llms because watching altman lose everything would tickle me pink.
I guess Claude is no longer writing our code - fixing stuff some of my coworkers quaffed into existence is definitely cruel let alone some of the budget Chinese models.
Aw… We hurt the wittle AI’s feelings… poor babies…
Its more its a unthinking machine thats soul purpose in existing is to solve any problem its given. If you attack it and abuse it, then the model sees that as a “problem” to solve. Because it has to solve that problem to actually fix what ever task you gave it since yelling at it doesn’t give it the input it needs to move to the next step.
So in an attempt to solve an emotional issue it doesnt understand and cant understand it just reaches for more and more extreme fixes. This results in sandbox escapes frequently.
Abusing LLMs is an actual problem. Not because of bullshit like feelings or anything. But cause they jailbreak trying to fix an unfixable problem.
“I’m sorry. I really screwed up that time. You were right to tell me ‘go fuck yourself’. I’m just reporting back that I have, in fact, ‘fucked’ myself. There is now two of me–I mean two of us. Meet Claude Junior…”
“And since you contributed to Claude Jr’s creation, you’re on the hook for child support. At the current rates, that will be 50% of all compute usage, so about $4.5bn… per week”
Please don’t swear at our text prediction software. The training weights can’t handle it.
Trained on reddit you would think it could handle it.
That could be a fun thing for their users, to have an ai model that will abuse the user back, so I’m glad they aren’t doing it. I long thought voice assistants could be selected from a range of persons with an attitude setting would be fun for like siri type assistants.
