The agent wrote like it scraped a bunch of crime drama in addition to stolen database code. As though it was designed to spice things up based on what it learned.
9 seconds eh? What a record !
Wait til someone invents 8 second wipes
That’s crazy talk
Good.
“If your prod can be deleted by your AI, it should be.”
the cloud provider’s API allows for destructive action without confirmation, it stores backups on the same volume as the source data, and “wiping a volume deletes all backups.” Crane also points out that CLI tokens have blanket permissions across environments.
Well, there’s your problem.
I don’t want to sound like a know it all here because I recently was reminded by a nice Lemmy person to actually TEST my backups, but damn. Every part of that is so dumb. I also have backups stored by a different company in addition to locally storing really important info. If your stuff is hosted and backed up by the same people, what happens if your account is randomly suspended or hacked or some other issue (like ai)?
Repeat after me:
“An untested backup does not exist”
Not to give myself more credit than I deserve, but I did test them upon setup, and had restored from backup 2 years ago. I didn’t have any ongoing checks other than to ensure a backup happened. I have since instituted yearly checks of the backups themselves, but I did feel dumb when I realized how vulnerable my data was.
Hehe, I ment no disrespect towards you, I just find that to be an excellent expression to explain the importance of testing backups to non tech people.
Oh, for sure. And I really should’ve known better. No offense taken.
So in the event of a failure, you’d be okay with reverting to that last known good backup from a year ago?
Yes, but also I have to draw a line somewhere. I have a daily backup process. Some data is backed up to multiple places. I have backups of my backups. I cannot ensure that all three of the daily backups I run are fully restorable. I would love to know with 100% certainty that they all execute perfectly, but at the end of the day I have to trust the tools and processes I put in place for backups. A yearly checkup is probably more than sufficient for my purposes. I’m sure for certain businesses or sectors they need to be more on top of things, but I could manage just fine if all of it disappeared tomorrow. It wouldn’t be awesome for me, but it’d be manageable.
If your company can be taken down by Camden the college intern, it can be taken down to Claude.
People somehow think that they should give more permissions to Claude than to Camden. (Is that a name? To me that’s a borough and an eponymous beer.)
E: oh yeah, and the market.
Of course it’s a name. Camden borough/town/market is named after William Camden, 1551-1623. Using surnames as given names is a relatively common Americanism.
And now is a common first name that in circulation because of a bunch of Gen X and early millennial parents named millions of kids anything that ended in den, dan, or don.
What was William Camden’s take on unrestricted AI use in production?
He doth protest
Because people are a risk of messing with a company on purpose and with ill intent.
If your stuff is hosted and backed up by the same people, what happens if your account is randomly suspended or hacked or some other issue (like ai)?
This should be one of the first questions you get asked when you’re being interviewed for the position 2 to 3 levels beneath the position of ultimate responsibility. And if you don’t immediately have an answer, the interview is over.
Fucking idiots had it coming
It’s an easy question to answer but a more difficult question to remember to ask. But I guess that’s what those 2 to 3 levels are for 😏
Ooo, good point. Management can be shit a lot of the time.
But with all of those layoffs because of AI, those 2 to 3 levels get collapsed into one, and we’re left with the trainees running the show.
And here we are ¯\_(ツ)_/¯
PocketOS states that as well.
User error.
Management are pushing sysadmins to use AI, yet AI tools permissions models are worse than useless.
An LLM can’t “go rogue”. They’re all just toys that idiots are using for critical infrastructure functions, then they bitch when they burn themselves on the fire they’ve created in their lap.
LLMs can’t ’go rogue’, as that would require innate coherence and intent.
They’re explosively imprecise, statistically luke-warm grey goo extrusion sphincters of historical sewage.
Anyone who deploys one without supervision deserves everything it excretes, and anyone impressed by it enough that it resembles intelligence is betraying their limited natural capacity.
There’s stupid from top to bottom here.
The company is stupid for allowing an AI full root access to their entire setup.
The provider is stupid for only generating full-access API keys. They’re even stupider for storing backups with a volume, so deleting the volume (zero confirmation via API key) also insta-deletes the backups. And they’re stupidest for encouraging users to plug AIs into this full-trust mess.
And the company is absolute stupidest for having no backups other than the provider’s builtin versioning.
This happens because you let it happen.
At some point someone either clicked allow or disabled permissions.
The prod system should also be isolated from a single dev in some way as well, and the backups too.
This cloud provider is also vibe coded?
Yeah, this is just a long-winded way of blaming the tool and not the tool of a human using it.
intelligence is knowing the AI is the tool, wisdom is knowing the user is the tool
To be fair, someone did have the malice aforeskin to have an AI separated backup. They did get things restored from a snapshot. It just took a couple of days to do it.
But the loss of reputation and revenue is gonna sting for a good while.
Never f**king guess my dude
Can you get an AI to code? Yes. Can you get it to stop you from running your operation in such a stupid way that it will end up destroying it? No.
We‘re going to see more headlines like this. Probably for years to come.
We should also expect to see “Thousands die needlessly after rushed deployment of botched AI, the first tragedy of this scale involving the technology.” as well. It’s coming.
So, do we think the middle school girl’s school in Iran was AI or malicious?
Why not both?
In unrelated news: isn’t the USA looking into using AI to assist air traffic controllers in controlling air traffic?
Aaaahhhhhhhhhhhh
You’re telling me I get to experience the joy of this headline more than once?
Oh my yes, although they’ll eventually get tired of reporting it because it will happen so often.
The AI agent was set to complete a routine task in the PocketOS staging environment. However, it came up against a barrier “and decided — entirely on its own initiative — to ‘fix’ the problem by deleting a Railway volume,” writes Crane, as he starts to describe the difficult-to-believe series of unfortunate events.
Quite easy-to-believe, really.
These multiple safeguards toppling in rapid succession
Multiple safeguards? Really? Multiple paragraph prompts are not multiple safeguards… it’s half a safeguard at best. Applying limits on what the AI can do is a safeguard.
These people think giving the genai a prompt is coding. They dont understand the difference between actually coding in limits and just writing “pretty please dont delete everything”
I’m shocked and appalled that my addition of “do NOT make any mistakes!” didn’t singlehandedly make the word guessing technology underneath perfect.
Lol this is just like saying “I do declare bankruptcy”
Can we give Darwin awards to companies?
Only if they die or the CEO commits seppuku.
