Discussion
A rogue AI led to a serious security incident at Meta
jasonpeacock: I'm concerned that someone had the permissions to make such a change without the knowledge of how to make the change.And there was no test environment to validate the change before it was made.Multiple process & mechanism failures, regardless of where the bad advice came from.
welfare: Behind paywall, is there another link to the article?
krupan: I hit back, clicked the link again, and it let me through
yomismoaqui: https://archive.is/A2hmz
krupan: "A human, however, might have done further testing and made a more complete judgment call before sharing the information"Because a human would have been fired for posting something that incorrect and dangerous
Uhhrrr: The two errors, then, were that the LLM hallucinated something, and that a human trusted the LLM without reasoning about its answer. The fix for this common pattern is to reason about LLM outputs before making use of them.
JKolios: "A rogue AI led to a serious security incident" is certainly a way to write "Someone vibe coded too hard and leaked data".
SlinkyOnStairs: > The fix for this common pattern is to reason about LLM outputs before making use of them.That is politics. Not engineering.Assigning a human to "check the output every time" and blaming them for the faults in the output is just assigning a scapegoat.If you have to check the AI output every single time, the AI is pointless. You can just check immediately.
somewhereoutth: However - Automation bias is a common problem (predating AI), the 'human-in-the-loop' ends up implicitly trusting the automated system.
leptons: If "the level of awareness that created a problem, cannot be used to fix the problem", then you're asking too much if you expect a human to reason about an LLM output when they are the ones that asked an LLM to do the thinking for them to begin with.
krupan: If you have to do all that, then what's the point of the AI? I'm joking, but I'm afraid many others say the same thing 100% seriously
advisedwang: AI can be used to move fast. So management expects us to move at that speed. AI can be used to move even faster if you don't check it's output. The ever ratcheting demand for faster output will make it infeasible to diligently check AI output all the time. AI errors being acted on without due care is inevitable.
yieldcrv: very misaligned! sprays bottle at mac mini
amelius: How long until an AI puts all our personal data on the streets?
paxys: But funny enough the person who was responsible for setting up the bot will likely face no repercussions. Will probably be rewarded instead for transitioning their team's workflows to AI.
paxys: A big problem now both internally to a company and externally is that official support channels are being replaced by chatbots, and you really have no option but to trust their output because a human expert is no longer available.If I post a question to the internal payment team's forum about a critical processing issue and some "payments bot" replies to me, should I be at fault for trusting the answer?
RussianCow: I know this is happening with external customer support, but is this really happening internally at big companies? Preventing you from talking to a human in the correct department about an issue feels like a bomb waiting to explode.
krupan: It's more like, the LLM "hallucinated" (I hate that term) and automatically posted the information to the forum. It sounds like the human didn't get a chance to reason about it. At least not the original human that asked the LLM for an answer
c-linkage: If you don't like hallucinate, try bullshit. [NB: bullshit is a technical term; see https://en.wikipedia.org/wiki/On_Bullshit]https://www.psypost.org/scholars-ai-isnt-hallucinating-its-b...
krupan: That is my preferred term, but it seems to derail discussions that might have otherwise been productive (might...the hope I have)
fhd2: Well, I'd say there's two dimensions:1. Check frequency (between every single time and spot checks).2. Check thoroughness (between antagonistic in-depth vs high level).I'd agree that, if you're towards the end of both dimensions, the system is not generating any value.A lot of folks are taking calculated (or I guess in some cases, reckless) risks right now, by moving one or both of those dimensions. I'd argue that in many situations, the risk is small and worth it. In many others, not so much.We'll see how it goes, I suppose.
ex-aws-dude: This agent stuff is really making me lose respect for our industryAll the years of discussing programming/security best practicesThen cut to 2026 and suddenly its like we just collectively decided software quality doesn't matter and its becoming standard practice to have bots on our local PC constantly running unknown shell commands
aeblyve: People salivate so hard at the thought of the high level of automation promised that they're willing to do away with privacy altogether and live in Data Communism.My thinking is, this will increase the demand for backup and other resilience solutions.
worik: > A rogue AI led to a serious security incident at MetaThe AI "led to" the incident , true. But do nt forget that this, like all similar incidents , is a human failureAI is a tool with no agency. People make mistakes using it, thone mistakes are the responsibility of the humans
Apocryphon: Turns out all of the frenzy of the ZIRP era is piddling compared to what happens when ZIRP is taken away.
nytesky: I’m not in AI, but what is happening is that it is building output from the long tail of its training data? Instead of branching down the more common probability paths, something in this interaction had it travel into the data wilderness?So I asked AI to give it a good name, and it said “statistical wandering” or “logical improv”.
testplzignore: Our industry has never been serious about security. We all download and run unvetted code via package managers every day. At least now the insanity is out in the open. We won't change until Skynet fires off the nukes.
dgxyz: This is never getting to skynet launching the nukes stage. It's not that clever and never will be.Humans will kill us by it damage amplifying their worst characteristics.Thus we'll die of a pandemic because some idiot LLM'ed up positive looking virology data when they were being too lazy to verify something. Everyone will trust it because they don't really care as long as it looks about right.
antonvs: The media isn’t helping. This wasn’t a “rogue AI”. It was a system that was given permission by a human operator.We don’t say “a rogue plane killed 300 people today when it crashed into a mountain”.The only difference in the AI case is that some people are attempting to shift blame for their incompetence into a computer system, and the media is going along with it because it increases clicks.
