Discussion

My minute-by-minute response to the LiteLLM malware attack

Fibonar: Callum here, I was the developer that first discovered and reported the litellm vulnerability on Tuesday. I’m sharing the transcript of what it was like figuring out what was going on in real time, unedited with only minor redactions.I didn’t need to recount my thought process after the fact. It’s the very same ones I wrote down to help Claude figure out what was happening.I’m an ML engineer by trade, so having Claude walk me through exactly who to contact and a step by step guide of time-critical actions felt like a game-changer for non-security researchers.I'm curious whether the security community thinks more non-specialists finding and reporting vulnerabilities like this is a net positive or a headache?

dot_treo: Looks like we discovered it at essentially the same time, and in essentially the same way. If the pth file didn't trigger a fork-bomb like behavior, this might have stayed undiscoverd for quite a bit longer.Good thinking on asking Claude to walk you through on who to contact. I had no idea how to contact anyone related to PyPI, so I started by shooting an email to the maintainers and posting it on Hacker News.While I'm not part of the security community, I think everyone who finds something like this, should be able to report it. There is no point in gatekeeping the reporting of serious security vulnerabilities.

dmitrygr: Consider this your call to write native software. There is yet to be a supply chain attack on libc

woodruffw: This is presumably because libc just doesn't change very often (not meaning code changes, but release cadence). But the average native software stack does have lots of things that change relatively often[1]. So "native" vs. not is probably not a salient factor.[1]: https://en.wikipedia.org/wiki/XZ_Utils_backdoor

ddp26: Sure, but this is a pretty onerous restriction.Do you think supply chain attacks will just get worse? I'm thinking that defensive measures will get better rapidly (especially after this hack)

simonw: First time I've seen my https://github.com/simonw/claude-code-transcripts tool used to construct data that's embedded in a blog post, that's a neat way to use it. I usually share them as HTML pages in Gists instead, e.g. whttps://gisthost.github.io/?effbdc564939b88fe5c6299387e217da...

Fibonar: I’m a big proponent of it within our company! CC tried to style it to blend in with our blog but it was kind of a disaster. Definitely had a new appreciation for the out-of-the-box experience. I also tried to include the individual sub-pages of Claude investigating but it really trawled my whole machine looking for malware. Don’t know if you’ve thought of any systematic ways of redacting the endless pages of detailed logs?

Fibonar: The best part was that I didn't even mean to ask Claude who to contact! I was still in disbelief that I was one of the first people affected, so I asked for existing reports on the assumption that if it was real I definitely wasn't the first.The fork-bomb part still seems really weird to me. A pretty sophisticated payload, caught by missing a single `-S` flag in the subprocess call.

MACCRE: Hello all, I was hoping to get some community feedback on the tools that ive been developing, especially in light of the litellm compromise. I published my authentication layer last night but since I dont have a social media presence im having trouble getting any engagement. The authorization layer that is part of my native tool stack was designed to avoid relying on things like litellm in an effort towards as a data sovereignty and threat mitigation. Any feedback whatsoever, even if negative, would be greatly appreciated.https://news.ycombinator.com/item?id=47526672

everforward: I think that article proves the opposite.> While xz is commonly present in most Linux distributions, at the time of discovery the backdoored version had not yet been widely deployed to production systems, but was present in development versions of major distributions.Ie if you weren’t running dev distros in prod, you probably weren’t exposed.Honestly a lot of packaging is coming back around to “maybe we shouldn’t immediately use newly released stuff” by delaying their use of new versions. It starts to look an awful lot like apt/yum/dnf/etc.I would wager in the near future we’ll have another revelation that having 10,000 dependencies is a bad thing because of supply chain attacks.

Bullhorn9268: The fact pypi reacted so quickly and quarantined the package in like 30 minutes after the report is pretty great!

notatallshaw: > I had no idea how to contact anyone related to PyPIhttps://pypi.org/security/:> If you've identified a security issue with a project hosted on PyPI Login to your PyPI account, then visit the project's page on PyPI. At the bottom of the sidebar, click Report project as malware.

0cf8612b2e1e: The existing account to report is an unfortunate obstacle. Presumably not a huge deal if you were auditing code for vulnerabilities, but still an annoyance.

notatallshaw: The threat actor was sophisticated enough to spam GitHub issues with dozens of different accounts. I imagine they could completely overwhelm PyPI with unauthenticated reports.

S0y: > Where did the litellm files come from? Do you know which env? Are there reports of this online?> The litellm_init.pth IS in the official package manifest — the RECORD file lists it with a sha256 hash. This means it was shipped as part of the litellm==1.82.8 wheel on PyPI, not injected locally.> The infection chain:> Cursor → futuresearch-mcp-legacy (v0.6.0) → litellm (v1.82.8) → litellm_init.pthThis is the scariest part for me.

lq9AJ8yrfs: As a sometimes peripheral and sometimes primary program manager for vulnerability disclosure, for companies you nearly can't avoid, $0.02 follows.It's a signal vs noise thing. Most of the grief is caused by bottom feeders shoveling anything they can squint at and call a vulnerability and asking for money. Maybe once a month someone would run a free tool and blindly send snippets of the output promising the rest in exchange for payment. Or emailing the CFO and the General Counsel after being politely reminded to come back with high quality information, and then ignored until they do.Your report on the other hand was high quality. I read all the reports that came my way, and good ones were fast tracked for fixes. I'd fix or mitigate them immediately if I had a way to do so without stopping business, and I'd go to the CISO, CTO, and the corresponding engineering manager if it mattered enough for immediate response.

RALaBarge: Maybe the people who use emacs for everything are the only safe ones?

dmitrygr: your link disproves your claim. no naive app depended on xz version >= latest. Most sane distros take time to up-rev. That is why the xz backdoor was, in fact, in NO stable distroAnd not changing often is a feature, yes.

woodruffw: I don't think it does; I think the industry opinion on xz is that we got lucky in terms of early detection, and that we shouldn't depend on luck.(I don't know what a "sane" distro is; empirically lots of distros are bleeding-edge, so we need to think about these things regardless of value judgements.)

dmitrygr: Sane: debian-stable

cdcarter: If it weren't for the 11k process fork bomb, I wonder how much longer it would have taken for folks to notice and cut this off.

intothemild: Thats the thing, i noticed it almost instantly when trying to install a package that depended on it, as soon as it started, it hard locked my laptop, didn't get to infect it.. but if they had slowed down that fork bomb.. it would have done more damage.

__mharrison__: Interesting world we live in.I just finished teaching an advanced data science course for one of my clients. I found my self constantly twitching everytime I said "when I write code..." I'm barely writing code at all these days. But I created $100k worth of code just yesterday recreating a poorly maintained (and poor ux) library. Tested and uploaded to pypi in 90 minutes.A lot of the conversation in my course was directed to leveraged AI (and discussions of existential dread of AI replacement).This article is a wonderful example of an expert leveraging AI to do normal work 100x faster.

cedws: GitHub, npm, PyPi, and other package registries should consider exposing a firehose to allow people to do realtime security analysis of events. There are definitely scanners that would have caught this attack immediately, they just need a way to be informed of updates.

simonw: PyPI does exactly that, and it's been very effective. Security partners can scan packages and use the invite-only API to report them: https://blog.pypi.org/posts/2024-03-06-malware-reporting-evo...

charcircuit: It is not effective if it just takes a simple base64 encode to bypass. If Claude is trivially able to find that it is malicious then Pypi is being negligent.

simonw: The package in question was live for 46 minutes. It generally takes longer than that for security partners to scan and flag packages.PyPI doesn't block package uploads awaiting security scanning - that would be a bad idea for a number of reasons, most notably (in my opinion) that it would be making promises that PyPI couldn't keep and lull people into a false sense of security.