Tell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised

880 points by dot_treo a day ago

About an hour ago new versions have been deployed to PyPI.

I was just setting up a new project, and things behaved weirdly. My laptop ran out of RAM, it looked like a forkbomb was running.

I've investigated, and found that a base64 encoded blob has been added to proxy_server.py.

It writes and decodes another file which it then runs.

I'm in the process of reporting this upstream, but wanted to give everyone here a headsup.

It is also reported in this issue: https://github.com/BerriAI/litellm/issues/24512

detente18 - a day ago

LiteLLM maintainer here, this is still an evolving situation, but here's what we know so far:

1. Looks like this originated from the trivvy used in our ci/cd - https://github.com/search?q=repo%3ABerriAI%2Flitellm%20trivy... https://ramimac.me/trivy-teampcp/#phase-09

2. If you're on the proxy docker, you were not impacted. We pin our versions in the requirements.txt

3. The package is in quarantine on pypi - this blocks all downloads.

We are investigating the issue, and seeing how we can harden things. I'm sorry for this.

- Krrish

detente18 - a day ago

Update:
- Impacted versions (v1.82.7, v1.82.8) have been deleted from PyPI - All maintainer accounts have been changed - All keys for github, docker, circle ci, pip have been deleted
We are still scanning our project to see if there's any more gaps.
If you're a security expert and want to help, email me - krrish@berri.ai
- cosmicweather - a day ago
  
  > All maintainer accounts have been changed
  What about the compromised accounts(as in your main account)? Are they completely unrecoverable?
  - detente18 - 2 hours ago
    
    I deleted it, to be safe.
- MadsRC - a day ago
  
  Dropped you a mail from mads.havmand@nansen.ai
  - ting0 - a day ago
    
    [flagged]
    
    kvdveer - a day ago
    
    > If you're a security expert and want to help, email me ...
    And
    > Dropped you a mail from [email]
    I don't think there is any indication of a compromise, they are just offering help.
    
    ij23 - 21 hours ago
    
    Hi all, Ishaan from LiteLLM here (LiteLLM maintainer)
    The compromised PyPI packages were litellm==1.82.7 and litellm==1.82.8. Those packages have now been removed from PyPI. We have confirmed that the compromise originated from the Trivy dependency used in our CI/CD security scanning workflow. All maintainer accounts have been rotated. The new maintainer accounts are @krrish-berri-2 and @ishaan-berri. Customers running the official LiteLLM Proxy Docker image were not impacted. That deployment path pins dependencies in requirements.txt and does not rely on the compromised PyPI packages. We are pausing new LiteLLM releases until we complete a broader supply-chain review and confirm the release path is safe.
    From a customer exposure standpoint, the key distinction is deployment path. Customers running the standard LiteLLM Proxy Docker deployment path were not impacted by the compromised PyPI packages.
    The primary risk is to any environment that installed the LiteLLM Python package directly from PyPI during the affected window, particularly versions 1.82.7 or 1.82.8. Any customer with an internal workflow that performs a direct or unpinned pip install litellm should review that path immediately.
    We are actively investigating full scope and blast radius. Our immediate next steps include:
    reviewing all BerriAI repositories for impact, scanning CircleCI builds to understand blast radius and mitigate it, hardening release and publishing controls, including maintainership and credential governance, and strengthening our incident communication process for enterprise customers.
    We have also engaged Google’s Mandiant security team and are actively working with them on the investigation and remediation.
    
    MadsRC - 20 hours ago
    
    We were not. I reached out to the team at BerriAI to offer my assistance as a security professional, given that they requested help from security experts.
harekrishnarai - a day ago

> it seems your personal account is also compromised. I just checked for the github search here https://github.com/search?q=%22teampcp+owns%22
driftnode - 14 hours ago

the chain here is wild. trivy gets compromised, that gives access to your ci, ci has the pypi publish token, now 97 million monthly downloads are poisoned. was the pypi token scoped to publishing only or did it have broader access? because the github account takeover suggests something wider leaked than just the publish credential
- sobellian - 5 hours ago
  
  If the payload is a credential stealer then they can use that to escalate into basically anything right?
  - driftnode - 5 hours ago
    
    Yes and the scary part is you might never know the full extent. A credential stealer grabs whatever is in memory or env during the build, ships it out, and the attacker uses those creds weeks later from a completely different IP. The compromised package gets caught and reverted, everyone thinks the incident is over, meanwhile the stolen tokens are still valid. I wonder how many teams who installed 1.82.7 actually rotated all their CI secrets after this, not just uninstalled the bad version.
- kreelman - 9 hours ago
  
  I wonder if there are a few things here....
  It would be great if Linux was able to do simple chroot jails and run tests inside of them before releasing software. In this case, it looks like the whole build process would need to be done in the jail. Tools like lxroot might do enough of what chroot on BSD does.
  It seems like software tests need to have a class of test that checks whether any of the components of an application have been compromised in some way. This in itself may be somewhat complex...
  We are in a world where we can't assume secure operation of components anymore. This is kinda sad, but here we are....
  - driftnode - 5 hours ago
    
    The sad part is you're right that we can't assume secure operation of components anymore, but the tooling hasn't caught up to that reality. Chroot jails help with runtime isolation but the attack here happened at build time, the malicious code was already in the package before any test could run. And the supply chain is deep. Trivy gets compromised, which gives CI access, which gives PyPI access. Even if you jail your own builds you're trusting that every tool in your pipeline wasn't the entry point. 97 million monthly downloads means a lot of people's "secure" pipelines just ran attacker code with full access.
vintagedave - a day ago

This must be super stressful for you, but I do want to note your "I'm sorry for this." It's really human.
It is so much better than, you know... "We regret any inconvenience and remain committed to recognising the importance of maintaining trust with our valued community and following the duration of the ongoing transient issue we will continue to drive alignment on a comprehensive remediation framework going forward."
Kudos to you. Stressful times, but I hope it helps to know that people are reading this appreciating the response.
- nextos - 20 hours ago
  
  I think we really need to use sandboxes. Guix provides sandboxed environments by just flipping a switch. NixOS is in an ideal position to do the same, but for some reason they are regarded as "inconvenient".
  Personally, I am a heavy user of Firejail and bwrap. We need defense in depth. If someone in the supply chain gets compromised, damage should be limited. It's easy to patch the security model of Linux with userspaces, and even easier with eBPF, but the community is somehow stuck.
  - staticassertion - 18 hours ago
    
    What would be really helpful is if software sandboxed itself. It's very painful to sandbox software from the outside and it's radically less effective because your sandbox is always maximally permissive.
    But, sadly, there's no x-platform way to do this, and sandboxing APIs are incredibly bad still and often require privileges.
    > It's easy to patch the security model of Linux with userspaces, and even easier with eBPF, but the community is somehow stuck.
    Neither of these is easy tbh. Entering a Linux namespace requires root, so if you want your users to be safe then you have to first ask them to run your service as root. eBPF is a very hard boundary to maintain, requiring you to know every system call that your program can make - updates to libc, upgrades to any library, can break this.
    Sandboxing tooling is really bad.
    
    eichin - 16 hours ago
    
    If the whole point of sandboxing is to not trust the software, it doesn't make sense for the software to do the sandboxing. (At most it should have a standard way to suggest what access it needs, and then your outside tooling should work with what's reasonable and alert on what isn't.) The android-like approach of sandboxing literally everything works because you are forced to solve these problems generically and at scale - things like "run this as a distinct uid" are a lot less hassle if you're amortizing it across everything.
    (And no, most linux namespace stuff does not require root, the few things that do can be provided in more-controlled ways. For examples, look at podman, not docker.)
    
    staticassertion - 16 hours ago
    
    > If the whole point of sandboxing is to not trust the software, it doesn't make sense for the software to do the sandboxing.
    That's true, sort of. I mean, that isn't the whole point of sandboxing because the threat model for sandboxing is pretty broad. You could have a process sandbox just one library, or sandbox itself in case of a vulnerability, or it could have a separate policy / manifest the way browser extensions do (that prompts users if it broadens), etc. There's still benefit to isolating whole processes though in case the process is malicious.
    > (And no, most linux namespace stuff does not require root, the few things that do can be provided in more-controlled ways. For examples, look at podman, not docker.)
    The only linux namespace that doesn't require root is user namespace, which basically requires root in practice. https://www.man7.org/linux/man-pages/man2/clone.2.html
    Podman uses unprivileged user namespaces, which are disabled on the most popular distros because it's a big security hole.
    
    ashishb - 17 hours ago
    
    > It's very painful to sandbox software from the outside and it's radically less effective because your sandbox is always maximally permissive.
    Not really.
    Let's say I am running `~/src/project1 $ litellm`
    Why does this need access to anything outside of `~/src/project1`?
    Even if it does, you should expose exactly those particular directories (e.g. ~/.config) and nothing else.
    
    staticassertion - 16 hours ago
    
    How are you setting that sandbox up? I've laid out numerous constraints - x-platform support is non-existent for sandboxing, sandboxing requires privileges to perform, whole-program sandboxing is fundamentally weaker, maintenance of sandboxing is best done by developers, etc.
    > Even if it does, you should expose exactly those particular directories (e.g. ~/.config) and nothing else.
    Yes, but now you are in charge of knowing every potential file access, network access, or possibly even system call, for a program that you do not maintain.
    
    ashishb - 16 hours ago
    
    > Yes, but now you are in charge of knowing every potential file access, network access, or possibly even system call, for a program that you do not maintain.
    Not really. I try to capture the most common ones for caching [1], but if I miss it, then it is just inefficient, as it is equivalent to a cache miss.
    I'll emphasize again, "no linter/scanner/formatter (e.g., trivy) should need full disk access".
    1 - https://github.com/ashishb/amazing-sandbox/blob/fddf04a90408...
    
    staticassertion - 16 hours ago
    
    Okay, so you're using docker. Cool, that's one of the only x-plat ways to get any sandboxing. Docker itself is privileged and now any unsandboxed program on your computer can trivially escalate to root. It also doesn't limit nearly as much as a dev-built sandbox because it has to isolate the entire process.
    Have you solved for publishing? You'll need your token to enter the container or you'll need an authorizing proxy. Are cache volumes shared? In that case, every container is compromised if one is. All of these problems and many more go away if the project is built around them from the start.
    It's perfectly nice to wrap things up in docker but there's simply no argument here - developers can write sandboxes for their software more effectively because they can architect around the sandbox, you have to wrap the entire thing generically to support its maximum possible privileges.
    
    ashishb - 15 hours ago
    
    > Docker itself is privileged and now any unsandboxed program on your computer can trivially escalate to root.
    Inside the sandbox but not on my machine. Show me how it can access an unmounted directory.
    > Have you solved for publishing? You'll need your token to enter the container or you'll need an authorizing proxy.
    Amazing-sandbox does not solve for that. The current risk is contamination; if you are running `trivy`, it should not need access to tokens in a different env/directory.
    > All of these problems and many more go away if the project is built around them from the start.
    Please elaborate on your approach that will all me to run markdown/JS/Python/Go/Rust linters and security scanners. Remember that `trivy` which caused `litellm` compromise is a security scanner itself.
    > developers can write sandboxes for their software more effectively because they can architect around the sandbox,
    Yeah, let's ask 100+ linter providers to write sandboxes for you. I can't even get maintainers to respond to legitimate & trivial PRs many a time.