Shai-Hulud Returns: Over 300 NPM Packages Infected
helixguard.ai833 points by mrdosija 14 hours ago
833 points by mrdosija 14 hours ago
https://www.aikido.dev/blog/shai-hulud-strikes-again-hitting-zapier-ensdomains
ProTip: use PNPM, not NPM.
PNPM 10.x shutdown a lot of these attack vectors. 1. Does not default to running post-install scripts (must manually approve each) 2. Let's you set a min age for new releases before `pnpm install` will pull them in - e.g. 4 days - so publishers have time to cleanup. NPM is too insecure for production CLI usage. And of course make a very limited scope publisher key, bind it to specific packages (e.g. workflow A can only publish pkg A), and IP bound it to your self hosted CI/CD runners. No one should have publish keys on their local, and even if they got the publish keys, they couldn't publish from local.
(Granted, GHA fans can use OIDC Trusted Publishers as well, but tokens done well are just as secure) Npm is what happens when you let tech debt stack up for years too far. It took them five attempts to get lock files to actually behave the way lock files are supposed to behave (lockfile version 3, + at least 2 unversioned attempts before that). It’s clear from the structure and commit history they’ve been working their asses off to make it better, but when you’re standing at the bottom of a well of suck it takes that much work just to see daylight. The last time I chimed in on this I hypothesized that there must have been a change in management on the npm team but someone countered that several of the maintainers were the originals. So I’m not sure what sort of Come to Jesus they had to realize their giant pile of sins needed some redemption but they’re trying. There’s just too much stupid there to make it easy. I’m pretty sure it still cannot detect premature EOF during the file transfer. It keeps the incomplete file in the cache where the sha hash fails until you wipe your entire cache. Which means people with shit internet connections and large projects basically waste hours several times a week doing updates that fail. > I’m not sure what sort of Come to Jesus they had to realize their giant pile of sins needed some redemption but they’re trying. If they were trying, they'd stop doubling down on sunk costs and instead publicly concede that lock files and how npm-the-tool uses them to attempt to ensure the integrity of packages fetched from npm-the-registry is just a poor substitute for content-based addressing that ye olde DVCS would otherwise be doing when told to fetch designated shared objects from the code repo—to be accompanied by a formal deprecation of npm-install for use in build pipelines, i.e. all the associated user guides and documentation and everything else pushing it as best practice. npm-install has exactly one good use case: probing the registry to look up a package by name to be fetched by the author (not collaborators or people downstream who are repackaging e.g. for a particular distribution) at the time of development (i.e. neither run time nor build time but at the time that author is introducing the dependency into their codebase). Every aspect of version control should otherwise be left up to the underlying SCM/VCS. > cannot detect premature EOF during the file transfer. It keeps the incomplete file in the cache where the sha hash fails until you wipe your entire cache. I wonder what circumstances led to saying “this is okay we’ll ship it like that” I think we can blame the IO streaming API in NodeJS on this. It’s a callback and you just know you got another block. My guess is chunked mode and not checking whether the bytes expected and the bytes received matched. Not to diminish the facepalm but the blame can be at least partially shared. Our UI lead was getting the worst of this during Covid. I set up an nginx forward proxy mostly for him to tone this down a notch (fixed a separate issue but helped here a bit as well) so he could get work done on his shitty ISP. Ignorance. Most programmers in open source operate on the "works on my machine" but this stuff is basically solved. We have enough history with languages and distribution of packages, repositories, linux, public trust, signing, maintainers, etc. One key shift is there is no packager anymore. Its just - trust the publisher. Any language as big as Node should hire a handful of old unix wizards to teach them the way the truth and the life. Likely they wouldn’t listen. Modern languages and environments seem intent on reinventing bad solutions to solved problems. I get it if it’s a bunch of kids that have never seen anything better but there is no excuse these days not to have at least a passing knowledge of older systems if you’ve been around a while. there's certainly a piece of it. Also most seasoned people are not very interested in new languages and environments, and most languages are not 'spec built' by experts like Rob Pike building Go who explicitly set out to solve a lot of his problems, but are more naturally grown and born. > One key shift is there is no packager anymore. Its just - trust the publisher. Repositories like NPM's, and PyPI, contain many more packages than any Linux distro. And the Linux Foundation actually gets funded. > And of course make a very limited scope publisher key, bind it to specific packages (e.g. workflow A can only publish pkg A), and IP bound it to your self hosted CI/CD runners. No one should have publish keys on their local, and even if they got the publish keys, they couldn't publish from local. I've by now grown to like Hashicorp Vaults/OpenBao's dynamic secret management for this. It's a bit complicated to understand and get to work at first, but it's powerful: You mirror/model the lifetime of a secret user as a lease. For example, a nomad allocation/kubernetes pod gets a lease when it is started and the lease gets revoked immediately after it is stopped. We're kinda discussing if we could have this in CI as well - create a lease for a build, destroy the lease once the build is over. This also supports ttl, ttl-refreshes and enforced max-ttls for leases. With that in place, you can tie dynamically issued secrets to this lease and the secrets are revoked as soon as the lease is terminated or expires. This has confused developers with questionable practices a lot. You can print database credentials in your production job, run that into a local database client, but as soon as you deploy a new version, those secrets are deleted. It also gives you automated, forced database credential rotation for free through the max_ttl, including a full audit log of all credential accesses and refreshes. I know that would be a lot of infrastructure for a FOSS project by Bob from Novi Zagreb. But with some plugin-work, for a company, it should be possible to hide long-term access credentials in Vault and supply CI builds with dropped, enforced, short-lived tokens only. As much as I hate running after these attacks, they are spurring interesting security discussions at work, which can create actual security -- not just checkbox-theatre. I would love to use this (for homelab stuff currently) but I would love a way to have vault/openbao be fully configuration-as-code and version controlled, and only have the actual secret values (those that would not be dynamic) in persistent storage. Definitely curious if you've come up with a way to give each build a short lived vault approle somehow in any CI system. Or just 'npm ci' so you install exactly what's in your package-lock.json instead of the latest version bumps of those packages. This "automatic updating" is a big factor in why these attacks are working in the first place. Make package updating deliberate instead of instant or on an arbitrary lag. For Python ecosystem people: > Does not default to running post-install scripts (must manually approve each) To get equivalent protection, use `--only-binary=:all:` when running `pip install` (or `uv pip install`). This prevents installing source distributions entirely, using exclusively pre-built wheels. (Note that this may limit version ability or even make your installation impossible.) Python source packages are built by following instructions provided with the package (specifying a build system which may then in turn be configured in an idiosyncratic way; the default Setuptools is configured using a Python script). As such, they effectively run a post-install script. (For PAPER, long-term I intend to design a radically different UI, where you can choose a named "source" for each package or use the default; and sources are described in config files that explain the entire strategy for whether to use source packages, which indexes to check etc.) > Let's you set a min age for new releases before `pnpm install` will pull them in - e.g. 4 days - so publishers have time to cleanup. Pip does not support this; with uv, use `--exclude-newer`. This appears to require a timestamp; so if you always want things up to X days old you'll have to recalculate. There were some recent posts I saw about "dependency cooldowns", which seem to be what you're referring to in item 2. The idea really resonated with me. That said, I hard pin all our dependencies and get dependabot alerts and then look into updates manually. Not sure if I'm a rube or if that's good practice. That's good practice. God knows how many times I've been bitten by npm packages breaking on minor or even patch version changes, even when proudly proclaiming to use semver You shouldn't have any keys anywhere at all. Use OIDC https://docs.npmjs.com/trusted-publishers Unfortunately you need to `npm login` with username and password in order to publish the very first version of a package to set up OIDC. I'm struggling to understand why Trusted Publishers is any better. Let's say you have a limited life, package specific scoped, IP CIDR bound publishing key, running on a private GH workflow runner. That key only exists in a trusted clouds secret store (e.g. no one will have access it from their laptop). Now let's say you're a "trusted" publisher, running on a specific GitHub workflow, and GitHub Org, that has been configured with OIDC on the NPM side. By virtue of simply existing in that workflow, you're now a NPM publisher (run any publish commands you like). No need to have a secret passed into your workflow scope. If someone is taking over GitHub CI/CD workflows by running `npm i` at the start of their workflow, how does the "Trusted Publisher" find themselves any more secure than the secure, very limited scope token? A whole single supported CI partner outside their own corporate family. They really planned this out well. Both NPM and Yarn have a way to disable install scripts which everyone should do if at all possible. Good point, but until many popular packages stop requiring install.sh to operate, you'll still need to allowlist some of them. That is built into the PNPM tooling, luckily :) Reading through the post it looks like this infects via preinstall? > The new versions of these packages published to the NPM registry falsely purported to introduce the Bun runtime, adding the script preinstall: node setup_bun.js along with an obfuscated bun_environment.js file. Is there a way to set a minimum release age globally for my pnpm installation? I was only able to find a way to set it for each individual project. Did you try putting it in your global config file? Windows: ~/AppData/Local/pnpm/config/rc macOS: ~/Library/Preferences/pnpm/rc Linux: ~/.config/pnpm/rc I think it's a `pnpm-workspace.yaml` setting, for now, but PNPM has been pretty aggressive with expanding this feature set [1]. How does bun compare? Does it have similar features as well? yes bun does both of the things mentioned in the parent comment: > Unlike other npm clients, Bun does not execute arbitrary lifecycle scripts like postinstall for installed dependencies. Executing arbitrary scripts represents a potential security risk. https://bun.com/docs/pm/cli/install#lifecycle-scripts > To protect against supply chain attacks where malicious packages are quickly published, you can configure a minimum age requirement for npm packages. Package versions published more recently than the specified threshold (in seconds) will be filtered out during installation. What does it do with packages that download binaries for specific architecture in the post script? You don't need post-install scripts for this. Use optionalDependencies instead https://github.com/nrwl/nx/blob/master/packages/nx/package.j... Each of those deps contains a constraint installing only for the relevant platform. As far as I can understand from the documentation, that doesn't actually specify in that config that one of them is required, does it? That is, if they _all_ fail to install as far as the system is concerned there's nothing wrong? There will be runtime errors of course, but that's sort of disappointing… That’s cool, now I wish all libraries that need binaries would opt to use that instead of post script Do keep in mind that the binaries are still binaries. Even if your installation process doesn't run any untrusted code from the package, you can't audit the binaries like you might the .js files prior to first run. ProTip: `use bun` Funny that this is getting downvoted, but it installs dependencies super fast, and has the same approval feature as pnmp, all in a simple binary. This is like saying "use MacOS and you won't get viruses" in the 2000s Bun disables post-install scripts by default and one can explicitly opt-in to trusting dependencies in the package.json file. One can also delay installing updated dependencies through keys like `minimumReleaseAge`. Bun is a drop-in replacement for the npm CLI and, unlike pnpm, has goals beyond performance and storage efficiency. Not sure what your analogy is trying to imply. The suggestion was to use pnpm, and I'm suggesting something I prefer more than pnpm. > NPM is too insecure for production CLI usage. NPM was never "too insecure" and remains not "too insecure" today. This is not an issue with npm, JavaScript, NodeJS, the NodeJS foundation or anything else but the consumer of these libraries pulling in code from 3rd parties and pushing it to production environments without a single review. How this still fly today, and have been since the inception of public "easy to publish" repositories remains a mystery to me even today. If you're maintaining a platform like Zapier, which gets hacked because none of your software engineers actually review the code that ends up in your production environment (yes, that includes 3rd party dependencies, no matter where they come from), I'm not sure you even have any business writing software. The internet been a hostile place for so long, that most of us "web masters" are used to it today. Yet it seems developers of all ages fall into the "what's the worst that can happen?" trap when pulling in either one dependency with 10K LoC without any review, or 1000s of dependencies with 10 lines each. Until you fix your processes and workflows, this will continue to happen, even if you use pnpm. You NEED to be responsible for the code you ship, regardless of who wrote it. They didn't deploy the code. That's not how this exploit works. They _downloaded_ the code to their machine. And npm's behavior is to implicitly run arbitrary code as part of the download - including, in this case, a script to harvest credentials and propagate the worm. That part has everything to do with npm behavior and nothing to do with how much anybody reviewed 3P deps. For all we know they downloaded the new version of the affected package to review it! If people stop running install scripts, isn't Shai-Hulud 3: Electric Boogaloo just going to be designed to run its obfuscated malware at runtime rather than install time? Who manually reviews new versions of their project dependencies after installing them but before running them? GP is correct. This is a workflow issue. Without a review process for dependencies, literally every package manager I know of is vulnerable to this. (Yes, even Maven.) > If people stop running install scripts, isn't Shai-Hulud 3: Electric Boogaloo just going to be designed to run its obfuscated malware at runtime rather than install time? Many such forms of malware have already been published and detected. > Who manually reviews new versions of their project dependencies after installing them but before running them? One person putting in this effort can protect everyone thereafter. The PyPI website has a "Report project as malware" button on each project page for this purpose. But yes, this is the world we live in. Without this particular form of insecurity, there is no "ecosystem" at all. wait, I short-circuited here. wasn't the very concept of "libraries" created to *not* have to think about what exactly the code does? imagine reviewing every React update. yes, some do that (Obsidian claims to review every dependency, whether new or an update), but that's due to flaws of the ecosystem. take a look at Maven Central. it's harder to get into, but that's the price of security. you have to verify the namespace so that no one will publish under e.g. `io.gitlab.bpavuk.` namespace unless they have access to the `bpavuk` GitLab group or user, or `org.jetbrains.` unless they prove the ownership of the jetbrains.com domain. Go is also nice in that regard - you are depending on Git repositories directly, so you have to hijack into the Git repo permissions and spoil the source code there. > wasn't the very concept of "libraries" created to not have to think about what exactly the code does? If you care about security, you only have to care once, during the audit. And you can find a pretty high percentage of malware in practice without actually having a detailed understanding of the non-malicious workings of the code. Libraries allow you to not think about what the code does at development time, which in general is much more significant than audit time. Also, importantly, they allow you not to have to design and write that part of the code. > Go is also nice in that regard - you are depending on Git repositories directly, so you have to hijack into the Git repo permissions and spoil the source code there. That in itself is scary because Git refs are mutable. Even with compromised credentials, no one can replace artifacts already deployed to Maven Central, because they simply don't allow it. There is nothing stopping someone from replacing a Git tag with one that points to compromised code. The surface area is smaller because Go does locking via go.sum, but I could certainly see a tired developer regenerating it over the most strenuous of on-screen objections from the go CLI. I don't know if it's a common or even a good practice, but I like to go mod vendor and add the result to my repo. Go also includes a database of known package hashes so altering git tag to point to another commit will be detected. Are GitHub creds any harder for malware to steal than NPM creds? I don't see how that helps at all. “Personally, I never wear a seatbelt because all drivers on the road should just follow the road rules instead and drive carefully.” I don’t control all the drivers on the road, and a company can’t magically turn all employees into perfect developers. Get off your high horse and accept practical solutions. > and a company can’t magically turn all employees into perfect developers Sure, agree, that's why professionals have processes and workflows, everyone working together to build the greatest stuff you can. But when not a single person in the entire company reviews the code that gets deployed and run by users, you have to start asking what kind of culture the company has, it's borderline irresponsible I'd say. The "use cooldown" [0] blog post looks particularly relevant today. I'd argue automated dependency updates pose a greater risk than one-day exploits, though I don't have data to back that up. That's harder to undo a compromised package already in thousands of lock files, than to manually patch a already exploited vulnerability in your dependencies. [0] https://blog.yossarian.net/2025/11/21/We-should-all-be-using... Why not take it further and not update dependencies at all until you need to because of some missing feature or systems compatibility you need? If it works it works. The arguments for doing frequent releases partially apply to upgrading dependencies. Upgrading gets harder the longer you put it off. It’s better to do it on a regular schedule, so there are fewer changes at once and it preserves knowledge about how to do it. A cooldown is a good idea, though. There's another variable, though, which is how valuable "engineering time now" is vs. "engineering time later." Certainly, having a regular/automated update schedule may take less clock time in total (due to preserved knowledge etc.), and incur less long-term risk, than deferring updates until a giant, risky multi-version multi-dependency bump months or years down the road. But if you have limited engineering resources (especially for a bootstrapped or cost-conscious company), or if the risks of outages now are much greater than the risks of outages later (say, once you're 5 years in and have much broader knowledge on your engineering team), then the calculus may very well shift towards freezing now, upgrading later. And in a world where supply chain attacks will get far more subtle than Shai-Hulud, especially with AI-generated payloads that can evolve as worms spread to avoid detection, and may not require build-time scripting but defer their behavior to when called by your code - macro-level slowness isn't necessarily a bad thing. (It should go without saying that if you choose to freeze things, you should subscribe to security notification services that can tell you when a security update does release for a core server-side library, particularly for things like SQL injection vulnerabilities, and that your team needs the discipline to prioritize these alerts.) > Why not take it further and not update dependencies at all until you need to because of some missing feature or systems compatibility you need? If it works it works. Indeed there are people doing that and communities with a consensus such approach makes sense, or at least is not frowned upon. (Hi, Gophers) There is a Goldilocks effect. Dependency just came out a few minutes ago? There is no time for the community to catch the vulnerability, no real coverage from dependency scans, and it's a risk. Dependency came out a few months ago? It likely has a large number of known vulns That is indeed what one should do IMO. We've known for a long time now in the ops world that keeping versions stable is a good way to reduce issues, and it seems to me that the same principle applies quite well to software dev. I've never found the "but then upgrading is more of a pain" argument to be persuasive, as it seems to be equally a pain to upgrade whether you do it once every six months or once every six years. The 'pain' comes from breaking changes, at worst if you delay you're going to ingest the same quantity of changes, and at best you might skip some short-lived ideas. This works until you consider regular security vulnerability patching (which we have compliance/contractual obligations for). This only makes sense for vulnerabilities that can actually be exploited in your particular use-case and configuration of the library. A lot of vulns might be just noise and not exploitable so no need to patch. Yes and no. Problem is code bases are continuously evolving. A safe decision now, might not be a safe decision in the future. It's very easy to accidentally introduce a new code path that does make you vulnerable. Because updates don't just include new features but also bug and security fixes. As always, it probably depends on the context how relevant this is to you. I agree that cooldown is a good idea though. > Because updates don't just include new features but also bug and security fixes. This practice needs to change, although it will be almost impossible to get a whole ecosystem to adopt. You shouldn’t have to take new features (and associated new problems) just to get bug fixes and security updates. They should be offered in parallel. We need to get comfortable again with parallel maintenance branches for each major feature branch, and comfortable with backporting fixes to older releases. I maintain both commercial and open source libs. This is a non starter in both cases. It would easily double if not triple the workload. For open source, well these are volunteer projects on my own time, you are always welcome to fork a given version and backport any fixes that land on main/master. For commercial libs, our users are not willing to pay extra for this service, so we don't provide it. They would rather stay on an old version and update the entire code base at given intervals. Even when we do release patch versions, there is surprisingly little uptake. Semver was invented to facilitate that. Only if everyone adhered to it. Semver doesn't help. The primary issue is effort. If it's an open source project with 1-2 devs, they probably won't be able to handle supporting multiple branches unless they're being paid to do this. > Semver was invented to facilitate that First time I've heard that. How does semver facilitate backporting? Of course it doesn't provide backports by itself, it's a versioning system. But version number changes with SemVer are meant to indicate whether an update includes new fearhews or not (minor bump means new features, patch bump means bugfixes only). Of course, the actual issue is that maintaining backports isn't free, so expecting it from random single-person projects is a little unrealistic. Bug fixes in new code often need to be rewritten to work on old code. I do maintain old release branches for some projects and backporting single patches can cause whole new bugs that were never present in the main branch quite easily. IMO for “boring software” you usually want to be on the oldest supported main/minor version, keeping an eye on the newest point version. That will have all the security patches. But you don't need to take every bug fix blindly. For any update: - it usually contains improvements to security - except when it quietly introduces security defects which are discovered months later, often in a major rev bump - but every once in a while it degrades security spectacularly and immediately, published as a minor rev CI fights this. But that’s peanuts compared to feature branches and nothing compared to lack of a monolith. We had so many distinct packages on my last project that I had to massively upgrade a tool a coworker started to track the dependency tree so people stopped being afraid of the release process. I could not think of any way to make lock files not be the absolute worst thing about our entire dev and release process, so the handful of deployables had a lockfile each that was only utilized to do hotfix releases without changing the dep tree out from underneath us. Artifactory helps only a little here. Just make sure to update when new CVEs are revealed. Also, some software are always buggy and every version is a mixed bag of new features, bugs and regressions. It could be due to the complexity of the problem the software is trying to solve, or because it's just not written well. Because AppSec requires us to adhere to strict vulnerability SLA guidelines and that's further reinforced by similar demands from our customers. Because if you're too far behind, when you "need" takes days instead of hours. But even then you are still depending on others to catch the bugs for you and it doesn't scale: if everybody did the cooldown thing you'd be right back where you started. I don't think that this Kantian argument is relevant in tech. We've had LTS versions of software for decades and it's not like every single person in the industry is just waiting for code to hit LTS before trying it. There are a lot of people and (mostly smaller) companies who pride themselves on being close to the "bleeding edge", where they're participating more fully in discovering issues and steering the direction. The assumption in the post is that scanners are effective at detecting attacks within the cooldown period, not that end-device exploitation is necessary for detection. (This may end up not being true, in which case a lot of people are paying security vendors a lot of money to essentially regurgitate vulnerability feeds at them.) To find a vulnerability, one does not necessarily deploy a vulnerable version to prod. It would be wise to run a separate CI job that tries to upgrade to the latest versions of everything, run tests, watch network traffic, and otherwise look for suspicions activity. This can be done relatively economically, and the responsibility could be reasonably distributed across the community of users. It does scale against this form of attack.
This attack propagates by injecting itself into the packages you host. If you pull only 7d after release you are infected 7d later. If your customers then also only pull 7d later they are pulling 14d after the attack has launched, giving defenders a much longer window by slowing down the propagation of the worm. That worried me too, a sort of inverse tragedy of the commons. I'll use a weeklong cooldown, _someone else_ will find the issue... Until no-one does, for a week. To stretch the original metaphor, instead of an overgrazed pasture, we grow a communally untended thicket which may or may not have snakes when we finally enter. That is statistically not possible, unless you are dealing with very small sample size. The "until no one does" is not something that can happen in something like npm ecosystem, or even among the specific user of "left-pad". For Python's uv, I think the closest thing to a cooldown is something like: Pretty easy to do using npm-check-update: https://www.npmjs.com/package/npm-check-updates#cooldown In one command: The docs list this caveat: > Note that previous stable versions will not be suggested. The package will be completely ignored if its latest published version is within the cooldown period. Seems like a big drawback to this approach. I could see it being a good feature. If there have been two versions published within the last week or two, then there are reasonable odds that the previous one had a bug. some lib literally publish a new package at every PR merged, so multiple times a day. I don't buy this line of reasoning. There are zero/one day vulnerabilities that will get extra time to spread. Also, if everyone switches to the same cooldown, wouldn't this just postpone the discovery of future Shai-Huluds? I guess the latter point depends on how are Shai-Huluds detected. If they are discovered by downstreams of libraries, or worse users, then it will do nothing. Your line of reasoning only makes sense if literally almost all developers in the world adopt cooldowns, and adopt the same cooldown. That would be a level of mass participation yet unseen by mankind (in anything, much less something as subjective as software development). I think we're fine. I don't think so. What fraction of developers would even notice the malware? Some malware seems to barely be noticed. There are companies like Helix Guard scanning registries. They advertise static analysis / LLM analysis, but honeypot instances can also install packages & detect certain files like cloud configs being accessed But relying on the goodwill of commercial sec vendors is it's own infrastructure risk. So don't rely on their goodwill? Instead, pay them, under a contract.. or do it yourself. You can also pay a commercial sec vendor if you don't want to rely on their goodwill. For zero/one days, the trick is that you'd pair dependency cooldowns with automatic scanning for vulnerable dependencies. And in the cases where you have vulnerable dependencies, you'd force update them before the cooldown period had expired, while leaving everything else you can in place. co-founder of PostHog here. We were a victim of this attack. We had a bunch of packages published a couple of hours ago. The main packages/versions affected were: - posthog-node 4.18.1, 5.13.3 and 5.11.3 - posthog-js 1.297.3 - posthog-react-native 4.11.1 - posthog-docusaurus 2.0.6 We've rotated keys and passwords, unpublished all affected packages and have pushed new versions, so make sure you're on the latest version of our SDKs. We're still figuring out how this key got compromised, and we'll follow up with a post-mortem. We'll update status.posthog.com with more updates as well. You're probably already planning this, but please setup an alarm to fire off if a new package release is published that is not correlated with a CI/CD run. Or require manual intervention to publish a new package. I'm not sure why we need to have a fully automated pipeline here to go from CI/CD to public package release. It seems like having some kind of manual user interaction to push a new version of a library would be a good thing. The basic issue with manual interaction is a question of authority: a pretty common problem for companies (and open source groups) is when $EARLY_EMPLOYEE/$EARLY_CONTRIBUTOR creates and owns the entire publishing process for a key package, and then leaves without performing a proper transfer of responsibility. This essentially locks the company/group out of its own work, and increases support load on community maintained indices to essentially adjudicate rightful ownership of the package name. (There are a variety of ways to solve this, but the one I like best is automated publishing a la Trusted Publishing with environment mediated manual signoffs. GitHub and other CI/CD providers enable this.) I get that it can be useful sometimes. But requiring physical MFA to make a package available to the general public seems like a no-brainer to me. Users who really want to could opt in to the bleeding edge. This is built in NPM. You can get an email on every pkg publishing. Sure, it might be a little bit of noise, but if you get a notice @ 3am of an unexpected publishing, you can jump on unpublishing it. Did the client side JS being infected produce any issues which would have affected end users? As in if a web owner were on an affected version and deployed during the window would the end user of their site have had any negative impact? No, just the host that was running the package (the exploit was pretty generic and not targeted at PostHog specifically). In fact, so far we think there were 0 production deployments of PostHog because the package was only live for a little bit. If we don't know how it got compromised, chances are this attack is still spreading? If anything people should use an older version of the packages. Your newest versions had just been compromised, why should anyone believe this time and next time it will be different?! The packages were published using a compromised key directly, not through our ci/cd. We rolled the key, and published a new clean version from our repo through our CI/CD: https://github.com/PostHog/posthog-js/actions/runs/196303581... Why do you keep using token auth? This is unacceptable negligence these days. NPM supports GitHub workflow OIDC and you can make that required, disabling all token access. OIDC is not a silver bullet either and has its own set of vectors to consider too. If it works for your org model then great, but it doesn't solve every common scenario. Trusted Publishing addresses the vector here, which is arbitrary persistence and delayed use of credentials by attackers. You're right that it's not a silver bullet (anything claiming to be one is almost certainly a financially induced lie), but it eliminates/foreshortens the attack staging window significantly. Glad you updated on this front-page post. Your Twitter post is buried on p3 for me right now. Good luck on the recovery and hopefully this helps someone. > so make sure you're on the latest version of our SDKs. Probably even safer to not have been on the latest version in the first place. Or safer again not to use software this vulnerable. As a user of Posthog, this statement is absurd:
> Or safer again not to use software this vulnerable. Nearly all software you use is susceptible to vulnerabilities, whether it's malicious or enterprise taking away your rights. It's in bad taste to make a comment about "not using software this vulnerable" when the issue was widespread in the ecosystem and the vendor is already being transparent about it. The alternative is you shame them into not sharing this information, and we're all worse for it. Popularity and vulnerability go hand in hand though. You could be pretty safe by only using packages with zero stars on GitHub, but would you be happy or productive? Serious question: should someone develop new technologies using Node any more? A short time ago, I started a frontend in Astro for a SaaS startup I'm building with a friend. Astro is beautiful. But it's build on Node. And every time I update the versions of my dependencies I feel terrified I am bringing something into my server I don't know about. I just keep reading more and more stories about dangerous npm packages, and get this sense that npm has absolutely no safety at all. It's not "node" or "Javascript" the problem, it's this convenient packaging model. This is gonna ruffle some feathers, but it's only a matter of time until it'll happen on the Rust ecosystem which loves to depend on a billion subpackages, and it won't be fault of the language itself. The more I think about it, the more I believe that C, C++ or Odin's decision not to have a convenient package manager that fosters a cambrian explosion of dependencies to be a very good idea security-wise. Ambivalent about Go: they have a semblance of packaging system, but nothing so reckless like allowing third-party tarballs uploaded in the cloud to effectively run code on the dev's machine. I've worried about this for a while with Rust packages. The total size of a "big" Rust project's dependency graph is pretty similar to a lot of JS projects. E.g. Tauri, last I checked, introduces about 600 dependencies just on its own. Like another commenter said, I do think it's partially just because dependency management is so easy in Rust compared to e.g. C or C++, but I also suspect that it has to do with the size of the standard library. Rust and JS are both famous for having minimal standard libraries, and what do you know, they tend to have crazy-deep dependency graphs. On the other hand, Python is famous for being "batteries included", and if you look at Python project dependency graphs, they're much less crazy than JS or Rust. E.g. even a higher-level framework like FastAPI, that itself depends on lower-level frameworks, has only a dozen or so dependencies. A Python app that I maintain for work, which has over 20 top-level dependencies, only expands to ~100 once those 20 are fully resolved. I really think a lot of it comes down to the standard library backstopping the most common things that everybody needs. So maybe it would improve the situation to just expand the standard library a bit? Maybe this would be hiding the problem more than solving it, since all that code would still have to be maintained and would still be vulnerable to getting pwned, but other languages manage somehow. It's already happening: https://cyberpress.org/malicious-rust-packages/ My personal experience (YMMV): Rust code takes 2x or 3x longer to write than what came before it (C in my case), but in the end you usually get something much more likely to work, so overall it's kind of a wash, and the product you get is better for customers - you basically front load the cost of development. This is terrible for people working in commercial projects that are obsessed with time to market. Rust developers on commercial projects are under incredible schedule pressure from day 0, where they are compared to expectations from their previous projects, and are strongly motivated to pull in anything and everything they can to save time, because re-rolling anything themselves is so damn expensive. In my experience Rust development is no slower than C development (in a different environment) or C++ development (in a comparable project) I think they were using "writing Rust" in the most strict sense: the part of the development cycle that involves typing the majority of the code, before you really start debugging in earnest and really make things work. But their point is that "developing Rust" (as in, the entire process) ends up being a similar total effort to C, only with more up front "writing" and less work on the debugging phase. Thank you for the clarification, that's exactly what I was trying to say :). Perhaps another way to phrase this: in Rust, you spend more time telling the compiler how your code is expected to work (making the borrow checker happy, adding sync traits on objects you "know" are thread safe because of how you use them or assurances the underlying hardware provides, etc etc etc). In return, the compiler does a lot of work to make sure the code will actually work how you think it's going to work. A good example is a simple producer-consumer problem. Make a ring buffer. Push the current sys clk tick count register to the ring buffer every time there's a rising edge interrupt on a GPIO (e.x. hook up a button or something to it). Poll the ring buffer in another thread and as soon as there is a timestamp in the buffer, pop it and log it to a UART. Compare writing this in C vs Rust. In the text book implementation of a producer-consumer ring buffer, you don't need locks. In C, this is a problem I'd expect a senior-level candidate to be able to knock out in a 60 minute interview. (aside: I'd never actually ask a question like this in an interview, because it heavily favors people who just happen to be familiar with the algorithm, which doesn't give me enough signal about the candidate. This is borderline like asking someone to implement a sort algorithm in an interview - it just tells you they know how to google or memorize things. /aside). (aside 2: If I did ask this, I'd be more interested how they design the thing - I'd be looking for questions like "how many events per second? How should I size the ring buffer? Wait, you want me to hook up an IRQ to a button? Is there a risk of interrupt storm from bounce or someone being malicious? Do you want me to add a cooldown timer between events? If we overflow the ring buffer, what should the behavior be? How fast can the UART go on this system - can it even keep up with the input?" - I'd be far more interested in that conversation than actually seeing them write code for this. /aside2). In Rust, it's a bit more tricky. You'll need to give the compiler hints (in the form of Sync traits that are no-ops) to tell it you know what you're doing is thread safe. It's not rocket science, but the syntax is kind of weird and it will take some putzing around or aid from your favorite AI to get it right. All of Rust ends up like this - you must be more verbose telling the compiler your intent. In exchange, it verifies the code matches your intent. So the up front cost is higher. I suspect a lot of people will just pull a ring buffer off crates.io instead of figuring out the right incantations to make the compiler happy. I don't spend significant time making the borrow checker happy, because I learned how to write C++ that works. ¯\_(ツ)_/¯ I'd like to lean into that "YMMV" in my post, I'm coming from low level C that interacts with hardware, can't really speak to higher-level C++. Some things in Rust just don't translate to the way you'd do them in C - e.x. using different types to say if a GPIO pin is input or output adds a ton of boiler plate, but lets the compiler assure you don't mistakenly try and use a pin configured as an input for output. In general, the whole zero sized types paradigm in Rust leads to way more lines of code to accomplish the same thing (it all ends up compiled out in the end though). For embedded, I'll stand by what I said: it takes longer to write idiomatic Rust but you are more likely to get functionally correct code in the end. I wouldn't call the Rust stdlib "small". "Limited" I could agree with. On the topics it does cover, Rust's stdlib offers a lot. At least on the same level as Python, at times surpassing it. But because the stdlib isn't versioned it stays away from everything that isn't considered "settled", especially in matters where the best interface isn't clear yet. So no http library, no date handling, no helpers for writing macros, etc. You can absolutely write pretty substantial zero-dependency rust if you stay away from the network and async Whether that's a good tradeoff is an open question. None of the options look really great Rand, uuid, and no built in logging implementation are three examples that require crates but probably shouldn’t. No built in logging seems pretty crazy. Is there a story behind that? Rust's standard library hasn't received any major additions since 1.0 in 2015, back when nobody was writing web services in Rust so no one needed logging. println!() exists but there are more fancy crates like http://lib.rs/tracing and https://lib.rs/crates/log > But because the stdlib isn't versioned I honestly feel like that's one of Rust's biggest failings. In my ideal world libstd would be versioned, and done in such a way that different dependencies could call different versions of libstd, and all (sound/secure) versions would always be provided. E.g. reserve the "std" module prefix (and "core", and "alloc"), have `cargo new` default to adding the current std version in `cargo.toml`, have the prelude import that current std version, and make the module name explicitly versioned a la `std1::fs::File`, `std2::fs::File`. Then you'd be able to type `use std1::fs::File` like normal, but if you wanted a different version you could explicitly qualify it or add a different `use` statement. And older libraries would be using older versions, so no conflicts. I'm afraid it won't work. The point of std lib is to be universal connection for all the libraries. But with versioned std I just can't see how can you have DateTime in std1, DateTime in std2 and use them interchangeably, for example being able to pass std2::DateTime to library depending on std1 etc. Maybe conversion methods, but it get really complicated really quickly Network without async works fine in std. However, rand, serde, and num_traits always seem to be present. Not sure why clap isn't std at this point. Clap went through some major redesigns with the 4.0 release just three years ago. That wouldn't have been possible if clap 2.0 or 3.0 had been added to the stdlib. It's almost a poster child for things where libraries where being outside the stdlib allows interface improvements (date/time handling would be the other obvious example). Rand has the issue of platform support for securely seeding a secure rng, and having just an unsecure rng might cause people to use it when they really shouldn't. And serde is near-universal but has some very vocal opponents because it's such a heavy library. I have however often wished that num_traits would be in the stdlib, it really feels like something that belongs in there. FWIW, there is an accepted proposal (https://github.com/rust-lang/libs-team/issues/394) to add random number generation to std, and adding traits like in `num-traits` is wanted, but blocked on inherent traits. > Not sure why clap isn't std at this point. The std has stability promises, so it's prudent to not add things prematurely. Go has the official "flag" package as part of the stdlib, and it's so absolutely terrible that everyone uses pflag, cobra, or urfave/cli instead. Go's stdlib is a wonderful example of why you shouldn't add things willy-nilly to the stdlib since it's full of weird warts and things you simply shouldn't use. > and it's so absolutely terrible that everyone uses pflag, ../ This is just social media speak for inconvenient in some cases. I have used flag package in lot of applications. It gets job done and I have had no problem with it. > since it's full of weird warts and things you simply shouldn't use. The only software that does not have problem is thats not written yet. This is the standard one should follow then. Go is also famous for encouraging a culture of keeping down dependency count while exposing a simple to use package manager and ecosystem. https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-... this article made the rounds here after the author pulled the thread on dependencies in Go > why clap isn't std at this point. Too big for many cases, there is also a lot of discussion around whether to use clap, or something smaller. Clap is enormous and seems way too clever for everything I do. Last I looked it added 10+ seconds to compile time and hundreds of kbs to binary size. Maybe something like ffmpeg requires that complexity, but if I am writing a CLI that takes three arguments, it is a heavy cost. > On the topics it does cover, Rust's stdlib offers a lot. At least on the same level as Python, at times surpassing it. Curious, do you have specific examples of that? > if you stay away from the network and async That's some "small print" right there. > Rust and JS are both famous for having minimal standard libraries I'm all in favor of embiggening the Rust stdlib, but Rust and JS aren't remotely in the same ballpark when it comes to stdlib size. Rust's stdlib is decidedly not minimal; it's narrow, but very deep for what it provides. C standard library is also very small. The issue is not the standard library. The issue is adding libraries for snippets of code, and in the name of convenience, let those libraries run code on the dev machine. The issue is that our machines run 1970s OSes with a very basic security model, and are themselves so complex that they’re likely loaded with local privilege escalation attack vectors. Doing dev in a VM can help, but isn’t totally foolproof. It’s a good security model because everyone has the decency to follow a pull model. Like “hey, I have this thing, you can get it if you’re interested”. You decide the amount of trust you give to someone. But NPM is more like “you’ve added me to your contact list, then it’s totally fine for me to enter your bedroom at night and wear your lingerie because we’re already BFF”. It’s “I’m doing whatever I want on your computer because I know best and you’re dumb” mentality that is very prevalent. It’s like how zed (the editor) wants to install node.js and whatever just because they want to enable LSP. The sensible approach would have been to have a default config that relies on $PATH to find the language server. > “you’ve added me to your contact list, then it’s totally fine for me to enter your bedroom at night and wear your lingerie because we’re already BFF” I don't know if everyone will appreciate this, but I am in stitches right now... lol This is a reason why so many enterprises use C#. Most of the time you just use Microsoft made libraries and rarely brings in 3rd party. Having worked on four different enterprise grade C# codebases, they most certainly have plenty of 3rd party dependencies. It would absolutely be the exception to not have 3rd party dependencies. Yes, but the 3rd party dependencies tend to be conveniences rather than foundational. Easier mapping, easier mocking, easier test assertions, so a more security minded company can very easily just disallow their use without major impact. If it's something foundational to your project then what you're doing is probably somewhat niche. Most of the time there's some dependency from Microsoft that's rarely worse enough to justify using the 3rd party one. Or purchase third party libraries.
This does two things - limits what you drag in and also if you drag it in you can sue someone for errors. This definitely not why enterprise "chooses" C# and neither of these were design decisions like implied. MS would have loved to have the explosive, viral ecosystem of Node earlier in .NET's life. Regardless a lot of companies using C# still use node-based solutions on the web so a insular development environment for one tier doesn't protect them. I am not so sure about that. .net core is the moment they opened up, making it cross platform, going against the grain of owning it as a platform. If they see a gap in .net, which is filled in by a third party, they would have no problem qualms about implementing their own solution in .net that meets their quality requirements. And to be fair, .net delivers on that.
This might anger some, but the philosophy is that it should be a batteries included one-stop shop, maybe driven by the culture of quite some ms shops that wouldn't eat anything unless ms feeds it them. This has a consequence that the third-party ecosystem is a lot smaller, but I doubt MS regrets that.
If you compare that to F#, things are quite different wrt filling in the gaps, as MS does not focus on F#. A lot of good stuff for F# comes from the community. They actually had a pretty active community on CodePlex - I used and contributed to many projects there... they killed that in ... checks the web... 2017, replaced with GitHub, and it just isn't the same... It won't, it's a culture issue Most rust programmers are mediocre at best and really need the memory safety training wheels that rust provides. Years of nodejs mindrot has somehow made pulling into random dependencies irregular release schedules to become the norm for these people. They'll just shrug it off come up with some "security initiative* and continue the madness I only personally know one Rust programmer (works in scientific HPC) and he’s fantastic, but in general I do get the sense that most Rust devs migrated from JS and are just now figuring out “omg strong typing and compiled code native to the client hardware is really nice!” and think it’s a ground breaking revelation. Saying this as someone who is cautiously optimistic about Rust for my own work. And yet of course the world and their spouse import requests to fetch a URL and view the body of the response. It would be lovely if Python shipped with even more things built in. I’d like cryptography, tabulate/rich, and some more featureful datetime bells and whistles a la arrow. And of course the reason why requests is so popular is that it does actually have a few more things and ergonomic improvements over the builtin HTTP machinery. Something like a Debian Project model would have been cool: third party projects get adopted into the main software product by a sworn-in project member who who acts as quality control / a release manager. Each piece of software stays up to date but also doesn’t just get its main branch upstreamed directly onto everyone’s laps without a second pair of eyes going over what changed. The downside is it slows everything down, but that’s a side-effect of, or rather a synonym for stability, which is the problem we have with npm. (This looks sort of like what HelixGuard do, in the original article, though I’ve not heard of them before today.) Requests is a great example of my point, actually. Creating a brand-new Python venv and running `uv add requests` tells me that a total of 5 packages were added. By contrast, creating a new Rust project and running `cargo add reqwest` (which is morally equivalent to Python's `requests`) results in adding 160 packages, literally 30x as many. I don't think languages should try to include _everything_ in their stdlib, and indeed trying to do so tends to result in a lot of legacy cruft clogging up the stdlib. But I think there's a sweet spot between having a _very narrow_ stdlib and having to depend on 160 different 3rd-party packages just to make a HTTP request, and having a stdlib with 10 different ways of doing everything because it took a bunch of tries to get it right. (cf. PHP and hacks like `mysql_real_escape_string`, for example.) Maybe Python also has a historical advantage here. Since the Internet was still pretty nascent when Python got its start, it wasn't the default solution any time you needed a bit of code to solve a well-known problem (I imagine, at least; I was barely alive at that point). So Python could afford to wait and see what would actually make good additions to the stdlib before implementing them. Compare to Rust which _immediately_ had to run gauntles like "what to do about async", with thousands of people clamoring for a solution _right now_ because they wanted to do async Rust. I can definitely sympathize with Rust's leadership wanted to do the absolute minimum required for async support while they waited for the paradigm to stabilize. And even so, they still get a lot of flak for the design being rushed, e.g. with `Pin`. So it's obviously a difficult balance to strike, and maybe the solution isn't as simple as "do more in the stdlib". But I'd be curious to see it tried, at least. IMHO, the ideal for package management in a programming language ecosystem might recognise multiple levels of “standardisation”. At the top, you have the true standard library for the language. This has very strong stability guarantees. Its purpose is twofold: to provide universal implementations of essentials and to define standard/baseline interfaces for common needs like abstract data types, relational databases, networking and filesystems to encourage compatibility and portability. Next, you have a tier of recognised but not yet fully standardised libraries. These might be contributed by third parties, but they have requirements for identifying maintainers, appropriate licensing and mandatory peer review of all contributions. They have a clear versioning policy and can make breaking changes in new major releases, but they also provide some stability guarantees along the lines of semver and older releases are normally available indefinitely. The purpose of this tier is to provide a wider range of functionality and/or alternative implementations, but in a relatively stable way and implementing standard interfaces where applicable to improve portability. Finally, you have the free-for-all, anyone-can-contribute tier. This should still have a sane security model where people can’t just upload malware scripts that run automatically just because someone installed a package. However, it comes with few guarantees about stability or compatibility, except that releases of published packages will be available indefinitely unless there’s a very good reason to pull them where you obviously wouldn’t want to use one anyway. A package you like might be written by a single contributor who no longer maintains it, but if someone does write something useful that simply doesn’t need any further maintenance once it’s finished and does its job, there is still a place to share it. Or maybe just get comfortable with adding versions and deprecation. eg optparse to argparse (though tbf, I would have just preferred it was optparse2). Or maybe the problem is excessive stability commitments. I think I prefer languages that realize things can improve and are willing to say if you want to run 10 year old code, use a 10 year old compiler/runtime. I think I prefer languages that realize things can improve and are willing to say if you want to run 10 year old code, use a 10 year old compiler/runtime. IMHO, the trouble with that stance is that it leaves no path to incrementally update a long-lived system to benefit from any of those improvements. Suppose we have an application that runs on 2025’s most popular platform and in ten years we’re porting it to whatever new platform is popular in 2035. Personally, I’d like to know that all the business logic and database queries and UI structure and whatever else we wrote that was working before will still be working on the new platform, to whatever extent that makes sense. I’d like to make only some reasonably necessary set of changes for things that are actually different between the two platforms. If we can’t do that, our only other option is a big rewrite. That is how you get a Python 2 to Python 3 situation. And that, in turn, is how you get a lot of systems stuck on the older version for years, despite all the advantages any later versions might offer. > leaves no path to incrementally update Sure it does? Or do I not understand? It pushes the work to the application, not the language. And it works fine on rails. If you want to stay on an EOL rails, you're on your own, or you can buy ongoing security backports from at least 2 (that I know of, maybe more) vendors. LTS lives roughly 2 years and then you either upgrade or deal with eol yourself. That's not an apple-to-apple comparison, since Rust is a low-level language, and also because `reqwest` builds on top of `tokio`, an async runtime, and `hyper`, which is also a HTTP server, not just a HTTP client. If you check `ureq`, a synchronous HTTP client, it only adds 43 packages. Still more, but much less. And in Go I can build a production-ready HTTPS (not just HTTP) server with just the standard library and a few lines of code. (0 packages). That Rust does not have standard implementations of commonly-used features (such as an async runtime) is problematic for supply chain security, since then everyone is pulling in dozens (or hundreds) of fragmented 3rd-party packages instead of working with a bulletproof standard library. And this is exactly why Go is winning: because it's actually rather easy to write "pure Go" utilities (no dependencies outside the standard library), which statically compile to boot (avoiding shared libraries). > (cf. PHP and hacks like `mysql_real_escape_string`, for example.) PHP is a fantastic resource to learn how to do proper backward compatibility and package management. By doing the exact opposite of whatever PHP does, mostly. It might solve the problem, in as much as the problem is that not only can it be done, but it’s profitable to do so. This is why there’s no Rust problem (yet). I agree partly. I love cargo and can’t understand why certain things like package namespaces and proof of ownership isn’t added at a minimum. I was mega annoyed when I had to move all our Java packages from jcenter, which was a mega easy setup and forget affair, to maven central. There I suddenly needed to register a group name (namespace mostly reverse domain) and proof that with a DNS entry. Then all packages have to be signed etc. In the end it was for this time way ahead. I know that these measures won’t help for all cases. But the fact that at least on npm it was possible that someone else grabs a package ID after an author pulled its packages is kind of alarming. Dependency confusion attacks are still possible on cargo because the whole - vs _ as delimiter wasn’t settled in the beginning.
But I don’t want to go away from package managers or easy to use/sharable packages either. > But the fact that at least on npm it was possible that someone else grabs a package ID after an author pulled its packages is kind of alarming. Since your comment starts with commentary on crates.io, I'll note that this has never been possible crates.io. > Dependency confusion attacks are still possible on cargo because the whole - vs _ as delimiter wasn’t settled in the beginning. I don't think this has ever been true. AFAIK crates.io has always prevented registering two different crates whose names differ only in the use of dashes vs underscores. > package namespaces See https://github.com/rust-lang/rust/issues/122349 > proof of ownership See https://github.com/rust-lang/rfcs/pull/3724 and https://blog.rust-lang.org/2025/07/11/crates-io-development-... You are right. I remembered it wrong. https://rust-lang.github.io/rfcs/0940-hyphens-considered-har... Was from 2015 and the other discussions I remember were around default style and that cargo already blocks a crate when normalized name is equal. The trusted publishing is rather new or? Awesome to see that they implemented it. Just saying that maven central required it already years ago. Maven Central does not currently support OIDC-based authentication (commonly called "Trusted Publishing"). Didn’t know this term. After reading I wonder why short lived tokens get this monocle. But yeah I prefer OIDC over token based access as well. Only small downside I see is the setup needed for a custom OIDC provider. Don’t know the right terms out of my head but we had quite the fun to register our internal Jenkins to become a create valid oidc tokens for AWS. GitHub and GitHub Actions come with batteries included. I mean the downside that a huge vendor can easily provide this and a custom rolled CI needs extra steps / infrastructure. I'm a huge Go proponent but I don't know if I can see much about Go's module system which would really prevent supply-chain attacks in practice. The Go maintainers point [1] at the strong dependency pinning approach, the sumdb system and the module proxy as mitigations, and yes, those are good. However, I can't see what those features do to defend against an attack vector that we have certainly seen elsewhere: project gets compromised, releases a malicious version, and then everyone picks it up when they next run `go get -u ./...` without doing any further checking. Which I would say is the workflow for a good chunk of actual users. The lack of package install hooks does feel somewhat effective, but what's really to stop an attacker putting their malicious code in `func init() {}`? Compromising a popular and important project in this way would likely be noticed pretty quickly. But compromising something widely-used but boring? I feel like attackers would get away with that for a period of time that could be weeks. This isn't really a criticism of Go so much as an observation that depending on random strangers for code (and code updates) is fundamentally risky. Anyone got any good strategies for enforcing dependency cooldown? A big thing is that Go does not install the latest version of transitive dependencies. Instead it uses Minimal version selection (MVS), see https://go.dev/ref/mod#minimal-version-selection. I highly recommend reading the article by Russ Cox mentioned in the ref. This greatly decreases your chances of being hit by malware released after a package is taken over. In Go, access to the os and exec require certain imports, imports that must occur at the beginning of the file, this helps when scanning for malicious code. Compare this JavaScript where one could require("child_process") or import() at any time. Personally, I started to vendor my dependencies using go mod vendor and diff after dependency updates. In the end, you are responsible for the effect of your dependencies. In Go you know exactly what code you’re building thanks to gosum, and it’s much easier to audit changed code after upgrading - just create vendor dirs before and after updating packages and diff them; send to AI for basic screening if the diff is >100k loc and/or review manually. My projects are massive codebases with 1000s of deps and >200MB stripped binaries of literally just code, and this is perfectly feasible. (And yes I do catch stuff occasionally, tho nothing actively adversarial so far) I don’t believe I can do the same with Rust. You absolutely can, both systems are practically identical in this respect. > In Go you know exactly what code you’re building thanks to gosum Cargo.lock > just create vendor dirs before and after updating packages and diff them [...] I don’t believe I can do the same with Rust. cargo vendor The Go standard library is a lot more comprehensive and usable than Node, so you need less dependencies to begin with. > However, I can't see what those features do to defend against an attack vector that we have certainly seen elsewhere: project gets compromised, releases a malicious version, and then everyone picks it up when they next run `go get -u ./...` without doing any further checking. Which I would say is the workflow for a good chunk of actual users. You can't, really, aside from full on code audits. By definition, if you trust a maintainer and they get compromised, you get compromised too. Requiring GPG signing of releases (even by just git commit signing) would help but that's more work for people to distribute their stuff, and inevitably someone will make insecure but convenient way to automate that away from the developer > It's not "node" or "Javascript" the problem, it's this convenient packaging model. That and the package runtime runs with all the same privileges and capabilities as the thing you're building, which is pretty insane when you think about it. Why should npm know anything outside of the project root even exists, or be given the full set of environment variables without so much as a deny list, let alone an allow list? Of course if such restrictions are available, why limit them to npm? The real problem is that the security model hasn't moved substantially since 1970. We already have all the tools to make things better, but they're still unportable and cumbersome to use, so hardly anything does. pnpm (maybe yarn too?) requires explicit allowlisting of build scripts, hopefully npm will do the same eventually > security model yep, some kind of seccomp or other kind of permission system for modules would help a lot. (eg. if the 3rd party library is parsing something and its API only requires a Buffer as input and returns some object then it could be marked "pure", if it supports logging then that could be also specified, and so on) For all the other things I like about yarn, it still executes build scripts willy-nilly, so I am looking at switching to pnpm. I'm sure my $work is going to love me changing up the build toolchain again... PHP's composer on the other hand requires an allowlist in the project's composer.json. I never would have thought PHP would be the one to be getting stuff like this right. Still, I think the "allow-scripts" section or whatever it's called should be named "allow-unrestricted-access-to-everything". Or maybe just stick "dangerously-" in front, I dunno, and drop it when the mechanism is capable of fine-grained privileges. Deno also requires allowlisting npm scripts. It also has a deeper permissions model in general. Historically, arguments of "it's popular so that's why it's attacked" have not held up. Notable among them was addressing Windows desktop security vulnerabilities. As Linux and Mac machines became more popular, not to mention Android, the security vulnerabilities in those burgeoning platforms never manifested to the extent that they were in Windows. Nor does cargo or pip seem to be infected with these problems to the extent that npm is. > Nor does cargo or pip seem to be infected with these problems to the extent that npm is. Easy reason. The target for malware injections is almost always cryptocurrency wallets and cloud credentials (again, mostly to mine cryptocurrencies). And the utter utter majority of stuff interacting with crypto and cloud, combined with a lot of inexperienced juniors who likely won't have the skill to spot they got compromised, is written in NodeJS. Compared to the JS ecosystem and number of users both Python and Rust are puny, also the the NPM ecosystem also allowed by default for a lot of post-install actions since they wanted to enable a smooth experience with compiling and installing native modules (Not entirely sure how Cargo and PIP handles native library dependencies). As for Windows vs the other OS's, yes even the Windows NT family grew out of DOS and Win9x and tried to maintain compatiblity for users over security up until it became untenable. So yes, the base _was_ bad when Windows was dominant but it's far less bad today (why people target high value targets via NPM,etc since it's an easier entry-point). Android/iOS is young enough that they did have plenty of hindsight when it comes to security and could make better decisions (Remember that MS tried to move to UWP/Appx distribution but the ecosystem was too reliant on newer features for it to displace the regular ecosystem). Remember that we've had plenty of annoyed discourse about "Apple locking down computers" here and on other tech forums when they've pushed notarization. I guess my point is that, people love to bash on MS but at the same time complain about how security is affecting their "freedoms" when it comes to other systems (and partly MS), MS is better at the basics today than they were 20-25 years ago and we should be happy about that. This comment seems to address users intentionally installing malware. I mean to address cracking, the situation where an attacker gains root or installs software that the user does not know about. Preventing the user from installing something that they want to install is another issue completely. I'm hesitant to call it exactly security, though I agree that it falls under the auspices of security. Cracking is a term related to removing copy-protections. Rooting or privilege escalation is better terms for what you're mentioning. As for "users intentionally installing malware", Windows in the early 00s had a bunch of fundamentally insecure deployment models like ActiveX controls and browsers (IE especially) were more or less swiss cheese in terms of security even outside the ActiveX controls. Visiting the wrong webpage was often enough to get crap on your computer. My view is that once you have bad native code running on your computer there's a large chance that it's game-over (the modern sandboxes like WASM were designed to enforce a probably safe subset where regular kernel mistakes are shielded by another layer of abstraction that needs to be broken). Even Linux has had privilege escalations every year as far as I know. Notarization/stores is just a way to try to keep check on what code runs on end-user computers that isn't sandboxed (and allow for revoking that code if found to be malicious), maybe Linux is slightly safer still but that's probably due to less older features in the Kernel, but Windows has for example recently gotten a rewritten font-parser in Rust (the previous font parser was a common exploitation point that was placed with a too high privilegie). You can have security without having a walled garden. By trusting the user with the key of their own property. You mean like the developers holding the npm-publishing keys that just allowed a worm to spread? No. By NPM not allowing any package to run code on the developer's machine. I can trust npm (the software), but not the library. It's a very weird choice to just allow any package to run post install script. Especially when there's little to none verification done on npmjs side. Developers can feel free to not secure their computer or sell their keys. But that not means npm should allow straight code push from their computers to everyone that has downloaded their library. Every time I look at a new project, my face falls when it's written in Rust. I simply don't trust a system that pulls in gigabytes of god-knows-what off the cloud, and compiles it on my box. It's a real barrier to entry, for me. When I download a C project, I know that it only depends on my system libraries - which I trust because I trust my distro. Rust seems to expect me to take a leap in the dark, trusting hundreds of packagers and their developers. That might be fine if you're already familiar with the Rust ecosystem, but for someone who just wants to try out a new program - it's intimidating. On Debian you can use the local registry for Rust which is backed by packages. Though I will say, even as someone who works at a company that sells Linux distributions (SUSE), while the fact we have an additional review step is nice, I think the actual auditing you get in practice is quite minimal. For instance, quite recently[1] the Debian package for a StarDict plugin was configured automatically upload all text selected in X11 to some Chinese servers if you installed it. This is the kind of thing you'd hope distro maintainers to catch. Though, having build scripts be executed in distribution infrastructure and shipped to everyone mitigates the risk of targeted and "dumb" attacks. C build scripts can attack your system just as easily as Rust or JavaScript ones can (in fact it's probably even easier -- look at how the xz backdoor took advantage of the inscrutability of autoconf). [1]: https://www.openwall.com/lists/oss-security/2025/08/04/1 You don't know that about a C project. And you still don't know what lurks in its 1000th reimplementation of http header parsing. I think this is right about Rust and Cargo, but I would say that Rust has a major advantage in that it implements frozen + offline mode really well (which if you use, obviously significantly decreases the risks). Any time I ever did the equivalent with NPM/node world it was basically unusable or completely impractical Pnpm (a very popular npm replacement) makes completely locked packages easy and natural and ultra fast: Benchmarks: pnpm is so laughably terrible compared to Cargo it's not even comparable in the same breath. Why specifically? Your comment isn't very informative. Anyhow, here a Claude.ai comparison: https://claude.ai/share/72d2c34c-2c86-44c4-99ec-2a638f10e3f0 Because it doesn't perform as advertised: wild amounts of inconsistencies in behavior (within and between versions), performance issues (pnpm exec adds 15s to all shebang'd execution time over npm/yarn/bun/etc.), etc. Version-to-version stability has been traditionally bad - it's half-baked software. Claude doesn't know this, of course, because it can only read superficial summaries posted on the internet and has zero real experience actually using this dumpster fire. There are ecosystems that have package managers but also well developed first party packages. In .NET you can cover a lot of use cases simply using Microsoft libraries and even a lot of OSS not directly a part of Microsoft org maintained by Microsoft employees. 2020 State of the Octoverse security report showed that .NET ecosystem has on average the lowest number of transitive dependencies. Big part of that is the breadth and depth of the BCL, standard libraries, and first party libraries. The .NET ecosystem has been moving towards a higher number of dependencies since the introduction of .NET Core. Though many of them are still maintained by Microsoft. The "SDK project model" did a lot to reduce that back down. They did break the BCL up into a lot of smaller packages to make .NET 4.x maintenance/compatibility easier, and if you are still supporting .NET 4.x (and/or .NET Standard), for whatever reason, your dependency list (esp. transitive dependencies) is huge, but if you are targeting .NET 5+ only that list shrinks back down and the BCL doesn't show up in your dependency lists again. Even some of the Microsoft.* namespaces have properly moved into the BCL SDKs and no longer show up in dependency lists, even though Microsoft.* namespaces originally meant non-BCL first-party. I have a similar opinion but I think Java's model with maven and friends hits the sweet spot: - Packages are always namespaced, so typosquating is harder
- Registries like Sonatype require you to validate your domain
- Versions are usually locked by default My professional life has been tied to JVM languages, though, so I might be a bit biased. I get that there are some issues with the model, especially when it comes to eviction, but it has been "good enough" for me. Curious on what other people think about it. Maven does not support "scripts" as NPM does, such as the pre-install script used for this exploit. With scripts enabled, the mere act of downloading a dependency requires a high degree of trust in it. Downloading a dependency also requires a high degree of trust in whatever transitive dependencies that a trusted dependency decides to pull in. Supply chain attacks are scary because you do everything "right", but the ecosystem still compromises you. But realistically, I think the sum total of compromises via package managers attacks is much smaller than the sum total of compromises caused by people rolling their own libraries in C and C++. It's hard to separate from C/C++'s lack of memory safety, which causes a lot of attacks, but the fact that code reuse is harder is a real source of vulnerabilities. Maybe if you're Firefox/Chromium, and you have a huge team and invest massive efforts to be safe, you're better off with the low-dependency model. But for the median project? Rolling your own is much more dangerous than NPM/Cargo. Agreed, rust's cargo model is basically the worst part of that ecosystem right now. I've had developers submit pretty simple cli tools with hundreds and hundreds of dependencies. I guess there wasn't any lessons learned from the state of NPM. Rust (and really, any but JS) ecosystem have a bit more "due dilligence" applied everywhere; I don't doubt someone will try to namesquat but chance of success are far smaller > The more I think about it, the more I believe that C, C++ or Odin's decision not to have a convenient package manager that fosters a cambrian explosion of dependencies to be a very good idea security-wise. There was no decision in case of C/C++; it was just not a thing languages had at the time so the language itself (especially C) isn't written in a way to accommodate it nicely > Ambivalent about Go: they have a semblance of packaging system, but nothing so reckless like allowing third-party tarballs uploaded in the cloud to effectively run code on the dev's machine. Any code you download and compile is running code on dev machine; and Go does have tools to do that in compile process too. I do however like the by default namespacing by domain, there is no central repository to compromise, and forks of any defunct libs are easier to manage. > Rust (and really, any but JS) ecosystem have a bit more "due dilligence" applied everywhere; I don't doubt someone will try to namesquat but chance of success are far smaller I really agree, and I feel like it's a culture difference. Javascript was (and remains) an appealing programming language for tinkerers and hobbyists, people who don't really have a lot of engineering experience. Node and npm rose to prominence as a wild west with lots of new developers unfamiliar with good practices, stuck with a programming environment that had few "batteries included," and at a time when supply chain attacks weren't yet on everybody's minds. The barriers to entry were low and, well, the ecosystem sort of reflected that. You can't wash that legacy away overnight. Rust in contrast attracts a different audience because of the language's own design objectives. Obviously none of this makes it immune, and you can YOLO install random dependencies in any programming language, but I don't think any language is ever going to suffer from this in quite the same way and to the same extent that JS has simply due to when and how the ecosystem evolved. And really, even JS today is not JS of yesteryear. Sure there are lots of bad actors and these bad NPM packages sneak in, but also... how widely are all of them used? The maturation of and standardization on certain "batteries included" frameworks rather than ad hoc piecing stuff together has reduced the liklihood of going astray. While I agree that dependency tree size can be sometimes a problem in Rust, I think it often gets overblown. Sure, having hundreds of dependencies in a "simple" project can be scary, but: 1) No one forces you to use dependencies with large number of transitive dependencies. For example, feel free to use `ureq` instead of `reqwest` pulling the async kitchen sink with it. If you see an unnecessary dependency, you could also ask maintainers to potentially remove it. 2) Are you sure that your project is as simple as you think? 3) What matters is not number of dependencies, but number of groups who maintain them. On the last point, if your dependency tree has 20 dependencies maintained by the Rust lang team (such as `serde` or `libc`), your supply chain risks are not multiplied by 20, they stay at one and almost the same as using just `std`. On your last note, I wish they would get on that signed crate subset. Having the same dependency tree as cargo, clippy, and rustc isn't increasing my risk. Rust has already had a supply chain attack propagating via build.rs some years ago. It was noticed quickly, so staying pinned to the oldest thing that worked and had no cve pop in cargo audit is a decent strategy. The remaining risk is that some more niche dependency you use is and always has been compromised. Is serde maintained by the Rust team? I thought it was basically a one-man show owned by dtolnay My feeling is that languages with other packing models are merely less convenient,
and there is no actual tangible difference security-wise. Just take C and replace "look for writable repositories". It just takes more work and is less uniform to say write a worm that looks for writable cmake/autoconf and replicate that way. What would actually stop this is writing compilers and build systems in a way that isolates builds from one another. It's kind of stupid that all a compiler really needs is an input file, a list of dependencies, and an output file. Yet they all make it easy to root around, replicate and exfiltrate. It can be both convenient and not suffer from these style of attacks. Not really. cmake and automake are for compiling the library, not for the downloading it. The gap between the two is what's get erased from npm. And it made worse because of the auto update set by default when `npm install` is run. Not having a convenient package manager doesn't mean you don't need the functionality that's otherwise offered by third-party packages, it just means that you either need other means to obtain those third-party packages (usually reducing the visibility this dependency!) or implement them yourself (sometimes this is good, but sometimes this can also be very bad for security. Your DYI code won't get as many eyes and audits as the popular third party package!). Must read: https://wiki.alopex.li/LetsBeRealAboutDependencies I don’t get this I installed the package, obviously I intend to run it. How does getting pwned once I run it manually differ from getting pwned once I install it? I’m still getting pwned NPM default installation method does not really lock down you dependencies. It allows for update when the patch number (semver) is increased. Which is why those malware bump it up. Anyone who then run `npm install` will get it and will run the code. Using C++ daily, whenever I do js/ts are some javascript variant, since I don't use it daily, and update becomes a very complex task. frameworks and deps change APIs very frequently. It's also very confusing (and I think those attack vectors benefit exactly from that), since you have a dependency but the dep itself dependent on another dep version. Building basic CapacitorJS / Svelte app as an example, results many deps. It might be a newbie question, but,
Is there any solution or workflow where you don't end up with this dependency hell? Don't use a framework? Loading a JS script on a page that says "when a update b" hasn't changed much in about 20 years. Maybe I'm being a bit trite but the world of JavaScript is not some mysterious place separate from all other web programming, you can make bad decisions on either side of the stack. These comments always read like devs suddenly realizing the world of user interactions is more complicated and has more edge cases than they think. There's no solution. The JS world is just nonstop build and dependency hell. Being incredibly strict with TS compiler and linter helps a bit. Don't worry about C or C++, we create the vulnerabilities ourselves ! I get the joke, but that makes me think. What is worse between writing potentially vulnerable code yourself and having too many dependencies. Finding vulnerabilities and writing exploits is costly, and hackers will most likely target popular libraries over your particular software, much higher impact, and it pays better. Dependencies also tend to do more than you need, increasing the attack surface. So your C code may be worse in theory, but it is a smaller, thus harder to hit target. It is probably an advantage against undiscriminating attacks like bots and a downside against targeted attacks by motivated groups. > The more I think about it, the more I believe that C, C++ or Odin's decision not to have a convenient package manager that fosters a cambrian explosion of dependencies to be a very good idea security-wise. The safest code is the code that is not run. There is no lack of attacks targeting C/C++ code, and odin is just a hobby language for now. Not knowing that much about apt, isn't _any_ package system vulnerable, and purely a question of what guards are in place and what rights are software given upon install? It's not the packaging tech. Apt will typically mean a Debian-based distro. That means the packages are chosen by the maintainers and updated only during specific time periods and tested before release. Even if the underlying software gets owned and replaced, the distro package is very unlikely to be affected. (Unless someone spent months building trust, like xz) But the basic takeover... no, it usually won't affect any Debian style distro package, due to the release process. Given the years (or decades) it takes updates to happen in Debian stable, it’s immune to supply chain attacks. You do get to enjoy vulnerabilities that have been out for years, though. > it’s immune to supply chain attacks Thats a strong statement that I can see aging very badly. Agreed with the first half, but giving up on convenient packaging isn't the answer. Things like cargo-vet help as does enforcing non-token auth, scanning and required cooldown periods. Indeed, Rust's supply chains story is an absolute horror, and there are countless articles explaining what should be done instead (e.g. https://kerkour.com/rust-stdx) TL;DR: ditch crates.io and copy Go with decentralized packages based directly on and an extended standard library. Centralized package managers only add a layer of obfuscation that attackers can use to their advantage. On the other hand, C / C++ style dependency management is even worse than Rust's... Both in terms of development velocity and dependencies that never get updated. > countless articles explaining what should be done instead (e.g. https://kerkour.com/rust-stdx) Don't make me tap the sign: https://news.ycombinator.com/item?id=41727085#41727410 > Centralized package managers only add a layer of obfuscation that attackers can use to their advantage. They add a layer of convenience. C/C++ are missing that convenience because they aren't as composable and have a long tail of pre-package manager projects. Java didn't start with packages, but today we have packages. Same with JS, etc. I believe you, in that package management with dependencies without security mitigation is both convenient and dangerous. And I certainly agree this could happen for other package managers as well. My real worry, for myself re the parent comment is, it's just a web frontend. There are a million other ways to develop it. Sober, cold risk assessment is: should we, or should we have, and should anyone else, choose something npm-based for new development? Ie not a question about potential risk for other technologies, but a question about risk and impact for this specific technology. Surely in this case the problem is a technical one, and with more work towards a better security model and practices we can have the best of both worlds, no? Just a last month someone was trying to figure the cargo tree on which Rust package got imported implicitly via which package. This will totally happen in rust as well as long as you use some kind of package manager. Go for zero or less decencies. It already did happen. It propogated via build.rs as well. But as I said elsewhere, ut doesn't help you to forgo dependencies part of rust tooling itself. less? Roll your own standard library - or go without one entirely `#![no_std]` It’ll probably happen eventually with Rust, but ecosystem volume and informal packaging processes / a low barrier to entry seem to be significant driver in the npm world. (These are arguably good things in other contexts.) Why the word "semblance" with regard to Go modules? Are you trying to say this system is lacking something? Node is the embodiment of move and break things. Probably will not build anything that should last more than a few months on node. > The more I think about it, the more I believe that C, C++ or Odin's decision not to have a convenient package manager that fosters a cambrian explosion of dependencies to be a very good idea security-wise. Ambivalent about Go: they have a semblance of packaging system, but nothing so reckless like allowing third-party tarballs uploaded in the cloud to effectively run code on the dev's machine. The alternative that C/C++/Java end up with is that each and every project brings in their own Util, StringUtil, Helper or whatever class that acts as a "de-facto" standard library. I personally had the misfortune of having to deal with MySQL [1], Commons [2], Spring [3] and indirectly also ATG's [4] variants. One particularly unpleasant project I came across utilized all four of them, on top of the project's own "Utils" class that got copy-and-paste'd from the last project and extended for this project's needs. And of course each of these Utils classes has their own semantics, their own methods, their own edge cases and, for the "organically grown" domestic class that barely had tests, bugs. So it's either a billion "small gear" packages with dependency hell and supply chain issues, or it's an amalgamation of many many different "big gear" libraries that make updating them truly a hell on its own. [1] https://jar-download.com/artifacts/mysql/mysql-connector-jav... [2] https://commons.apache.org/proper/commons-lang/apidocs/org/a... [3] https://docs.spring.io/spring-framework/docs/current/javadoc... [4] https://docs.oracle.com/cd/E55783_02/Platform.11-2/apidoc/at... That is true, but the hand-rolled StringUtil won't steal your credentials and infect your machine, which is the problem here. And what is wrong with writing your own util library that fits your use case anyway? In C/C++ world, if it takes less than a couple hours to write, you might as well do it yourself rather than introduce a new dependency. No one sane will add a third-party git submodule, wire it to the main Makefile, just to left-pad a string. > That is true, but the hand-rolled StringUtil won't steal your credentials and infect your machine, which is the problem here. Yeah, that's why I said that this is the other end of the pendulum. > In C/C++ world, if it takes less than a couple hours to write, you might as well do it yourself rather than introduce a new dependency. Oh I'm aware of that. My point still stands - that comes at a serious maintenance cost as well, and I'd also say a safety cost because you're probably not wrapping your homebrew StringUtils with a bunch of sanity checks and asserts, meaning there will be an opportunity for someone looking for a cheap source of exploits. Wait what? That’s just fearmongering, how hard is it to add a few methods that split a string or pad it? It’s not rocket science. > how hard is it to add a few methods that split a string or pad it? In full generality, pretty hard. If you're just dealing with ASCII or Latin-1, no problem. Then add basic Unicode. Then combining characters. Then emojis. It won't be trivial anymore. Full generality is not a practical target. You select your subset of the problem and you solve it. Supporting everything in a project is usually a fever dream. > how hard is it to add a few methods that split a string or pad it? Well, if you're in C/C++, you always risk dealing with null pointers, buffer overruns, or you end up with use-after-free issues. Particularly everything working with strings is nasty and error-prone if one does not take care of proper testing - which many "homegrown" libraries don't. And that's before taking the subtleties of character set encodings between platforms into account. Or locale. Or any other of the myriad ways that C/C++ and even Java offer you to shoot yourself in the foot with a shotgun. And no, hoping for the best and saying "my users won't ever use Unicode" or similar falls apart on the first person copying something from Outlook into a multi-line paste box. Or someone typing in their non-Latin name. Oh, and right-to-left languages, don't forget about these. What does "pad from left" even mean there? Is the intent of the user still "at the beginning of the string itself?" Or does the user rather want "pad at the beginning of the word/sentence", which in turn means padding at the end of the string? There's so much stuff that can go horribly horribly wrong when dealing with strings, and I've seen more than my fair share just reading e-mail templates from supposed "enterprise" software. maybe the solution is what linux & co used for many years: have a team of people who vet and package dependencies. do they follow the same process ? or is it harder to submit a package and vet it on rust/cargo ? > but it's only a matter of time until it'll happen on the Rust ecosystem Totally 100% agree, though tools like cargo tree make it more of a tractable problem, and running vendored dependencies is first class at least. The one I am genuinely most concerned of is Golang. The way Dependencies are handled leaves much to be desired, I'm really surprised that there haven't been issues honestly. > C/C++ .. a convenient package manager Every time I fire up "cmake" I chant a little spell that protects me from the goblins that live on the other side of FetchContent to promise to the Gods of the Repo that I will, eventually, review everything to make sure I'm not shipping poop nuggets .. just as soon as I get the build done, tested .. and shipped, of course .. but I never, ever do. In the early days the Node ecosystem adopted (from Unix) the notion that everything has to be its own micro package. Not only was there a failure to understand what it was actually talking about, but it was never a good fit for package management to begin with. I understand that there's been some course correction recently (zero dependency and minimal dependency libs), but there are still many devs who think that the only answer to their problem is another package, or that they have to split a perfectly fine package into five more. You don't find this pattern of behavior outside of Node. > In the early days the Node ecosystem adopted (from Unix) the notion that everything has to be its own micro package. The medium is the message. If a language creates a very convenient package manager that completely eliminates the friction of sharing code, practically any permutation of code will be shared as a library. As productivity is the most important metric for most companies, devs will prefer the conveniently-shared third-party library instead of implementing something from scratch. And this is the result. I don't believe you can have packaging convenience and avoiding dependency hell. You need some amount of friction. It’s not even the convenience. It’s about trust. Npm makes it so that as soon as you add something to the dependency list, you trust the third party so completely you’re willing to run their code on your system as soon as they push an update. It’s essentially remote execution a la carte. I hate to be the guy saying AI will solve it, but this is a case where AI can help. I think in the next couple of years we’ll see people writing small functions with Claude/codex/whatever instead of pulling in a dependency. We might or might not like the quality of software we see, but it will be more resistant to supply chain attacks. When there's a depedency, it's typically not for a small function. If you want to replace a full dependency package by your own generated code, you'll need to review hundreds of even thousands of line of code. Now will you trust that AI didn't include its own set of security issues and will you have the ability to review so much code? For sure. I don't think the software ecosystem has come to terms with how things are going to change. Libraries will be providing raw tools like - Sockets, Regex Engine, Cryptography, Syscalls, specific file format libraries LLMs will be building the next layer. I have build successful running projects now in Erlang, Scheme, Rust - I know the basic syntax of two of those but I couldn't write my deployed software in any of them in the couple of hours of prompting. The scheme it had to do a lot of code from first principles and warned me how laborious it would be - "I don't care, you are doing it." I have tools now I could not have imagined I could build in a reasonable time. I wonder what the actual result will be. LLMs can generate functions quickly, but they're also keen to include packages without asking. I've had to add a "don't add new dependencies unless explicitly asked" to a few project configs. I don’t think I’ll live long enough to trust AI coding assistants with something like schema validation, just to name one thing I use dependencies for. How is this going to solve the supply chain attack problem at all though? It just obfuscates things even more, because once an LLM gets "infected" with malicious code, it'll become much more difficult to trace where it came from. If anything, blind reliance on LLMs will make this problem much worse. An approach I learnt from a talk posted to HN (I forget the talk, not the lesson) is to not depend on the outside project for its code, just lift that code directly in to your project, but to rely on it for the tests, requiring/importing it etc when running your own tests. That protects you from a lot of things (this kind of attack was not mentioned, afaic recall) but doesn’t allow bugs found by the other project to be missed either. I've started to feel it is much more an npm problem than a node problem. One of the things I've started leaning on more is prioritizing packages from JSR [0]. JSR is a part of Deno's efforts, so is often easiest to use in Deno packages, but most of the things with high scores on JSR get cross-published to npm and the few that prefer JSR only there's an alright JSR bridge to npm. Of course using more JSR packages does start to add more reason to prefer Deno to Node. Also, there are still some packages that are deno.land/x/ only (sort of the first version of JSR, but no npm cross-compatibility) worth checking out. For instance, I've been impressed with Lume [1], a thoughtful SSG that's sort of the opposite of Astro in that it iterates at a slow, measured pace, and doesn't try to be a kitchen sink but more of workbench with a lot of tools easy to find. It's deno.land/x/ only for now for reasons I don't entirely agree with but I can't deny that JSR can be quite a step up in publishing complexity for not exactly obvious gain. [0] https://jsr.io/ The problem isn't specific to node. NPM is just the most popular repo so the most value for attacks. The same thing could happen on RubyGems, Cargo, or any of the other package managers. NPM has about 4 million packages, Maven Central has about 3 million packages. If this were true, wouldn't there have been at least one Maven attack by now, considering the number of NPM attacks that we've seen? Been a while since I looked into this, but afaik Maven Central is run by Sonatype, which happens to be one of the major players for systems related to Supply Chain Security. From what I remember (a few years old, things may have changed) they required devs to stage packages to a specific test env, packages were inspected not only for malware but also vulnerabilities before being released to the public. NPM on the other hand... Write a package -> publish. Npm might scan for malware, they might do a few additional checks, but at least back when I looked into it nothing happened proactively. There were. They're just not as popular here. For example https://www.sonatype.com/blog/malware-removed-from-maven-cen... Maven is also a bit more complex than npm and had an issue in the system itself https://arxiv.org/html/2407.18760v4 As of 2024, Maven had 1.5 trillion requests annually vs npm's 4.5 trillion - regardless of package count, 3x more downloads in total does make it a very big target (numbers from https://www.sonatype.com/state-of-the-software-supply-chain/...). No. Having many packages might not be the only reason to start an attack.
This post shows it is/was possible in the Maven ecosystem: https://blog.oversecured.com/Introducing-MavenGate-a-supply-... One speculation would be is that most Java apps in the wild use way older Java versions (say 17/11, while the latest will LTS is 21). Okay then, explain to me why this is only possible with NPM? Does it have a hidden "pwn" button that I don't know about? >Does it have a hidden "pwn" button that I don't know about? Perhaps its package owners do. The concern is not 'could' happen, but _does_ happen. I know this could occur in many places. But where it seems highly prevalent is NPM. And I am genuinely thinking to myself, is this making using npm a risk? Just use dependency cooldown. It will mitigate a lot of risk. If you started your Node project yesterday, wouldn't that mean you'd get the fix later? no, because if you used dependency cooldown you wouldn't be using the latest version when you start your project, you would be using the one that is <cooldown period> days/versions old edit: but if that's also compromised earlier... \o/ NPM is the largest possible target for such an attack. Attack an important package, and you can get into the Node and Electron ecosystem. That's a huge prize. Value is one thing but the average user (by virtue of being popular) will be just less clued in on any security practices that could mitigate the problem. I’m not a node/js apologist, but every time there is a vulnerability in NPM package, this opinion is voiced. But in reality it has nothing to do with node/js. It’s just because it’s the most used ecosystem. So I really don’t understand the argument of not using node. Just be mindful of your dependencies and avoid updating every day. It has everything to do with node/js. Because the community believes in tiny dependencies that must be updated as often as possible and the tooling reflects that belief. it's interesting that staying up to date with your dependencies is considered a vulnerability in Node Having a cooldown is different from never updating. I don’t think waiting a few days is a bad security practice in any environment, node or otherwise. People who live on the edge of updates always risk vulnerabilities and incompatibility issues. It’s not about node, but anything software related. We chose to write our platform for product security analytics (1) with PHP, primarily because it still allows us to create a platform without bringing in over 100 dependencies just to render one page. I know this is a controversial approach, but it still works well in our case. "require": {
"php": ">=8.0", Not sure what the language has anything to do with it, we've built JavaScript applications within pulling in 100s of NPM packages before NPM was a thing, people and organizations can still do so today, without having to switch language, if they don't want to. Does it require disciple and a project not run by developers who just learned program? You betcha. I might say that every interpreter has a different minimum dependency level just to create a simple application. If we're talking about Node.js, there's a long list of dependencies by default. So yes, in comparison, modern vanilla PHP with some level of developer discipline (as you mentioned) is actually quite suitable, but unfortunately not popular, for low-dependency development of web applications. The language and capabilities of the platform indeed have a lot of influence on how many packages the average project depends on. With Swift on iOS/macOS for instance it’s not strange at all for an app to have a dependency tree consisting of only 5-10 third party packages total, and with a little discipline one can often get that number down to <5. Why? Because between the language itself, UIKit/AppKit, and SwiftUI, nearly all needs are pretty well covered. I think it’s time to beef up both JavaScript itself as well as the platforms where it’s run (such as the browser and Node), so people don’t feel nearly as much of a need to pull in tons of dependencies. You can do that with node.js too. It’s the libraries themselves that tries to bring in the whole world. It’s a matter of culture. > If we're talking about Node.js, there's a long list of dependencies by default. But that's not true? I initialize a project locally, there is zero dependencies by default, and like I did five years ago, I can still build backend/frontend projects with minimal set of dependencies. What changed is what people are willing/OK with doing. Yes, it'll require more effort, obviously, but if you want things to be built properly, it usually takes more effort. Perhaps, the right wording here might be that Node.js encourages the use of npm packages even for simple tasks. I agree that in any case, it's the courage/discipline that comes before the language choice when creating low-dependency applications. Ah yes PHP, the language known for its strong security... Oh yes, let's remember PHP 4.3 and all the nostalgic baggage from that era. Modern PHP is leagues above Javascript How so? 7.0 added scalar type declarations and a mechanism for strong typing. PHP 8.0 added union types and mixed types. PHP enforces types at runtime, Javascript/Typescript do not. PHP typesystem is built into the language, with Js u either need jsdoc or Typescript both of which wont enforce runtime type checks, Typescript even adds a buildstep. php-fpm allows u to not care about concurrency too much because of an isolated process execution model, with js based apps you need to be extremely careful about concurrency because of how easy you can create and access global stuff.
PHP also added a lot of syntax sugar over the time especially with 8.5 my beloved pipe operator.
And the ecosystem is not as fragile as Javascripts. Node is fine, the issue lies in its package model and culture: * Many dependencies, so much you don't know (and stop caring) what is being used. * Automatic and regular updates, new patch versions for minor changes, and a generally accepted best practice of staying up to date on the latest versions of things, due to trauma from old security breaches or big migrations after not updating for a while. * No review, trust based self-publishing of packages and instant availability * untransparent pre/postinstall scripts The fix is both cultural and technological: * Stop releasing for every fart; once a week is enough, only exception being critical security reasons. * Stop updating immediately whenever there's an update; once a week is enough. * Review your updates * Pay for a package repository that actually reviews changes before making them widely available. Actually I think the organization between NPM should set that up, there's trillion dollar companies using the Node ecosystem who would be willing and able to pay for some security guarantees. Microsoft owns npmjs.com. They could pay for AI analysis of published version deltas, looking for backdoors and malware. Professionally I am a fulltime FE Dev using Typescript+React. My Backends for my side projects are all done in C#, even so I'd be fluent in node+typescript for that very reason. In a current side project, my backend only has 3 external package dependencies, 2 of which are SQLite+ORM related. The frontend for that sideproject has over 50 (React/Typescript/MaterialUI/NextJS/NX etc.) .NET being so batteries-included is one of its best features. And when vulnerabilities do creep in, it's nice to know that Microsoft will fix it rather than hoping a random open source project will. There's only two kind of technologies. The ones that most people use and some people complain about, and the ones that nobody uses and people keep advocating for. This a common refrain on HN, frequently used to dismiss what may be perfectly legitimate concerns. It also ignores the central question of whether NPM is more vulnerable to these attacks than other package managers, and should therefore be considered an unreasonable security risk. It's not just npm, you should also not trust pypi, rubygems, cargo and all the other programming language package managers. They are built for programmers, not users. They are designed to allow any random untrusted person to push packages with no oversight whatsoever. You just make an account and push stuff. I have no doubt you can even buy accounts if you're malicious enough. Users are much better served by the Linux distribution model which has proper maintainers. They take responsibility for the packages they maintain. They go so far as to meet each other in person so they can establish decentralized root of trust via PGP. Working with the distributions is hard though. Forming relationships with people. Participating in a community. Establishing trust. Working together. Following packaging rules. Integrating with a greater dynamic ecosystem instead of shipping everything as a bloated container whose only purpose is to statically link dynamic libraries. Developers don't want to do any of that. Too bad. They should have to. Because the npm clusterfuck is what you get when you start using software shipped by totally untrusted randoms nobody cares to know about much less verify. Using npm is equivalent to installing stuff from the Arch User Repository while deliberately ignoring all the warnings. Malware's been found there as well, to the surprise of absolutely no one. There are far too many languages and many packages for each of them for this (good) idea to be practicable. Node doesn't have any particular relation to NPM? You don't have to download 1000 other people's code. Writing your own code is a thing that you are legally allowed to do, even if you're writing in Javascript. Yes, and you can code in assembly as well if you want it. But: that's not how 99% of the people using node is using it so that it is theoretically possible to code up every last bit yourself is true but it does not contribute to the discussion at all. An eco-system, if it insists on slapping on a package manager (see also: Rust, Go) should always properly evaluate the resulting risks and put proper safeguards in place or you're going to end up with a massive supply chain headache. Writing code yourself so as not to cultivate 1000 dependencies you can't possibly ensure the security of is not the same as writing assembly. That you even reach for that comparison is indicative of the deep rot in Javascript culture. Writing your own code is perceived as a completely unreasonable thing to be doing to 99% of JS-devs and that's why the web performs like trash and has breaches every other day, but it's actually a very reasonable thing to be doing and people who write most any other language typically engage in the writing of own code on a daily basis. At any rate, JS the language itself is fine, Node is fine, and it is possible to adopt better practices without forsaking the language/ecosystem completely. > That you even reach for that comparison is indicative of the deep rot in Javascript culture. Sorry? No, I'm the guy that does write all of his code from scratch so you're entirely barking up the wrong tree here. I am just realistic in seeing that people are not going to write more code than they strictly speaking have to because that is the whole point of using Node in the first place. The Assembly language example is just to point out the fact that you could plug in at a lower level of abstraction but you are not going to because of convenience, and the people using Node.js see it no different. JS is a perfectly horrible little language that is now being pushed into domains where it has absolutely no business being used (I guess you would object to running energy infrastructure on Node.js and please don't say nobody would be stupid enough to do that). Node isn't fine it needs a serious reconsideration of the responsibilities of the eco-system maintainers. See also: Linux, the BSDs and other large projects for examples of how this can be done properly. I feel like there are merits to your argument but that you have a larger anti-JS bias that's leaking through. Not that there aren't problems with Node itself, but as many people have pointed out, there are plenty of organizations writing in Node that aren't pwn'd by these sorts of attacks because we don't blindly update deps. Perfect is the enemy of good; dependency cooldown etc is enough to mitigate the majority of these risks. > I feel like there are merits to your argument but that you have a larger anti-JS bias that's leaking through. Familiarity breeds contempt. The truth is typically somewhere in the middle. I feel you though. I'm that way with Ruby/Bundler. What's the problem? I think JS is great. It's simple, anybody can use it. TypeScript is excellent too. The structural type system is very convenient. It's not going to replace Rust in cases where performance is essential or where you want strict runtime type checking or whatever, but for general use and graphical applications JS seems like a great pick. I often hear people complain about JS, but really, how is it any worse than say Python? > I often hear people complain about JS, but really, how is it any worse than say Python? That's not the flex you think it is.
twistedpair - 7 hours ago
hinkley - 6 hours ago
cxr - 4 hours ago
ornornor - 4 hours ago
hinkley - 4 hours ago
cyanydeez - 4 hours ago
calvinmorrison - an hour ago
smackeyacky - an hour ago
calvinmorrison - an hour ago
zahlman - an hour ago
tetha - 7 hours ago
LelouBil - 4 hours ago
mikestorrent - 5 hours ago
benoau - 6 hours ago
zahlman - an hour ago
jjice - 5 hours ago
jaapz - 2 hours ago
madeofpalk - 6 hours ago
twistedpair - 5 hours ago
c-hendricks - 6 hours ago
blktiger - 7 hours ago
twistedpair - 5 hours ago
dschofie - 3 hours ago
tripplyons - 7 hours ago
Operyl - 6 hours ago
twistedpair - 5 hours ago
poetril - 4 hours ago
flanbiscuit - 3 hours ago
halflife - 6 hours ago
madeofpalk - 6 hours ago
mook - 5 hours ago
halflife - 6 hours ago
zahlman - an hour ago
latchkey - 7 hours ago
benjifri - 4 hours ago
koito17 - an hour ago
latchkey - 3 hours ago
embedding-shape - 7 hours ago
jkrems - 6 hours ago
Ajedi32 - 3 hours ago
zahlman - an hour ago
bpavuk - 6 hours ago
zahlman - 44 minutes ago
tadfisher - 6 hours ago
JodieBenitez - 4 hours ago
fpoling - 4 hours ago
Ajedi32 - 3 hours ago
jama211 - 7 hours ago
embedding-shape - 6 hours ago
darkamaul - 13 hours ago
plomme - 9 hours ago
skybrian - 9 hours ago
btown - 2 hours ago
kunley - 8 hours ago
jonfw - 8 hours ago
bigstrat2003 - 9 hours ago
benoau - 3 hours ago
SkyPuncher - 8 hours ago
Nextgrid - 5 hours ago
SkyPuncher - 5 hours ago
tim1994 - 9 hours ago
ryandrake - 8 hours ago
nickserv - 5 hours ago
nine_k - 8 hours ago
scheme271 - 4 hours ago
ghurtado - 7 hours ago
cyphar - 6 hours ago
theptip - 9 hours ago
shermantanktop - 9 hours ago
hinkley - 6 hours ago
yupyupyups - 8 hours ago
Sparkle-san - 6 hours ago
parliament32 - 6 hours ago
jacquesm - 13 hours ago
falcor84 - 9 hours ago
woodruffw - 12 hours ago
nine_k - 8 hours ago
bootsmann - 7 hours ago
vintagedave - 9 hours ago
Aperocky - 4 hours ago
collinmanderson - 5 hours ago
uv is considering a native relative date: uv lock --exclude-newer $(date --iso -d "24 hours ago")
Sammi - 12 hours ago
npx npm-check-updates -c 7
tragiclos - 10 hours ago
nfriedly - 9 hours ago
hokkos - 7 hours ago
Ygg2 - 13 hours ago
wavemode - 8 hours ago
recursive - an hour ago
__s - 11 hours ago
Yokohiii - 10 hours ago
limagnolia - 8 hours ago
perlgeek - 7 hours ago
hyperpape - 9 hours ago
timgl - 13 hours ago
bilalq - 10 hours ago
mbreese - an hour ago
woodruffw - 27 minutes ago
magicalhippo - 44 minutes ago
twistedpair - 7 hours ago
silverlight - 8 hours ago
timgl - 7 hours ago
spiderfarmer - 13 hours ago
brabel - 13 hours ago
timgl - 13 hours ago
progbits - 13 hours ago
junon - 12 hours ago
woodruffw - 10 hours ago
_alternator_ - 13 hours ago
Y_Y - 13 hours ago
meesles - 10 hours ago
tclancy - 12 hours ago
vintagedave - 13 hours ago
sph - 13 hours ago
TheFlyingFish - 13 hours ago
QuiEgo - 9 hours ago
windward - 8 hours ago
ghurtado - 7 hours ago
QuiEgo - 6 hours ago
windward - 3 hours ago
QuiEgo - 2 hours ago
wongarsu - 13 hours ago
WD-42 - 9 hours ago
robot-wrangler - 9 hours ago
sapiogram - 8 hours ago
iknowstuff - 7 hours ago
SAI_Peregrinus - 9 hours ago
Zettroke - 8 hours ago
galangalalgol - 11 hours ago
wongarsu - 11 hours ago
afdbcreid - 9 hours ago
TheDong - 10 hours ago
geodel - 8 hours ago
poly2it - 9 hours ago
iknowstuff - 7 hours ago
Ygg2 - 11 hours ago
0cf8612b2e1e - 6 hours ago
atherton94027 - 8 hours ago
ghurtado - 7 hours ago
kibwen - 13 hours ago
skydhash - 12 hours ago
api - 12 hours ago
skydhash - 11 hours ago
ghurtado - 7 hours ago
metaltyphoon - 11 hours ago
latentsea - 10 hours ago
HighGoldstein - 8 hours ago
pasc1878 - 11 hours ago
skeeter2020 - 11 hours ago
exceptione - 9 hours ago
jjkaczor - 9 hours ago
mx7zysuj4xew - 8 hours ago
capyba - an hour ago
gorgoiler - 11 hours ago
TheFlyingFish - 10 hours ago
Chris_Newton - 9 hours ago
x0x0 - 7 hours ago
Chris_Newton - 4 hours ago
x0x0 - an hour ago
afdbcreid - 9 hours ago
auxiliarymoose - 8 hours ago
parliament32 - 6 hours ago
ghurtado - 7 hours ago
moomin - 10 hours ago
larusso - 13 hours ago
kibwen - 13 hours ago
larusso - 10 hours ago
larusso - 10 hours ago
di - 9 hours ago
larusso - 8 hours ago
gnfargbl - 13 hours ago
MatthiasDev - 8 hours ago
devttyeu - 11 hours ago
cyphar - 6 hours ago
asmor - 12 hours ago
PunchyHamster - 10 hours ago
chuckadams - 10 hours ago
pas - 9 hours ago
chuckadams - 8 hours ago
WorldMaker - 7 hours ago
dotancohen - 13 hours ago
mschuster91 - 13 hours ago
whizzter - 10 hours ago
dotancohen - 9 hours ago
whizzter - 3 hours ago
skydhash - 10 hours ago
whizzter - 3 hours ago
skydhash - 2 hours ago
alextingle - 10 hours ago
cyphar - 10 hours ago
iknowstuff - 5 hours ago
dwroberts - 12 hours ago
bhouston - 11 hours ago
15155 - 10 hours ago
bhouston - 8 hours ago
15155 - 6 hours ago
rafaelmn - 13 hours ago
CharlieDigital - 12 hours ago
CodesInChaos - 12 hours ago
WorldMaker - 7 hours ago
JD557 - 11 hours ago
oftenwrong - 10 hours ago
15155 - 9 hours ago
hyperpape - 12 hours ago
parliament32 - 6 hours ago
PunchyHamster - 10 hours ago
JeremyNT - 10 hours ago
newpavlov - 12 hours ago
galangalalgol - 11 hours ago
assbuttbuttass - 10 hours ago
ratmice - 6 hours ago
skydhash - an hour ago
SkiFire13 - 6 hours ago
lijok - 3 hours ago
skydhash - 2 hours ago
rock_artist - 11 hours ago
threetonesun - 9 hours ago
smt88 - 11 hours ago
Davidbrcz - 12 hours ago
GuB-42 - 10 hours ago
poulpy123 - 10 hours ago
moritonal - 13 hours ago
viraptor - 13 hours ago
trollbridge - 12 hours ago
alt227 - 10 hours ago
progbits - 13 hours ago
randomint64 - 12 hours ago
Ygg2 - 11 hours ago
vintagedave - 13 hours ago
fouronnes3 - 13 hours ago
zenmac - 13 hours ago
galangalalgol - 11 hours ago
actionfromafar - 13 hours ago
officialchicken - 12 hours ago
woodruffw - 11 hours ago
kunley - 8 hours ago
vachina - 13 hours ago
mschuster91 - 13 hours ago
sph - 12 hours ago
mschuster91 - 12 hours ago
skydhash - 12 hours ago
inejge - 11 hours ago
skydhash - 10 hours ago
mschuster91 - 9 hours ago
riffraff - 9 hours ago
agumonkey - 10 hours ago
dijit - 9 hours ago
aa-jv - 10 hours ago
tjpnz - 13 hours ago
sph - 12 hours ago
skydhash - 12 hours ago
testdelacc1 - 13 hours ago
jeromegv - 10 hours ago
delaminator - 10 hours ago
viraptor - 12 hours ago
elondaits - 7 hours ago
short_sells_poo - 12 hours ago
brigandish - 12 hours ago
WorldMaker - 9 hours ago
Gigachad - 13 hours ago
gred - 13 hours ago
chha - 13 hours ago
viraptor - 13 hours ago
pimterry - 11 hours ago
master-lincoln - 13 hours ago
skwee357 - 13 hours ago
AndroTux - 13 hours ago
tonyedgecombe - 11 hours ago
vintagedave - 13 hours ago
cluckindan - 13 hours ago
yoavm - 13 hours ago
flexd - 13 hours ago
Ygg2 - 13 hours ago
PunchyHamster - 10 hours ago
skwee357 - 12 hours ago
Aperocky - 5 hours ago
shortrounddev2 - 12 hours ago
bichiliad - 12 hours ago
skwee357 - 12 hours ago
reconnecting - 13 hours ago
1. https://github.com/tirrenotechnologies/tirreno "ext-mbstring": "*",
"bcosca/fatfree-core": "3.9.1",
"phpmailer/phpmailer": "6.9.3",
"ruler/ruler": "0.4.0",
"matomo/device-detector": "6.4.7" }
embedding-shape - 12 hours ago
reconnecting - 12 hours ago
cosmic_cheese - 10 hours ago
skydhash - 7 hours ago
embedding-shape - 12 hours ago
reconnecting - 12 hours ago
Zagitta - 12 hours ago
reconnecting - 12 hours ago
zwnow - 12 hours ago
root_axis - 10 hours ago
zwnow - 10 hours ago
Cthulhu_ - 12 hours ago
dboreham - 10 hours ago
littlecranky67 - 10 hours ago
noveltyaccount - 10 hours ago
paradite - 13 hours ago
monooso - 12 hours ago
matheusmoreira - 12 hours ago
doug713705 - 12 hours ago
anonymous908213 - 13 hours ago
jacquesm - 13 hours ago
anonymous908213 - 13 hours ago
jacquesm - 13 hours ago
notpachet - 13 hours ago
jacquesm - 13 hours ago
notpachet - 12 hours ago
user34283 - 8 hours ago
jacquesm - 7 hours ago