Summarizer

CI/CD pipeline security

Discussion of running npm ci --ignore-scripts in pipelines, using ephemeral runners, and rotating secrets after potential compromise

← Back to Axios compromised on NPM – Malicious versions drop remote access trojan

To fortify CI/CD pipelines against supply chain vulnerabilities, practitioners are shifting toward rigorous attestation tools like Sigstore and the "nightmarish" but necessary practice of pinning external actions to specific commit hashes. While some advocate for "zero-install" strategies and vendoring dependencies to ensure resilience during registry outages or compromises, others prioritize limiting the "blast radius" by running installations in ephemeral, rootless containers devoid of sensitive environment secrets. There is also a growing debate over balancing automation with security, with many calling for "human-in-the-loop" publishing workflows that require hardware-backed MFA or multi-party signatures for high-impact package releases. Ultimately, the consensus highlights that because registries and scanners are not infallible, a multi-layered defense strategy incorporating both technical isolation and human oversight is critical for modern software delivery.

23 comments tagged with this topic

View on HN · Topics
This is where attestation/sigstore comes into play. Github has a first-party action for it and I wish more projects would use it. Regarding javascript specifically, I believe npm has builtin support for sigstore. * https://docs.github.com/en/actions/concepts/security/artifac... * https://www.sigstore.dev/ * https://github.com/actions/attest
View on HN · Topics
> But then your "lockfile" equivalent is just... a list of commit SHAs scattered across import statements in your source? Managing that across a real dependency tree becomes a nightmare. The irony is that this is actually the current best practice to defend against supply chain attacks in the github actions layer. Pin all actions versions to a hash. There's an entire secondary set of dev tools for converting GHA version numbers to hashes
View on HN · Topics
Why would it get pulled in over the weekend? What automatic deployments are you running if there also isn't a human working to get it out? Do you run automatic dependency updates over the weekend? Wouldn't you rather do that during fully-staffed hours?
View on HN · Topics
There are many options. Here's a post just briefly listing a few of the ones that would be handled by package managers and registries, but there are also many things that would be best done in CI pipelines as well. https://news.ycombinator.com/item?id=47586241
View on HN · Topics
Good luck with any `npm audit` in a pipeline. Sometimes you have to pull the latest release because the previous one had a critical vulnerability.
View on HN · Topics
How does this stance work with your CICD?
View on HN · Topics
I suppose you would have to commit your node_modules, or otherwise cache your setup so that all prerequesite modules are built and ready to install without running post-install scripts?
View on HN · Topics
There are ways to limit the blast radius, like running them in ephemeral rootless containers with only the project files mounted.
View on HN · Topics
Number 1 would only be a win for zero-installs if it happened that registry was up when you made the security hotfix, since you'd need to install the depdencency the first time to get it in VC, but then suddenly down when doing a deploy. Seems like a highly unlikely scenario to me. Also, cases where npm CVEs must be patched with such urgency or bad things will happen are luckily very rare, in my experience. Most npm CVEs are stuff like DDoS vulnerabilities, and you should have mitigations for those in place for at the infra-level anyway (e.g. request timeouts, rate limits, etc), or you are pretty much guaranteed to be cooked sooner or later anyway. The really dangerous stuff like arbitrary command execution from a library that takes end user input is much much more rare. The most recent big one I remember is React2shell. Number 2 hasn't been much of an issue for a long time. npm doesn't allow unpublishing package after 72 hours (apart from under certain rare conditions). Don't know about number 3. Would feel to me that if you have something running that can modify lockfile, they can probably also modify the chekced-in tars. I can see how zero-installs are useful under some specific constraints where you want to minimize dependencies to external services, e.g. when your CI runs under strict firewalls. But for most, nah, not worth it.
View on HN · Topics
> you'd need to install the depdencency the first time to get it in VC, but then suddenly down when doing a deploy. Which dependency? It sounds like you are assuming some specific scenario, whereas the fix can take many forms. In immediate term, the quickest step could be to simply disable some feature. A later step may be vendoring in a safe implementation. The registry doesn’t need to be actually down for you , either; the necessary condition is that your CI infrastructure can’t reach it. > cases where npm CVEs must be patched with such urgency or bad things will happen are luckily very rare, in my experience. Not sure what you mean by “npm CVEs”. The registry? The CLI tool? As I wrote, if you are running compromised software in production, you want to fix it ASAP. In first moments you may not even know whether bad things will happen or not, just that you are shipping malicious code to your users. Even if you are lucky enough to determine with 100% confidence (putting your job on the line) that the compromise is inconsequential, you don’t want to keep shipping that code for another hour because your install step fails due to a random CI infra hiccup making registry inaccessible (as happened in my experience at least half dozen times in years prior, though luckily not in a circumstance where someone attempted to push an urgent security fix). Now imagine it’s not a random hiccup but part of a coordinated targeted attack, and somehow it becomes something anticipated. > Number 2 hasn't been much of an issue for a long time. npm doesn't allow unpublishing package after 72 hours (apart from under certain rare conditions). Those rare conditions exist. Also, you are making it sound as if the registry is infallible, and no humans and/or LLMs there accept untrusted input from their environment. The key aspect of modern package managers, when used correctly, is that even when the registry is compromised you are fine as long as integrity check crypto holds up and you hold on to your pre-compromise dependency tree. The latter is not a technical problem but a human problem, because conditions can be engineered in which something may slip past your eyes. If this slip-up can be avoided at little to no cost—in fact, with benefits, since zero-installs shortens CI times, and therefore time-to-fix, due to dramatically shorter or fully eliminated install step—it should be a complete no-brainer. > Don't know about number 3. Would feel to me that if you have something running that can modify lockfile, they can probably also modify the chekced-in tars. As I wrote, I suspect it’d complicate such attacks or make them easier to spot, not make them impossible.
View on HN · Topics
> Publishers to official registries should be forced to use 2FA. I proposed this a decade ago for crates.io and people lost their minds, like I was suggesting we drag developers to a shed to be shot. How is this enforced when it's pushed via a pipeline?
View on HN · Topics
Totally agree. Also, considering how prevalent TPM/Secure Enclaves are on modern devices, I would guess most package maintainers already have hardware capable of generating/using signing keys that never leave hardware. I think it is mostly a devex/workflow question. Considering the recent ci/cd-pipeline compromises, I think it would make sense to make a two phase commit process required for popular packages. Build and upload to the registry from a pipeline, but require a signature from a hardware resident key before making the package available.
View on HN · Topics
The ones you hear about are caught quickly, I’m more worried about the non obvious ones. So far none of these have been as simple as changing a true to a false and bypassing all auth for all products or something, and would that be caught by an automated scanner?
View on HN · Topics
There are definitely levels to this. Yes I think it can be caught by automated scanners in theory. Either commit by commit scanning and reproducible builds or fuzzing and getting the behavioral differences between versions
View on HN · Topics
Sounds great until trivy images get compromised, like last week.
View on HN · Topics
Hence why you source data from multiple vendors I'd say. Rather than putting all eggs in one basket
View on HN · Topics
> You'd think this would be collectively a trivial and high priority task (to ensure that all publishes for packages over a certain download volume are going through a session that authenticated via MFA, for instance). Because all mainstream packages are published via CI/CD pipeline not by an MFA'd individual uploading a GZIP to npm.com
View on HN · Topics
Requiring a human-in-the-loop for final, non-prerelease publication doesn't seem like that onerous of a burden. Even if you're publishing multiple releases a day on the regular (in which case ... I have questions, but anyway) there are all sorts of automations that stay secure while reducing the burden of having to manually download an artifact from CI, enter MFA, and upload it by hand.
View on HN · Topics
You can still have a step that requires a certain user/group to sign off, and you can still enforce that those users have MFA set up. Almost any serious shop that expects to pass audits already does this in some form or fashion before pushing code to prod.
View on HN · Topics
Correct, for CI/CD systems that want to push releases.
View on HN · Topics
Just sanity checking - if I only ever install axios in a container that has no secrets mounted in to its env, is there any real way I can get pwned by this kind of thing?
View on HN · Topics
Where would you suggest putting the sensitive credentials?
View on HN · Topics
Not in .env files next to your code that is exposed to supply chain risks.