Summarizer

If the contract is well defined, the LLM can infer what it's purpose is, implementation, possibly even your secret sauce. There is no software moat anymore.

This is exactly what happened with Log4Shell. Day -X + 1: Engineer at Alibaba finds the vuln and tells Apache. Patch is pushed to git while new release is coordinated. Day -X: A black hat sees commits fixing the bug. Attacks start happening. Day 0: Memes start circulating in Minecraft communities of people crashing servers. Some logs are shared on Twitter, especially in China, of people getting pwned. Day 0 + ~4 hours: My friend DMs me a meme on Twitter. I look up to find the CVE. Doesn't exist. My friend and I reproduce the exploit and write up a blog post about it. (We name it Log4Shell to differentiate it from a different, older log4j RCE vuln) Day ~1: Media starts picking it up. Apache is forced to release patches faster in response. CVE is actually published to properly allow security scanners to identify it. Today: AI makes this happen faster and more consistently. Patches probably should be kept private until a coordinated disclosure happens post-testing and CVE being published? Hard to say what the right move is, but this is gonna be happening a lot over the next 1-3 years. Lots of companies are going to be getting cooked until AI helps us patch faster than attackers can exploit these fresh 0-days.

I’m with you until that last sentence, which I’ve been thinking about as “… until AI code testing, vulnerability scanning, and developer support tools help to limit the number of 0-days and vulnerabilities making it into production”. So prevention will be more important than ai-assisted rapid containment or patching, though both of those capabilities will be necessary as part of defense in depth. And some sort of AI-enabled security analysis across the organization’s architecture that is done as part of testing ahead of new software entering production to ID potential vulnerabilities caused by configuration changes or upgrades that modify how systems interact with each other. I’ve been trying to guess the timeframe for seeing improved secure development, but I’m hoping it’s a bit closer to 6 months - 1 year given the speed of AI adoption and AI progression. May be closer to 3 years as you stated. In the meantime, is there more to be done than this (not in order)? - Patch COTS software - re-evaluate the scoring for previous vulnerabilities - set up up containment measures capabilities for systems that can’t be patched / high risk vendors - use frontier model vuln scanning and patching for home grown systems that may have more 0-days than COTS depending on the organization’s capability - limit the number of vendors / simplifying the tech stack. I’d be happy to hear how others are thinking about this.

> With skill, and usually not consistently and systematically. How do you know? If the people who like to crow about vulnerabilities aren't doing it, it doesn't mean that the people who are actually in a position to exploit them systematically and effectively aren't doing it. Those embargoes have always been dangerous, because they create a false sense of security. But, as you point out... > With AI, anyone can do this to any software. Yep. Even if it hadn't been true before, it's clear that now you just have to assume that everybody relevant will immediately recognize the security impact of any patch that gets published. That includes both bugs fixed and bugs introduced. ... and as the AI gets better, you're going to have to assume that you don't even have to publish a patch. Or source code. Within way less time than it's going to take people to admit it and adjust, any vulnerability in any software available for inspection is going to be instant public knowledge. Or at least public among anybody who matters.

>any vulnerability in any software available for inspection is going to be instant public knowledge. Or at least public among anybody who matters. Shouldn't this naturally lead to a state where all (new) code is vulnerability-free? If AI vulnerability detection friction becomes low enough it'll become common/forced practice to pre-scan code.

They're saying to do that scan to every diff before release, to see if it finds anything.

I believe their point was that: "How likely is this diff a patch for an existing vulnerability?" Seems to be an easier question to answer than "Are there any new vulnerabilities introduced by this diff?" In other words identifying that a patch is for a vulnerability is typically easier than finding the vulnerability in the first place.

If the diff will just be fed to LLMs regardless then what is easier is probably a moot point.

The point is that even if all code commits are scanned as safe by ai, black hats can still analyse the commits and diffs to find vulnerabilites for people who havent patched yet. Scanning every commit doesnt automatically make everyone in the world patch immediately, vulns can still be found from commits and diffs and used against those who havent patched yet.

Look at GP to my comment again, the one I was clarifying: they're not talking about black hats or any other kind of hacker, they're talking about the original developers and preventing such vulnerabilities from existing in the first place.

Yes I am aware, however that still does not stop anybody examining your commits and diffs to find vulnerabilities. Do you assume ai will just stop at a certain level? Or is it possible that it will keep increasing in intelligence? If the latter, then isnt it possible that even if you are auto checking all your commits, next week a more advanced ai model might be released that finds vulns in your old commits, even though they were checked by (an inferior) ai? Blinding saying that auto checking commits will make you safe from ai based attacks and vulnerability free is just madness.

> it'll become common/forced practice to pre-scan code. You'd think. But then you'd think people would do a lot of other things too. I hope, I guess. The other danger is that "the cloud" may become even more overwhelmingly dominant. Which of course has its own large security costs.

> How do you know? We know because we could see the effects of the average rate of vulnerabilities discovery and exploitation, and it's definitely going up very fast. Until recently, vulnerabilities were relatively hard to find, and finding them was done by a very restricted group of people world-wide, which made them quite valuable. Not any more.

That's correlation, not causation. It could equally be argued that the AI slop that's being produced makes for a lot more vulnerabilities being shipped. The bigger target makes for the easier discovery.

But don't we know that some of the vulnerabilities being discovered predate ai coding?

Certainly, and some discoveries have been attributed to AI (I was reading that mozilla firefox were praising mythos recently) But that's not accounting for all of the discoveries, not at all. I've also seen the npm people talking about the surge in AI code overwhelming the ability to properly review what's being distributed, and a large number of vulnerabilities being attributed to that

> That's correlation, not causation. Pragmatically, correlation *is* evidence of causation in favour of the best explanation, until somebody finds a better explanation. > It could equally be argued that the AI slop that's being produced makes for a lot more vulnerabilities being shipped. This is also true, and does not exclude the other, because for the moment the vast majority of production software in the world (and therefore the bulk of enticing targets) was written before AI. If LLM software will become prevalent in commercial setups, then LLM-generated code will eventually become the majority of targets.

> You have moved from "We know" to "We have an educated guess" No. You kept blabbering about "science" when most uses of knowledge are not about science. The original topic was also definitely not "science": it was about having a reasonable opinion about whether, empirically, the rate of discovery of vulnerabilities is increasing or not.

Trying to reframe this as 'not science' after being caught on a logical fallacy doesn't change the record. You started with a definitive claim ('We know') to shut down a question. When challenged on the lack of causation, you pivoted to 'educated guesses.' My point remains: if we misattribute the cause of the rising vulnerability rate (discovery vs. creation), our 'educated guesses' will lead to solutions that address the symptoms while the underlying problem continues to fester. Calling precision 'blabbering' is exactly how we end up with the 'false sense of security' mentioned earlier. Exhibit A: ragall 2 hours ago | root | parent | prev | next [–] > How do you know? We know because we could see the effects of the average rate of vulnerabilities discovery and exploitation, and it's definitely going up very fast. Until recently, vulnerabilities were relatively hard to find, and finding them was done by a very restricted group of people world-wide, which made them quite valuable. Not any more. Exhibit B: ragall 2 hours ago | root | parent | next [–] Very often you only have limited time for investigation and you have to act now. Action is almost always based on educated guesses. reply

> people were already diffing kernel commits and figuring out which ones were security fixes With skill, and usually not consistently and systematically. With AI, anyone can do this to any software. I would like to see actual evidence of this, not.. vibes I mean, this reeks of "Anyone is a Principal developer now" when the truth is there is still work to do.

I haven't been keeping tabs for the entirety of Linux development, but has it ever happened before that someone dropped a working exploit from the mailing list before the patch even hit the kernel? I haven't seen this kind of thing and I get the impression, despite all the hype, that this will be a frequent phenomenon now thanks to LLMs.

> This feels more like an old problem getting reframed as an AI problem. Sure the mechanics haven't changed, but the scale and risk has. The number of attackers capable of taking advantage of the vulnerability has dramatically increased.

I'd say it's an old problem be exacerbated by AI.

We have a huge problem. The US is at war. Much of the world is at war at the cyber attack level right now. The US, the EU, most of the Middle East, Israel, Russia... Major services have been attacked and have gone down for days at a time - Ubuntu, Github, Let's Encrypt, Stryker. Entire hospital systems have had to partially shut down. Now, in the middle of this, AI has made attacks much faster to generate. Faster than the defensive side can respond. Zero-day attacks used to be rare. Now they're normal. It's going to get worse before it gets better. Maybe much worse.

> before it gets better How is it going to get better?

If we assume that there will be an AI that is perfect in terms of ability to find vulnerabilities, cheap to run and widely available to everyone, then anyone can run it on any piece of software before deploying it. All vulnerabilities get found before they can be exploited. One of the big challenges with cybersecurity is that attackers only need to find one exploit, while defenders need to stop everything. When you have a large surface area and limited resources, it's much easier to be the side that only has to succeed once. AI eliminates the limited resources problem.

I'd speculate that at this point Linux etc are probably having vulnerabilities discovered and patched faster than created.

It's not only Linux though and many projects don't have the funding to perpetually use something like Mythos.

Right now we are at a point in time when AI can find bugs for attackers and defenders, but defenders did not fix/find those bugs yet. In time most of the bugs AI can find will be fixed, and things will calm down. Some bugs will be left, but will be too complex to find and weaponise (or rarely). Alin short, attackers have advantage for a brief time now, but ultimately defenders will win. I guess this "fight" might be over before the end of the year.

> So many security fixes are coming out now that examining commits is much more attractive: the signal-to-noise ratio is higher Why? > Additionally, having AI evaluate each commit as it passes is increasingly cheap and effective This is the key. With AI, the “people won't notice, with so many changes going past” assumption fails.

AI will shorten update windows dramatically. 2026 is the worst year to be thinking about dependency cooldowns, we need to think about dependency warmups instead. Soon, there will be no such thing as a safe way to disclose a vulnerability in an open source project. Centralized SaaS will have a major security advantage here.

Realistically, if you are scanning each kernel commit to check if they might be patching a security issue, you are going to be asking an LLM "is this security related, if so vaguely how" with low effort and taking "maybe" as a yes before feeding it to a more expensive model. You aren't trying to establish a probability of an ultimately unknowable fact, there is ground truth that you can find by producing an exploit, so you are just trying to pre-filter before spending the money to find it.

> Luckily AI can speed up defenders as well as attackers here, allowing embargoes that would previously have been uselessly short. This is an important facet of the problem space: security risks turning into an arms race for who wants to spend more tokens.

One interesting thing is that this makes closed source code even greater asset for the defenders. Attacker cannot spend tokens for it, but defenders can spend tokens for hardening based on source code, while attacker is stuck with blackbox testing.

The bugs are bugs description reads pretty insane to me personally but I know linux world has many people valueing principle of it over practical matters. 90d seems long too though. Think ultimately the big AI houses will need to help the core internet infra guys. Running latest and greatest AI over stuff like nginx and friends makes sense for us all collectively I think

The crowdstrike situation wasn't due to fast rollouts, it was due to a total lack of testing. You can do fast rollouts, with testing, and a mandatory QA signoff. It's called 'continuous delivery' rather than 'continuous deployment'. I actually don't think more layers of security will fix this. It would be nice if our systems were more secure... but people are, if nothing else, lazy af. Even when adding security isn't a lot of work, people resist it if it "sounds complicated". So I think we're stuck with the status quo. But the big issue now isn't novel bug types, it's the speed in which they're found. Therefore we need to speed up our response.