What they do: emulate a named adversary end-to-end against participating products in a lab and publish per-technique results with evidence. The 2025 Enterprise round used Scattered Spider — the first scenario to test cloud infrastructure and adversary reconnaissance — and Mustang Panda. The 2026 round introduces a single unified score combining detection and protection quality, weighted by technique criticality and, for the first time, including false-positive performance.

The most under-reported fact here: Microsoft, Palo Alto Networks and SentinelOne all withdrew from the 2025 round, citing the resource intensity of participating. Three of the largest endpoint vendors are absent from the industry's flagship independent evaluation. The evaluation's coverage of the market is shrinking as its rigour increases — which is a legitimate and rarely-made point.

The evaluation people forget is AV-Comparatives' Endpoint Prevention & Response test: 50 targeted attack scenarios across three phases, scoring both active response (blocking) and passive response (detection only), with certification requiring 92% average across both. Ten products certified in 2025 — and two failed and are anonymised as “Vendor A” and “Vendor B”, which is exactly backwards from a buyer's point of view, since which products failed is the useful information. Ignore its “savings” model; multiplying detection rates by an industry-average breach cost is a marketing construct, not a measurement.

There is no credible published evidence that deploying scanning tools reduces real-world security incidents. What exists instead: proxy metrics presented as outcomes — mean time to remediate, findings closed, coverage percentage — none of which is an incident; vendor case studies, which are selection bias by construction; and one survey finding, cited constantly as though it were causal, that organisations with extensive automation identify and contain breaches about 80 days faster. That is self-reported and cross-sectional. It cannot separate “automation causes faster containment” from “organisations mature enough to deploy automation are mature enough to contain faster”. The confounding is total.

Why the evidence does not exist, structurally: you would need a randomised or quasi-experimental design across comparable organisations with a common incident definition and honest disclosure. Incidents are rare, heterogeneous, under-reported, and confounded by organisational maturity — the same property that makes an organisation buy good tools makes it better at everything else. Nobody will run the experiment, and no vendor wants the result.

What genuinely works, with independent evidence. The strongest result in the field is a 2026 study of agent frameworks filtering static-analysis false positives, published at a peer-reviewed venue. On a deliberately contamination-controlled post-cutoff dataset, the best configuration identified 95.5% of false positives at 95.5% precision, against 36.4% for vanilla prompting — meaning the agentic scaffold, not just the model, is doing the work. On real Java projects it reached 93.3%.

And the caveat the vendors do not disclose, in the authors' own words: aggressive false-positive reduction can come at the cost of suppressing true vulnerabilities. Gains are strongly model- and vulnerability-class-dependent, and limited or inconsistent for weaker models. So when a vendor claims AI cut your false positives by 90%, the question is what it did to the true positives. The research says the answer is not zero.

What is marketing. “Autonomous SOC” — no independent evaluation supports autonomous incident response, and the broader agent literature finds enterprise-IT task success below 50%, a roughly 37-point gap between benchmark scores and production performance, and public benchmarks that are saturating and gameable. Also: AI-generated detection rules, because plausible-looking rules with unmeasured false-positive characteristics are worse than no rules; and “AI-powered” applied to ML classifiers that have been in the product for a decade.

The other direction, and it is currently winning. A longitudinal study across more than 150 models, testing four vulnerability classes with no security-specific prompting, finds roughly 45% of generated samples insecure — effectively unchanged from 2025 to 2026. Model size makes no difference: performance clusters at the same level whether the model has 20 billion parameters or 400 billion. The only variable that moved the number was reasoning models, which reached 70–72%. Note the conflict of interest — the publisher sells AppSec tooling and measured with its own scanner — but the method is published, the design is longitudinal, and the direction is corroborated by independent academic work.

The synthesis that matters: AI is simultaneously the most credible new triage technology in application security and the fastest-growing source of vulnerabilities and leaked credentials — leaked AI-service keys grew 81% in a year, and eight of the ten fastest-growing secret detectors are AI services. If you argue that AI tooling is net-positive or net-negative without acknowledging both directions, you have missed the actual dynamic.