242,072 malware flags on npm and PyPI, and what the registries did about them

Security feeds publish lists of malicious packages, and dependency scanners turn those lists into red alerts in your CI. How often is a flag right? Nobody publishes that number per feed, so we started measuring it, with the method written down and committed before we looked at a single result. We also ran four open-source scanners on known malware and on a week of new uploads, to see what a flag costs at a registry's real volume. For completeness of the analysis, we also measured what each feed's records show and how each feed answers the people who dispute them.

Our interest, first. One of the feeds below listed nine of our own PyPI packages as malware; it has since reviewed all nine, withdrawing one and reclassifying eight as vulnerabilities. That is why we looked. They are counted like every other row, each feed gets its numbers a week before this post, and their answers are part of it. The last section shows how every tool treated our own packages, labelled for what it is: one interested publisher's case, not a measure of accuracy.

What we measured

Two open sources, npm and PyPI only:

• OpenSSF malicious-packages: 234,710 records, each naming the tools that reported it. Nine tools have attributable data: Amazon Inspector, ReversingLabs, Checkmarx, OSSF Package Analysis, Google Open Source Security, GitHub's malware advisories, kam193, Oracle Macaron, and Vulnetix (7,362 malware records in its GCVE advisory database). Vulnetix points out that this database is not its malware feed: it went public on 18 October and was still being loaded in bulk when we took our snapshot on the 24th. Its malware feed proper is a set of STIX indicator bundles, which we did not measure here.
• For a random sample of 7,483 records (up to 400 per tool, ecosystem and kind), we asked npm and PyPI what they serve today. Metadata only: nothing was downloaded or run.

"Registry acted" means the package is gone, replaced by npm's 0.0.1-security placeholder, quarantined, or the flagged versions are no longer served. That is a registry agreeing, not proof: registries act on these same feeds' reports. And a package still up is not proof of a false positive: registries are slow and miss things. This stage gives no false-positive rate. The next one does.

Finding 1: most npm "malware" records are spam

63.2% of OpenSSF's npm records describe the tea.xyz token-farming spam that flooded npm in late 2025, almost all from one report by Amazon Inspector (140,729 records). Whether self-publishing reward spam is malware is a fair argument; Endor Labs called a similar set spam that steals nothing. We keep it apart. Registries acted on 56.0% of it (95% CI 51.1–60.8%). (A first count said 6.0%: npm also holds names as 0.0.1-security.N, and a held name can still list versions whose files are gone; our first check missed both. The same sample, checked again, is what this post reports.)

On PyPI the picture is cleaner: 90.2% of records describe malicious code, 9.8% describe security-research probes.

Finding 2: how often registries acted, per tool

For malicious-code flags 30 to 180 days old, registries acted on 87–100% of them for Amazon Inspector, GitHub, Google, kam193, OSSF Package Analysis and ReversingLabs. For Vulnetix's flags of the same age: 24.5% on npm and 11.0% on PyPI. (Intervals and sample sizes are in the full write-up.)

We can't tell from this stage why, and there are several honest explanations:

• The reporting path. OpenSSF records reach npm and PyPI directly; some tools are PyPI trusted reporters whose reports quarantine a project at once. A feed whose findings never reach the registries' queues gets few takedowns whatever its accuracy. Vulnetix told us its findings don't reach them: PyPI's security team has not answered its offer yet, and npm declined, since it runs its own scanning.
• A mix of kinds. Vulnetix records carry no text to sort spam out, and registries act on spam less often (56% against 92–100% for malicious code).
• Different findings. A feed can see malware the others miss.
• False positives.

Finding 3: how often feeds agree

Google's npm flags are also flagged by another tool 95.2% of the time; ReversingLabs' about half the time. Low agreement is not a defect on its own: Checkmarx is alone on 99.5% of its PyPI packages, and PyPI removed every one we sampled. Vulnetix is alone on 97.7% of its PyPI packages and 90.0% of its npm ones, and registries left most of them up.

Finding 4: one feed shows its work

Vulnetix is the only feed here that publishes the rule and the matched line behind its records (28 of the 30 we drew). That deserves credit: it is the only reason anyone can check its reasoning.

For completeness of the analysis, we also measured for every feed how transparent its records are and whether a verdict can be disputed, with the criteria fixed before we read the data (Finding 7).

A quarter of its matches are indicator-list hits; the rest are text rules such as /etc/passwd appearing in code (7.4%), curl … | sh (7.2%), python -c (5.8%), ~/.ssh (4.1%) and exec(compile(…)) (2.4%). These match anywhere in a package, tests and READMEs included. A USENIX Security 2026 paper found the same pattern in the open scanner GuardDog: most of its false positives were code-execution rules firing on legitimate idioms like exec(compile(…)). We asked Vulnetix whether a record built on such a match means malware or a lead worth a look. It answered in the terms of the standards it publishes in: each record states its class (malware, weakness, exploit or vulnerability) and how it was reviewed.

Finding 5: among flags registries left up, how many are wrong

Stage 2 took flags older than 30 days whose packages the registries still serve, a random sample of up to 50 per tool and ecosystem, and read them. Two model raters from different vendors, blind to which tool flagged a package, labelled each one against a codebook written beforehand ("malicious" = sends credentials, keys, files or environment out, runs a fetched payload, installs persistence, destroys data, or hides any of that). They were 29/30 right on held-out control packages and agreed with each other at κ = 0.715 on the sample; every disagreement was decided by a person reading the full code, and a blind re-read of 17 packages changed one label.

This is the error rate among flags registries did not act on, the flags most likely to be wrong; it is not a tool's overall false-positive rate. For OpenSSF's own analysis and Google, a flag that stayed up is almost always malware the registry missed. For Vulnetix, it is almost always a package that does nothing harmful: most of its records are matches on a domain or a string in a file. A day after our snapshot, Vulnetix withdrew 92% of its npm and PyPI records (Finding 7), and by 2 November 96 of the 100 we rated were withdrawn. Amazon Inspector's benign ones are mostly odd rather than harmful (a squatted name, a leaked API key), which a broader "suspicious" might still cover; we asked them.

Finding 6: scanners catch most known malware, and bury it in false alarms

Feeds are fed by scanners, so stage 3 ran four open ones with their defaults, offline in a sandbox, 300 s per package: GuardDog 3.2.0 (DataDog), malcontent 1.27.1 (Chainguard), OSSGadget 0.2.38 detect-backdoor (Microsoft), and a one-line baseline, "an install hook that makes a network call or starts a process". Three sets: 390 known malware samples from DataDog's public dataset (2025-2026), 396 popular packages no feed lists, and 397 random uploads from 10-17 October that no feed listed a week later.

Benchmarks usually stop there, on a set that is half malware. A registry is not. In that October week the open feeds flagged 20 of the 28,833 PyPI projects that released, 0.07%. At that rate, on PyPI's ~4,100 releasing projects a day:

Hundreds of false alarms for each real one, whichever you pick. That is not a reason to drop them; it is why they are triage for a person and not a verdict, and why a scanner's hit needs a reader before it becomes a published verdict. npm is worse: a far larger volume and a lower base rate.

What fires on ordinary new packages: GuardDog's rules for reading the environment, reading files, starting processes, downloading and running, which is what CLIs and agent tooling do all day; malcontent's URL, host:port and HTTP POST patterns; OSSGadget's "LOLBAS" and "data exfiltration" patterns, which match README headings and return this;. We searched all 397 recent packages for the codebook's behaviours and read every hit; none was malicious.

Two more things. On npm, recall drops for newer malware: malcontent catches 87% of samples from the first half of 2025 and 42% from the first half of 2026. And OSSGadget's tool package ships without its own rules, so out of the box it reports nothing for every package and does not say so; we added the rules of the same version. Caveat: the malware set is DataDog's and GuardDog is DataDog's scanner, so if DataDog finds malware partly with GuardDog, GuardDog's recall here is flattering. malcontent's author adds that every scanner has likely been trained or tested on this dataset, so recall on it is probably optimistic for malware a scanner has not seen; the repeat measurement adds samples published after 24 October.

Finding 7: who shows their work, and who answers

For completeness of the analysis, we measured what each record shows and whether its verdict can be disputed, with the criteria written down before we read the data.

What a record shows, in 30 random records per tool:

No tool does all three. Vulnetix shows the matched line and the rule; on 29 of the 30 its reason is a standard sentence rather than one about the package. Amazon Inspector's best records are real analyses: files, hosts, what the code does. GitHub's text is advice ("should be considered fully compromised"), and the 39,857 records that name no tool say only that the package "was found to contain malicious code".

How feeds answer, over 100 public disputes since January 2025:

• OpenSSF: 44 disputes, 34 answered, median 0.7 days; 24 records withdrawn or corrected; 10 never answered, the oldest for ten months.
• GitHub: 48, 36 answered, median 1.2 days; 26 changed; 6 package owners sent to npm support, where its malware advisories come from; 12 never answered.
• Vulnetix: 8 since its tracker opened on 21 October, one of them ours. All answered within three days, all changed. A week is not twenty-two months, so this compares nothing yet.

Then, on 25 October, Vulnetix ran an audit and withdrew 6,762 of the 7,362 npm and PyPI records we studied (92%), each withdrawal published with a statement that names the rule, the file and line, and why it is not malicious. It came the day after Vulnetix reviewed the records challenged in its tracker, ours among them, and found no malicious behaviour in any of those packages. We saw nothing on that scale from any other feed. Findings 2, 3 and 5 describe the feed as it stood the day before; the repeat measurement counts these withdrawals as made before our letters. Vulnetix explains that its database was in its first week, when an automatic approval rule published static-rule matches before human review was wired in. Of the 600 records it kept, 577 had been approved by that rule; by 2 November, after our letters, it had withdrawn 52 more and reclassified 5 as vulnerabilities. It has since published a dispute process in which a person decides every case.

What every tool said about our own packages

We publish 77 packages (52 on PyPI, 25 on npm). Here is what each tool says about them. One publisher is a case, not a sample; read it as what a maintainer runs into.

• Eight of the nine feeds (Amazon Inspector, ReversingLabs, Checkmarx, OSSF Package Analysis, Google, GitHub, kam193, Oracle Macaron): nothing. None of the 77 is in their records.
• Vulnetix: nine PyPI packages listed as malware, June to October. Every matched line we traced is a test, an example or a constant: a test asserting our fetch tool refuses file:///etc/passwd, curl examples in READMEs, Circle's USDC contract address, the secp256k1 curve order, a cron line in install instructions, key_path: ~/.ssh/id_rsa in a configuration guide, and the list of persistence locations our own sandbox watches when it runs other people's packages. We disputed six in their repository on 21 October; three more were found while writing this. Vulnetix reviewed all nine by hand within days: one record withdrawn, eight reclassified from malware to vulnerabilities. And those eight were real: an MCP server that answered anyone when no key was set, a probe that resolved a name twice so a second answer could point it inside our network, sanctions data read incompletely, a permission that survived a runtime change, among others. We fixed all eight and released the fixes; Vulnetix checked the published packages, set the fixed versions on all eight records, and opened five new ones for flaws we described in our own fixes. Thank you, Vulnetix: the review was quick and done in the open, it found real flaws that we have since fixed, and every step of it is public.
• GuardDog 3.2.0 (open source, DataDog), which we ran ourselves with defaults: no verdict, by design. It lists heuristics for a person to review. Its "threat" heuristics fired on 21 of 86 scans (the latest version of all 77, plus the nine versions Vulnetix flagged, four of which it also questioned): the Telegram API address our hub uses to notify its owner, .env and environment reads, WARDEN's own detection patterns for credential paths, ffmpeg run with its output silenced, a regular expression for Cyrillic. Each is code doing what it says; a reviewer clears it, and asking for that review is exactly what GuardDog promises.
• WARDEN, our own MCP firewall, so read this twice as interested: it reads the tool definitions a model sees, not package code, and gives our MCP servers no blocking match. It does attach advisory notes: 12 on our hub's 13 tools, mostly "this tool takes a credential parameter", which is true. 37.5% of all servers in the registry get advisory notes; 1.8% get a blocking match.

The pattern, if one case can show one, is that text rules matching anywhere in a package, tests and docs included, meet packages that describe attacks: security tools, sandboxes, test suites. Finding 5 shows how common that is beyond us.

What's next

• The same measurement again, two to three weeks after the feeds got their numbers: the same rows checked against the registries, what each feed withdrew or reclassified (before and after our letters, reported apart, since a letter can cause a change), whether registries removed the packages we read as malicious and kept the ones we read as benign, and a second week of recent uploads through the same scanners. The plan is in the methodology, committed before the run. Vulnetix's audit of 25 October counts as a change made before the letters. At Vulnetix's suggestion it also measures Vulnetix's STIX feed and runs its CLI's malware scan, set up the way Vulnetix advised.
• The feeds' answers to our letters: Vulnetix answered twice, and both answers are quoted in full in the write-up; kam193 and malcontent's author answered too. Every later answer is added with a date.

The method follows the ten pitfalls of Arp et al. (USENIX Security 2022), TESSERACT's handling of time, and the labelling practice of Guo et al. (ASE 2026). The full write-up in five languages, the methodology with its pitfall checklist, and the scripts and seeds to repeat every number: the full write-up · the methodology.

If you run one of these feeds and think we got something wrong, tell us; corrections go into the write-up with a date.

AI assistance was used for the analysis, the code and the writing.