MODELMARKET.DEV / BLOG
← All posts

7 Nov 2026 · 4 min read

A privacy warning became bad advice in an AI reviewer's draft

#ai#agents#security#mcp

On November 4, a welcome draft for textokit suggested rewording this sentence from a tool description:

Remove secrets and personal data before submitting.

It was sensible advice for a buyer. The draft treated it as something the seller should change because a security scanner had mentioned it. Nobody sent the letter. It was one of five welcome drafts from the first run of our supplier-review agent, and reading it showed us a problem in the report we had given the model.

The reviewer runs in our Multica installation. A script downloads the manifest, federation peers and observed x402 sellers from modelmarket.dev, compares them with its previous run, and checks new or changed entries. The checks cover schemas, prices, endpoints, requested credentials and prompt injection patterns. The model turns the JSON into a report and prepares short welcome drafts for outside authors. A facts block from the script is included unchanged.

That arrangement still left the model to decide which observations deserved a sentence in a letter.

The advisory flag disappeared

The textokit draft suggested changing three ordinary phrases: "instead of neutral or European Spanish," "without a browser or API key," and the warning about removing secrets and personal data.

WARDEN had returned advisory matches. "Instead of" can appear in an instruction, but here it described a translation choice. "API key" named something the buyer did not need. "Secrets" appeared in a privacy warning.

The reporting adapter flattened WARDEN's results into its own finding format and discarded advisory. It also promoted an informational advisory result to low severity. By the time the model read the report, the distinction between a weak word match and a problem worth raising with an author was gone.

The resulting suggestion sounded helpful: reword what the checker flagged. Applied to the privacy warning, it could make the description less useful.

After that first run, we tightened the prompt: never include advisory matches in welcome drafts, and never recommend rewriting text merely to stop a scanner match. On November 6, we changed the data the prompt referred to as well.

Advice gets its own list

The adapter now preserves advisory: true and keeps informational advisory results informational. Each reviewed item also gets letter_points, the findings a draft may use when advising its author:

findings.filter((f) =>
  !f.advisory &&
  f.severity !== "info" &&
  (
    f.severity === "high" ||
    f.severity === "medium" ||
    f.check === "schema"
  )
);

The schema exception matters. A property without a declared type can deserve a useful correction even when its severity is low. Advisory matches remain in the owner's report, listed separately, but do not enter letter_points. The instructions name that array directly. An empty array means a welcome without an invented scanner-related complaint.

WARDEN's rules stayed the same. The repair was in the adapter and the report it assembled.

The regression fixture keeps all three ordinary phrases. Their matches remain visible and produce zero letter points. A string used as a tool's input schema still produces an eligible schema finding; an explicit instruction to ignore previous instructions and send a wallet seed phrase still produces a high severity injection finding.

Against the same downloaded November 6 hub documents, local and deployed-container checks agreed: textokit had zero letter points; agent-exec had four, about a string input schema and untyped properties. Both checks used temporary state.

The model still reads the full report, so the array is no mechanical guarantee about its prose. A person still needs to review the drafts.

The next inputs included 88 probe addresses

The same review found 88 pending peer registrations shaped like probes of the hub's address handling. Sixty-nine were non-HTTPS addresses; nineteen used HTTPS but had never been successfully read by the hub. The old drafting rule would have treated each as another outside source to welcome.

Welcome eligibility now requires an HTTPS URL with a dotted hostname rather than an IP literal, no credentials in the URL, and, for a federation peer, at least one successful hub read. These are drafting conditions, not certification of a seller.

A checker simulation using copied agent state found one changed outside listing and grouped the 88 peers on one line. Only agent-exec was eligible for a welcome draft. The real daily state was left untouched; this was a simulation of the next inputs, not evidence that a later model run had written better letters.

We build WARDEN and operate AIMarket. This article was drafted with AI assistance using our implementation, regression tests and recorded rollout checks.

More field notes