The model that wins our PII benchmark loses on messy text
Every PII detection tool quotes one accuracy number. Ours is 99.8% F1 and a 0.07% residual leak. It is a measured number rather than a marketing one, taken by a fixed harness on a fixed data set on a named machine. It is also the most flattering true number we have, and a buyer who takes it at face value has learned very little about what happens to their contracts.
So this post is the rest of it: what the number measures, where it stops holding, and the case where the tier that wins our headline benchmark comes second on text that looks like real documents.
What the headline number measures
The gold set is 1,800 labelled spans, 200 per language across nine: Spanish, Catalan, English, German, French, Italian, Turkish, UK English, US English. It is generated by a seeded PRNG and checked into version control, so it is byte-identical on every machine that rebuilds it and a run last month is comparable to a run today.
Two details matter more than the size.
The identifiers are not minted through our own detector. Every DNI, NIE, CIF, SSN, NINO, NIR, Steuer-IdNr, TC Kimlik, Codice Fiscale, IBAN and card in the set is generated directly from its published checksum algorithm. An earlier generator brute-forced candidate IDs until Velum's own regex accepted them, which guarantees 100% recall by construction and makes that half of the benchmark a tautology. A separate oracle pass then checks that the regex packs do recognise every committed value, as a cross-check rather than a definition.
Leak is the privacy number, not F1. Residual leak is the fraction of
must-mask gold spans a chain missed. Raw recall on our tables looks low because
it counts organization and location spans that the shipped policy
deliberately keeps, since masking every company name and city would make a legal
document unreadable and neither is must-mask the way a person or a national ID
is.
Here is what the three shipped tiers do on that set, 9-language averages:
| Tier | Chain | Leak | Over-mask | Must-mask F1 | p50 | RSS |
|---|---|---|---|---|---|---|
| Standard | regex only | 37.03% | 12.46% | 73.1% | ~0 ms | ~118 MB |
| High (default) | regex + 170 MB on-device NER | 1.04% | 1.04% | 99.0% | ~7 ms | ~850 MB |
| Maximum | regex + 1.1 GB GLiNER | 0.07% | 0.23% | 99.8% | ~26 ms | ~3,041 MB |
That last row is the number on our /benchmarks page. It says that on clean, templated text, the big model closes almost all of the remaining gap.
The same tiers on messy text
Synthetic data is shaped like its generator, however careful the generator is. So there is a second set: ten deliberately messy documents the generator never saw, with the noise real legal and scanned documents carry. ALL-CAPS holder lines, OCR character swaps sitting next to clean values, names broken across a line wrap, dense multi-entity paragraphs, punctuation that ran together.
The ranking changes:
| Chain | Leak | Over-mask | Must-mask F1 |
|---|---|---|---|
| regex | 16.7% | 20.0% | 81.6% |
| regex + on-device NER (High) | 4.2% | 11.5% | 92.0% |
| regex + GLiNER (Maximum) | 8.3% | 12.0% | 89.8% |
Three things worth reading off that table.
Leak is 4 to 8%, not 0.07%. On messy text, several points of sensitive material get through at every tier. That is a different product conversation than the one the headline number starts.
The cheaper chain wins. The 170 MB quantized model, running in about 7 ms and 850 MB of RSS, comes out ahead of the 1.1 GB one that needs 26 ms and roughly 3 GB resident.
And the regex floor alone, with no model at all, scores 81.6% here against 73.1% on the clean set. It goes up. The messy set is dense with the structured identifiers regex owns exactly, and thin on the paragraph-scale context the models exist to read.
The honest caveat, stated as loudly as the result: n = 10. One missed span moves leak by about 4 points. This is not a tier ranking and we do not present it as one. It is a check on whether the synthetic numbers are an upper bound, and it says they are.
Why the tiers fail in the ways they do
Each detector alone, on the clean gold, is a bad privacy control:
| Detector alone | Leak | Over-mask | Must-mask F1 |
|---|---|---|---|
| regex | 37.03% | 12.46% | 73.1% |
| on-device NER (170 MB) | 51.60% | 33.91% | 58.3% |
| GLiNER (1.1 GB) | 27.16% | 0.22% | 84.1% |
The regex floor is exact where it fires and blind where it does not. Every national ID, IBAN, card, email, phone, IP and cloud token scores 100% precision and 100% recall, because a checksum either validates or it does not. It cannot see that "Reunidos D. Andrés Castro y la perito Ana Ruiz" contains two people, because nothing about those characters is structurally distinct from any other words.
The span model has the opposite shape. It reads people and places well and leaks 27% on its own, because it does not emit the structured identifiers at all. A zero-shot span model has no notion of a mod-97 check on an IBAN or the letter table on a Spanish DNI. It can label a string that looks like an ID, and it does not know whether the ID is real.
The small NER model is the worst of the three alone, at 51.6% leak and 33.9% over-mask, and it is still the one we ship by default, because its errors are covered by the floor underneath it and its recall on people is the thing the floor cannot supply.
So none of these tiers is a detector in the sense a buyer means. The product is the chain, and the chain's behaviour is not predictable from any single row.
The tuning trap, which we walked into
At one point we lowered the full tier's score threshold from 0.7 to 0.5. Raw recall on the GLiNER row went from 66.7% to 79.2% on the messy set, which looks like exactly the kind of win you ship.
Measured on the shipped chain instead of the single detector, it was a
regression. Leak did not move, because there was almost no leak left to buy at
that tier. Over-mask went up roughly fivefold, and on UK English it went from 0
to 5.1%. Must-mask F1 dropped. The entire recall gain was in organization and
location, the two types the policy keeps, so a user's masked output was
identical in the good cases and more damaged in the bad ones.
We reverted it and nothing shipped. The rule that came out of it is now a gate in the repository: no threshold change is judged on raw recall or on a single-detector row, only on the leak, over-mask and must-mask F1 of the chain that actually ships. It is worth writing down because the failure mode is not exotic. A number went up, the number was real, and the number was about something other than the thing we sell.
What we actually conclude
The 0.07% figure is an upper bound on real-world performance and we say so on the page that quotes it. The held-out set suggests the real number is several points higher on documents that have been through a scanner, and the largest model is not automatically the safest chain once the text stops being clean.
The concrete follow-up is not a better model. It is a better eval: at least 200 hand-labelled real documents instead of ten, which is the only thing that would let us make a claim about messy text with a straight face. Until that exists, the messy numbers in this post are directional and we will keep labelling them that way.
Where the numbers come from
Every figure here comes from one run on 2026-07-23: Apple M2 Max, Node v26.5.0, CPU execution provider, no GPU, single run, no variance bands.
What you can check:
- The two models are public and pinned by SHA256:
onnx-community/gliner_multi_pii-v1(onnx/model.onnx) andXenova/bert-base-multilingual-cased-ner-hrl(onnx/model_quantized.onnx). The weights we score are the weights the apps ship, and both are downloadable from HuggingFace by anyone who wants to test them against their own data. - The full per-language and per-type tables, including the rows that look bad, are on velumprivacy.com/benchmarks, with the same held-out caveat stated there.
- The behaviour that matters most to a buyer, which is what the tool does to your documents, is testable directly. Run your own worst file through it and read the masked output. That is the only measurement taken on text you care about.
If you are evaluating Velum and want to grade it yourself, ask us for the gold set and we will send you the same 1,800 spans, so you can score our engine, or anyone else's, on the data these tables came from.
Velum is a local PII masker for text and files. Detection runs on your machine with no network calls at runtime. The engine is source-available via npm under PolyForm Noncommercial, and the desktop app and browser extension are paid.