01
Standard
Regex floor, zero model
73.2%
must-mask F1 · instant · none
residual leak37%, misses contextual names and places by design
Benchmarks
We measured what gets through.
A privacy tool is only as good as the data it misses. So we scored the engine against 1,800 labelled spans across nine languages, using the exact models the apps ship, and published the leak, including the part that still gets past us.
Escritura de compraventa, masked before it reaches any AIsynthetic
Comparecen ⟦PERSON_1⟧, mayor de edad, con NIF ⟦NIF_1⟧, y ⟦PERSON_2⟧, para formalizar la transmisión del inmueble sito en ⟦ADDRESS_1⟧. El pago se domicilia en la cuenta ⟦IBAN_1⟧ a nombre de M. Arregui, por importe de 200.000 €.
Everything under a bar is restored only on your machine. The one span left exposed is a miss. At the Maximum tier, about one must-mask span in 1,500 slips past. That number is the point of this page.
Three intensities. You trade speed for the last of the leak.
Every figure is the must-mask F1 and residual leak, averaged across all nine languages. Must-mask F1 scores only the data the policy is meant to hide. Organisations and places it deliberately keeps are left out, so the number reflects the privacy job and nothing else.
02
High
Regex + on-device NER
99.2%
must-mask F1 · ~140 ms per document · bundled, offline
residual leak0.97% average · 2.6% on CA, 0.7% on ES
03
Maximum
Regex + GLiNER on documents, Privacy Filter on source files
99.6%
must-mask F1 · ~350 ms per document · ~2.9 GB local models
residual leak0.04% on documents, 0.00% on source files
Standard is the free regex floor: instant and exact where it fires, but blind to the names and places only context reveals. High is the default the apps ship with. Maximum adds the full model for the lowest leak we measured.
Where the leak lives, and where it can’t.
The residual leak is always in the fuzzy, contextual types, like a name the model scored just under threshold. The structured identifiers never leak: they are matched deterministically from their published checksums. Across the whole gold set these scored 100% precision and 100% recall:
- IBAN
- credit card
- national ID
- phone
- IP address
- AWS / GitHub token
- JWT
LanguageLeakMust-mask F1
esSpanish0.3%99.2%
caCatalan0.0%99.4%
enEnglish0.0%99.8%
deGerman0.0%100.0%
frFrench0.0%100.0%
itItalian0.0%100.0%
trTurkish0.0%100.0%
ukUK English0.0%100.0%
usUS English0.0%98.3%
Maximum tier (regex + full NER). Leak = share of must-mask spans missed; lower is safer.
What these numbers cover
0.04% on the languages we ship packs for. Higher on document formats we have never seen, and we publish that too.
The table above is measured on nine languages we build country packs for, in the document shapes those packs target. That is the number that applies if your documents look like the ones Velum was tuned for, and it is the one the app is held to. It is not the whole picture, so we also score 900 contracts and statements in 6 languages from gretelai/synthetic_pii_finance_multilingual, published under Apache-2.0, which nobody here had a hand in. Unfamiliar formats are harder, and the tiers separate sharply:
regex + on-device NERF1 55.5%leak 49.9%
regex + full NERF1 65.1%leak 23.6%
regex + Privacy Filter + full NERF1 60.4%leak 15.1%
Which number applies to you depends on your documents, and we cannot tell from here. If they resemble the corpora our country packs target, expect the first table. If they are formats nobody has tuned for, expect closer to the second. The licence is permissive and the build script ships in the repository, so both are yours to reproduce.
What Velum leaves alone
A birth date is masked. An invoice date is not. Over-masking is an error too, so we measure it and publish it.
Any tool can reach high recall by redacting everything that looks like a date, a place or a link. What comes back is a document the model can no longer reason about, and you paid for the privacy with the answer. Velum draws the line at what actually identifies a person: a date of birth is masked, an ordinary date is left in place; a link carrying a credential is masked, a link to a public page is left in place.
The gold set carries 83 ordinary dates and 83 plain links that are deliberately left unlabelled, so a tier that masks them loses precision rather than passing unnoticed. This is how each tier scores against them:
TierOrdinary dates keptPlain links kept
Standardregex83/8383/83
Highregex + on-device NER83/8383/83
Maximumregex + full NER83/8383/83
Every tier leaves every one of them in place, and getting there took a correction. The span model at Maximum carries its own date-of-birth label, and on its own that label is not selective enough to tell a birth date from an invoice date: it took 27 of these 83 dates. It is now held to the same test the regex floor uses, and a date with no birth word beside it is recorded as an ordinary date rather than masked. The span is still found and still typed, so a policy that asks for every date still gets every date.
Separately from the tiers, Velum keeps locations, organisations, job titles, nationalities, ages, ordinary dates and plain links in plaintext by default. They are weak identifiers on their own and strong context for the model. If your documents need them gone, the policy is one setting away, and linkage escalation masks them automatically as soon as a direct identifier appears alongside them in the same document.
Methodology & verification
The gold data is seeded and version-controlled, the models are pinned by hash, and the run is one command. If you are evaluating Velum and want to grade it yourself, ask us for the gold set and we will send it.
- Gold set
- 1,800 labelled spans, 200 per language across 9 languages: es · ca · en · de · fr · it · tr · uk · us. Version-controlled and generated by a seeded PRNG, so the set is byte-identical on every machine that rebuilds it.
- No circularity
- National IDs, IBANs and cards are minted directly from their published checksum algorithms, never brute-forced through Velum’s own detector, which would otherwise guarantee a perfect score by construction. An independent oracle then re-checks that the regex packs recognise every committed ID.
- Pinned models
- Full NER (Maximum, documents): onnx-community/gliner_multi_pii-v1
onnx/model.onnx · sha256 7704865e414f2459… - Privacy Filter (Maximum, source files): openai/privacy-filter
onnx/model_quantized.onnx · sha256 a325fb5341567a73… - On-device NER (High): Xenova/bert-base-multilingual-cased-ner-hrl
onnx/model_quantized.onnx · sha256 5b65139844be260b…
Both are fetched from public HuggingFace repositories and verified against these hashes; the run uses the exact files the apps ship.
- Full NER (Maximum, documents): onnx-community/gliner_multi_pii-v1
- Metrics
- Residual leak is the share of must-mask spans the engine missed, after the shipped policy keeps context types. Must-mask F1 is precision and recall over the types the policy is meant to hide. Matching is span overlap with loose type, since label vocabularies differ across models.
- Machine
- Apple M2 Max · 12 cores · 64 GB. CPU execution provider, no GPU. Node v26.7.0. Latency figures are specific to this machine; a single run, no variance bands.
- How the run is made
pnpm --filter @velumprivacy/core bench:leak --chains regex,regex+lite,regex+full,regex+pfThe two pinned models are fetched, then the engine’s own bench harness scores them against the gold set. The engine makes no network calls at runtime. A static scan and a runtime network-block test enforce that in the build.
- Getting the gold set
- We share the gold set with security teams evaluating Velum, so you can score our engine, or anyone else’s, on the same 1,800 spans instead of taking this page on trust. Ask for the gold set.
Download the full report (PDF)A share-ready version for security review.
Questions an evaluator asks
What does residual leak mean?
The share of must-mask spans in the gold set that reached the output unmasked. On the Maximum tier it is 0.04%, 0.00% on source files. It is the number that matters, because a span you never see is the one that ends up in a prompt.
Which detection tier should I use?
High is the default: Regex + on-device NER, 99.2 F1 and 1.0% leak at ~140 ms per document, with the model bundled and offline. Maximum lowers leak to 0.04% at ~350 ms per document, at the cost of a ~2.9 GB local models. Standard is the regex floor with no model, which leaks 38% because it cannot see the names and places that only context reveals.
What is the benchmark measured against?
1,800 labelled spans, 200 per language across 9 languages (es · ca · en · de · fr · it · tr · uk · us). The set is version-controlled and generated by a seeded PRNG, so it is byte-identical on every machine that rebuilds it.
Do these numbers hold on real, messy text?
They hold on the languages and document shapes our country packs target, which is what the table above measures. On formats nobody has tuned for they do not, and we publish that rather than leave you to find it: on 900 documents from a corpus we did not author (gretelai/synthetic_pii_finance_multilingual, Apache-2.0), regex + on-device NER leaks 49.9% and regex + full NER leaks 23.6%. Both are reproducible; which one describes your documents is the question worth asking.
Could the test data be rigged in your favour?
National IDs, IBANs and cards are minted from their published checksum algorithms, never brute-forced through Velum's own detector, which would guarantee a perfect score by construction. An independent oracle then re-checks that the regex packs recognise every committed ID.
Can I reproduce the run myself?
Yes. The gold data is seeded and version-controlled, the models are pinned by hash, and the run is one command on Apple M2 Max · 12 cores · 64 GB using the CPU execution provider, no GPU. Ask us for the gold set and we will send it.
Request a demo
See it work on your own data.
Tell us what you work with and we will walk you through it on a call, on your own files. Nothing leaves your machine while we do.
Your address is used to reply to you, and for nothing else.