What does residual leak mean?

The share of must-mask spans in the gold set that reached the output unmasked. On the Maximum tier it is 0.04%, 0.00% on source files. It is the number that matters, because a span you never see is the one that ends up in a prompt.

Which detection tier should I use?

High is the default: Regex + on-device NER, 99.3 F1 and 1.1% leak at ~140 ms per document, with the model bundled and offline. Maximum lowers leak to 0.04% at ~350 ms per document, at the cost of a ~2.9 GB local models. Standard is the regex floor with no model, which leaks 37% because it cannot see the names and places that only context reveals.

What is the benchmark measured against?

1,800 labelled spans, 200 per language across 9 languages (es · ca · en · de · fr · it · tr · uk · us). The set is version-controlled and generated by a seeded PRNG, so it is byte-identical on every machine that rebuilds it.

Do these numbers hold on real, messy text?

They hold on the languages and document shapes our country packs target, which is what the table above measures. On formats nobody has tuned for they do not, and we publish that rather than leave you to find it: on 900 documents from a corpus we did not author (gretelai/synthetic_pii_finance_multilingual, Apache-2.0), regex + on-device NER leaks 51.2% and regex + full NER leaks 23.5%. Both are reproducible; which one describes your documents is the question worth asking.

Could the test data be rigged in your favour?

National IDs, IBANs and cards are minted from their published checksum algorithms, never brute-forced through Velum's own detector, which would guarantee a perfect score by construction. An independent oracle then re-checks that the regex packs recognise every committed ID.

Can I reproduce the run myself?

Yes. The gold data is seeded and version-controlled, the models are pinned by hash, and the run is one command on Apple M2 Max · 12 cores · 64 GB using the CPU execution provider, no GPU. Ask us for the gold set and we will send it.

Benchmarks

We measured what gets through.

A privacy tool is only as good as the data it misses. So we scored the engine against 1,800 labelled spans across nine languages, using the exact models the apps ship, and published the leak, including the part that still gets past us.

Escritura de compraventa, masked before it reaches any AIsynthetic

Comparecen ⟦PERSON_1⟧, mayor de edad, con NIF ⟦NIF_1⟧, y ⟦PERSON_2⟧, para formalizar la transmisión del inmueble sito en ⟦ADDRESS_1⟧. El pago se domicilia en la cuenta ⟦IBAN_1⟧ a nombre de M. Arregui, por importe de 200.000 €.

Everything under a bar is restored only on your machine. The one span left exposed is a miss. At the Maximum tier, about one must-mask span in 1,500 slips past. That number is the point of this page.

Three intensities. You trade speed for the last of the leak.

Every figure is the must-mask F1 and residual leak, averaged across all nine languages. Must-mask F1 scores only the data the policy is meant to hide. Organisations and places it deliberately keeps are left out, so the number reflects the privacy job and nothing else.

Standard

Regex floor, zero model

73.1%

must-mask F1 · instant · none

residual leak37%, misses contextual names and places by design

High

Regex + on-device NER

99.3%

must-mask F1 · ~140 ms per document · bundled, offline

residual leak1.1% average · ~2% on ES/CA

Maximum

Regex + GLiNER on documents, Privacy Filter on source files

99.7%

must-mask F1 · ~350 ms per document · ~2.9 GB local models

residual leak0.04% on documents, 0.00% on source files

Standard is the free regex floor: instant and exact where it fires, but blind to the names and places only context reveals. High is the default the apps ship with. Maximum adds the full model for the lowest leak we measured.

Where the leak lives, and where it can’t.

The residual leak is always in the fuzzy, contextual types, like a name the model scored just under threshold. The structured identifiers never leak: they are matched deterministically from their published checksums. Across the whole gold set these scored 100% precision and 100% recall:

email
IBAN
credit card
national ID
phone
IP address
AWS / GitHub token
JWT

LanguageLeakMust-mask F1

esSpanish0.3%99.7%

caCatalan0.0%99.5%

enEnglish0.0%99.3%

deGerman0.0%100.0%

frFrench0.0%100.0%

itItalian0.0%100.0%

trTurkish0.0%100.0%

ukUK English0.0%100.0%

usUS English0.0%100.0%

Maximum tier (regex + full NER). Leak = share of must-mask spans missed; lower is safer.

What these numbers cover

0.04% on the languages we ship packs for. Higher on document formats we have never seen, and we publish that too.

The table above is measured on nine languages we build country packs for, in the document shapes those packs target. That is the number that applies if your documents look like the ones Velum was tuned for, and it is the one the app is held to. It is not the whole picture, so we also score 900 contracts and statements in 6 languages from gretelai/synthetic_pii_finance_multilingual, published under Apache-2.0, which nobody here had a hand in. Unfamiliar formats are harder, and the tiers separate sharply:

regex + on-device NERF1 54.4%leak 51.2%

regex + full NERF1 65.2%leak 23.5%

regex + Privacy Filter + full NERF1 60.4%leak 15.1%

Which number applies to you depends on your documents, and we cannot tell from here. If they resemble the corpora our country packs target, expect the first table. If they are formats nobody has tuned for, expect closer to the second. The licence is permissive and the build script ships in the repository, so both are yours to reproduce.

Methodology & verification

The gold data is seeded and version-controlled, the models are pinned by hash, and the run is one command. If you are evaluating Velum and want to grade it yourself, ask us for the gold set and we will send it.

Gold set: 1,800 labelled spans, 200 per language across 9 languages: es · ca · en · de · fr · it · tr · uk · us. Version-controlled and generated by a seeded PRNG, so the set is byte-identical on every machine that rebuilds it.
No circularity: National IDs, IBANs and cards are minted directly from their published checksum algorithms, never brute-forced through Velum’s own detector, which would otherwise guarantee a perfect score by construction. An independent oracle then re-checks that the regex packs recognise every committed ID.
Pinned models: Full NER (Maximum, documents): onnx-community/gliner_multi_pii-v1
onnx/model.onnx · sha256 7704865e414f2459…
Privacy Filter (Maximum, source files): openai/privacy-filter
onnx/model_quantized.onnx · sha256 a325fb5341567a73…
On-device NER (High): Xenova/bert-base-multilingual-cased-ner-hrl
onnx/model_quantized.onnx · sha256 5b65139844be260b…
Both are fetched from public HuggingFace repositories and verified against these hashes; the run uses the exact files the apps ship.
Metrics: Residual leak is the share of must-mask spans the engine missed, after the shipped policy keeps context types. Must-mask F1 is precision and recall over the types the policy is meant to hide. Matching is span overlap with loose type, since label vocabularies differ across models.
Machine: Apple M2 Max · 12 cores · 64 GB. CPU execution provider, no GPU. Node v26.5.0. Latency figures are specific to this machine; a single run, no variance bands.
How the run is made: pnpm --filter @velumprivacy/core bench:leak --chains regex,regex+lite,regex+full,regex+pf
The two pinned models are fetched, then the engine’s own bench harness scores them against the gold set. The engine makes no network calls at runtime. A static scan and a runtime network-block test enforce that in the build.
Getting the gold set: We share the gold set with security teams evaluating Velum, so you can score our engine, or anyone else’s, on the same 1,800 spans instead of taking this page on trust. Ask for the gold set.

Download the full report (PDF)A share-ready version for security review.

Questions an evaluator asks

What does residual leak mean?: The share of must-mask spans in the gold set that reached the output unmasked. On the Maximum tier it is 0.04%, 0.00% on source files. It is the number that matters, because a span you never see is the one that ends up in a prompt.
Which detection tier should I use?: High is the default: Regex + on-device NER, 99.3 F1 and 1.1% leak at ~140 ms per document, with the model bundled and offline. Maximum lowers leak to 0.04% at ~350 ms per document, at the cost of a ~2.9 GB local models. Standard is the regex floor with no model, which leaks 37% because it cannot see the names and places that only context reveals.
What is the benchmark measured against?: 1,800 labelled spans, 200 per language across 9 languages (es · ca · en · de · fr · it · tr · uk · us). The set is version-controlled and generated by a seeded PRNG, so it is byte-identical on every machine that rebuilds it.
Do these numbers hold on real, messy text?: They hold on the languages and document shapes our country packs target, which is what the table above measures. On formats nobody has tuned for they do not, and we publish that rather than leave you to find it: on 900 documents from a corpus we did not author (gretelai/synthetic_pii_finance_multilingual, Apache-2.0), regex + on-device NER leaks 51.2% and regex + full NER leaks 23.5%. Both are reproducible; which one describes your documents is the question worth asking.
Could the test data be rigged in your favour?: National IDs, IBANs and cards are minted from their published checksum algorithms, never brute-forced through Velum's own detector, which would guarantee a perfect score by construction. An independent oracle then re-checks that the regex packs recognise every committed ID.
Can I reproduce the run myself?: Yes. The gold data is seeded and version-controlled, the models are pinned by hash, and the run is one command on Apple M2 Max · 12 cores · 64 GB using the CPU execution provider, no GPU. Ask us for the gold set and we will send it.

Request a demo

See it work on your own data.

Tell us what you are working with and we will walk you through it on your own data. Nothing leaves your hands.

We respect your privacy. Your details are only used to reach you.