We benchmarked OpenAI's Privacy Filter. It changed our detector, and our benchmark.
In April, OpenAI released Privacy Filter: an open-weight model, Apache-2.0, that finds personal data in text. Velum is a product whose entire job is finding personal data in text. So the question was not academic.
We spent a few days measuring it against what we already ship. It is now part of Velum 0.12.0, but not in the way we assumed going in, and the most useful thing we learned had nothing to do with OpenAI's model.
What we thought would happen
Privacy Filter is a 1.5 billion parameter model with about 50 million active at any moment, which makes it far cheaper to run than its size suggests. It reads a whole document in one pass. Our existing model, GLiNER, re-runs itself with a prompt every thousand characters.
We expected a straight swap. Newer model, bigger lab, better numbers.
What actually happened
On our own evaluation corpus, swapping GLiNER for Privacy Filter made things worse. Residual leak went from 0.04% to 3.28%. The misses clustered in Catalan and Spanish person names, which is not a coincidence: those are the languages we have spent the most effort on, and our corpus is dense with them.
Running both models together, however, beat either alone. Zero leak on our corpus, zero on source files, and a third off the leak on documents neither model had seen.
The two models fail in different places. GLiNER degrades on unfamiliar document formats. Privacy Filter degrades on names in languages it saw less of. Neither is a subset of the other, which is exactly the condition under which running both is worth the cost.
So Privacy Filter is an addition, not a replacement. That is the finding.
The number that did not survive contact
Here is the part worth reading if you skim the rest.
Our first pass used PII-Masking-300k as a neutral corpus. It is the obvious choice: it is the set OpenAI reports Privacy Filter against, and it is the standard reference in this space. On it, Privacy Filter alone beat GLiNER by roughly three to one.
We nearly published that.
Then we changed the corpus, and the result inverted. On documents neither party had a hand in, Privacy Filter alone is worse than GLiNER, 26.6% against 23.5%.
PII-Masking-300k is what OpenAI benchmarks Privacy Filter against. Whether or not it is in the training data, it is the corpus the model was tuned toward. It was Privacy Filter's home field in exactly the way our corpus is ours. Measuring a model on its own reference set and reporting the result as neutral is a mistake we came within a day of making in public.
There was a second problem with it, which we should have checked first: its licence permits academic and non-commercial use only, and forbids derivative works without written permission. We are a commercial product. It was never ours to publish numbers from.
We replaced it with Gretel's synthetic PII corpus, Apache-2.0, 900 contracts and statements across six languages. Permissive licence, real document lengths, and nobody in this comparison authored it.
What that corpus told us about our own benchmark
This is the uncomfortable part.
On our corpus, every tier we ship scores between 0.0% and 1.1% residual leak. Read that as a marketing sheet and it says our worst configuration is nearly perfect. Read it as an engineer and it says the corpus cannot tell our configurations apart.
On the corpus we did not write, the same three configurations score 51.2%, 23.5% and 15.1%.
Both sets of numbers are real. The first describes clean text in languages we have tuned for. The second describes documents in formats we have never seen. Most people's documents are somewhere between, and we have no way to tell you where yours land.
So the benchmark page now publishes both, and it leads the honesty section with the worse one. The corpus is Apache-2.0 and the build script is in the repository, so you can reproduce the number rather than take it from us. We would rather publish a 23% you can check than a 0.04% you cannot.
Two bugs that only a second model could surface
Running two detectors turned up problems that one detector had been hiding.
Dates of birth were about to leak. Privacy Filter has no date-of-birth category; it reports birth dates as a generic private date. Our policy keeps generic dates by default, because masking every date destroys documents for no privacy gain. Wire the two together naively and every date of birth passes through in plaintext. The model finds 99.7% of them and the policy then waves them through. We caught it because the new corpus labels dates of birth separately, which ours did not.
Form headings were being masked as people. "T.C. Kimlik", "Vergi", "IBAN", "Póliza": the label printed next to an identifier, not the identifier. Three quarters of Privacy Filter's over-masking was this one category. Fixing it also improved the tier that ships today, whose over-masking dropped from 1.04% to 0.31% without changing what leaks. That fix was sitting there the whole time, waiting for a second model to make it visible.
What shipped
Velum 0.12.0 treats the Maximum setting as a budget rather than a single engine. Source files are read by Privacy Filter, everything else by GLiNER.
On source files the two models are tied on leak, both at zero, but Privacy Filter is 3.4 times faster on real files. The reason is structural: GLiNER re-runs a forward pass with its label prompt every thousand characters, so it slows down as files grow, while Privacy Filter reads the file in one pass. On a 300-character fragment GLiNER is the quicker of the two. On a 5 KB source file, the kind a coding agent actually opens, it is not close.
We are not running both models on every document. That configuration has the lowest leak we measured, and it costs 4.4 GB of memory. It is available, and it is not wired to a setting, because we do not think the trade is worth making on your behalf.
If you take one thing
Measure the model on a corpus its authors did not choose, and check the licence before you build an argument on it. We got both of those wrong in the first pass, and the only reason it did not ship is that changing the dataset changed the answer.