Why a public test
A screening tool has one job it must not fail: find the designated party, under whatever name the customer typed. Buyers cannot check how well a tool does it. Supervisors test the systems banks run, using files of listed and altered names, and publish anonymised figures: in Sweden, customer-screening systems found on average 86.3% of names as listed (median 99.3%) and 63.9% of altered ones (median 73.4%), ranging from 5.0%–96.1%, on UN and EU list names. The United Kingdom's regulator reported that 90.0% of alerts raised on exact names, and 75.0% on variants, identified the right sanctioned party, a different measure from names caught. Neither names a vendor. Sweden's regulator wrote that there are no established benchmark indices showing how effective these tools are. This report is not such an index: it is one vendor's test of its own engine, with its rules and failures published.
Vendors fill the gap with figures of the form “X% fewer false positives”, measured against undisclosed baselines on undisclosed data. A number without its test set cannot be checked, compared or reproduced. This report does not yet publish its case files either. It publishes the rules, every per-cell count and the case file's fingerprint, so each figure can be checked against the method and the same file rerun.
Design
- 37,543
- pre-registered cases in the frozen case file
- 28,420
- screens answered before the run stopped
- 7
- gates, written and approved before the first call
Questions, suites, an alert policy and gates were written and approved before any case was sent. The policy sorts every variation into three bands: must alert (a miss fails the release), should alert (reported, not gated, because identity is genuinely ambiguous) and not expected (recorded so nobody calls it a miss). Cases were drawn by fixed seeds, split by subject so no listed party appears in both development and test, and frozen; the case file is identified by its SHA-256 fingerprint, beginning 110825465fe0.
- Listed names as each list publishes them (OFAC (3,016) and UN (384); the listed-name suite has no EU or UK name), and the weak aliases a list itself flags as low quality.
- Altered spellings: the same names changed in one documented way each — case or spacing, accents, a company legal form, a joined or split particle, word order, official and passport transliteration, a one-letter typo on names of eight letters or more — and harder changes in the should band.
- Cross-list spellings: how another official list spells the same subject, linked by the lists' own references.
- Original scripts: names in the script the list publishes, from Cyrillic and Arabic to Chinese and Korean.
- Clean names from independent populations (registered companies and person names), checked not to be on OFAC, UN, EU or UK.
Each case was sent once through the same public API a customer calls. Every response was classified as an alert, a clean pass, or unresolved. Unresolved means the engine could not give a conclusive answer: screening unavailable, an uncertain verdict, or an incomplete candidate set. For a listed name, unresolved counts as a failure; for a clean name, it is left out of the false-alert rate and reported beside it. Both choices were fixed in advance.
Statistics
Proportions carry Wilson score 95% intervals. Gates use an exact one-sided 95% lower bound, because “zero misses” is a claim about a rate, not about a sample. Sample size follows from the claim: showing a false-alert rate at or below 0.1% with no false alerts observed needs about 3,000 clean names (the rule of three), and estimating a rate near that level to within about a third needs tens of thousands. Precision, the share of alerts that are real listed parties, depends on how often a screened name is listed (p):
precision = recall · p ÷ (recall · p + false-alert rate · (1 − p))
Because p is small in real traffic, precision is dominated by the false-alert rate. A suite with more unresolved answers than a stated share (10%) is reported as inconclusive rather than given a rate.
Listed names
Figure 1
Listed and cross-list names: share that alerted
Show the data
| Measure | Estimate | 95% CI low | 95% CI high |
|---|---|---|---|
| Listed names, exactly as published | 99.90% | 99.71% | 99.97% |
| Weak aliases the list itself flags as low quality | 93.37% | 90.25% | 95.54% |
| Spelling another official list uses for the same subject | 99.29% | 98.18% | 99.72% |
| Other cross-list spellings | 94.76% | 92.89% | 96.16% |
Of 3,053 listed names sent exactly as published, 3,050 alerted and none was served clear. The remaining 3 came back unresolved; under the pre-registered rule that is enough to fail the gate, whose exact lower bound is 99.746% against a required 99.9%. Spellings that other official lists use for the same subject alerted on 99.3%, with no clean pass either.
Altered spellings
Figure 2
Must-alert altered spellings: share that alerted
Show the data
| Measure | Estimate | 95% CI low | 95% CI high |
|---|---|---|---|
| Case, punctuation or spacing changed | 99.0% | 97.1% | 99.7% |
| Accents added or removed | 97.7% | 95.3% | 98.9% |
| Company legal form added, dropped or swapped | 54.3% | 48.7% | 59.9% |
| Name particle joined or split (al-, bin, von) | 89.7% | 85.7% | 92.6% |
| Name parts in a different order | 99.0% | 97.1% | 99.7% |
| Official transliteration (BGN/PCGN) | 92.0% | 88.4% | 94.6% |
| Passport transliteration (ICAO 9303) | 93.3% | 89.9% | 95.6% |
| One-letter typo (names of 8+ letters) | 77.7% | 72.6% | 82.0% |
Simple changes were handled well: case, spacing and word order each alerted on 99%. Two failed badly. A company legal form added, dropped or swapped alerted on 54.3%, and a single typo on a long name on 77.7%; a joined or split particle reached 89.7%. The should-band variations, which the policy does not gate, ranged from 25.7% for initials to 90.0% for an added title. Across all 4,500 variations, must, should and not expected together, 77.2% alerted. Each failure has a cause, traced in what failed.
Other scripts
Figure 3
Names in their original script
| Script | Cases | Alerted | Unresolved | Not found | Alerted % |
|---|---|---|---|---|---|
| Cyrillic | 296 | 272 | 24 | 0 | 91.9% |
| Arabic | 297 | 289 | 8 | 0 | 97.3% |
| Armenian | 2 | 0 | 2 | 0 | 0.0% |
| Georgian | 1 | 0 | 1 | 0 | 0.0% |
| Greek | 169 | 99 | 70 | 0 | 58.6% |
| Chinese (Han) | 300 | 0 | 300 | 0 | 0.0% |
| Korean (Hangul) | 92 | 0 | 92 | 0 | 0.0% |
| Hebrew | 35 | 35 | 0 | 0 | 100.0% |
| Japanese | 8 | 0 | 8 | 0 | 0.0% |
| Thai | 11 | 0 | 11 | 0 | 0.0% |
Hebrew, Arabic and Cyrillic names are screened. Cyrillic's unresolved answers are names that mix Cyrillic and Latin letters, spelled exactly as the lists publish them, which the engine refuses rather than guesses. Greek is partly screened. Armenian, Georgian, Chinese (Han), Korean (Hangul), Japanese, Thai names are not screened yet: every one was answered HTTP 503, screening unavailable, and marked review, never clear. Whether a person then sees it depends on the integration, so route those answers to one. It is the honest answer for an engine that cannot read a script, but it is coverage a buyer screening those regions does not have.
Clean names and workload
Of 13,700 clean names screened, 9,202 received a conclusive answer and 213 of those alerted: 2.3% (95% CI 2.027–2.642%), an upper bound of 264 per 10,000 against a target of 10. A further 32.8% came back unresolved, so under the benchmark's own rule the result is inconclusive. The set that ran is all 10,000 company names (GLEIF) but only 3,700 scored person names (ParaNames); person names false-alerted at 11.99% against 1.57% for companies (of resolved names), so the rate understates the full set. Both numbers matter to a buyer: a false alert costs a review, and so does an unresolved answer. The figure below turns them into a monthly workload, counting false alerts per clean name screened.
Figure 4
Expected monthly workload at a chosen volume
An illustration, not a prediction. Rates come from the October 13, 2026 measurement of the engine as deployed then. The clean-name rate is inconclusive under the benchmark's own rule (too many unresolved answers), understates the full clean set (person names ran only in part), and its clean names were checked against four lists. Ranges carry each rate's 95% interval.
- Listed names screened
- 10
- 1 in 10,000 of 100,000
- Listed names that alert
- 10
- 99.9% measured
- Listed names served as clear or PEP-only
- 0
- 0.00% measured
- False alerts on clean names
- 1,361–1,776
- 1.6% of clean names screened
- Unresolved: a person must look
- 32,829
- 0 listed, 32,829 clean; never served as clear
- Of every 100 alerts, real listed parties
- about 0.6
- precision at this rate
Precision = recall × p ÷ (recall × p + false-alert rate × (1 − p)), where p is the share of screened names that are listed and the false-alert rate is per clean name screened. When real matches are rare, almost every alert is a false one, even with a low false-alert rate: that is why the false-alert rate decides the workload.
Show the data
| Listed names | Share of alerts that are real |
|---|---|
| 1 in 100 | 39.36% |
| 1 in 300 | 17.69% |
| 1 in 1,000 | 6.04% |
| 1 in 3,000 | 2.10% |
| 1 in 10,000 | 0.64% |
| 1 in 30,000 | 0.21% |
| 1 in 100,000 | 0.06% |
The gates
Figure 5
Pre-registered gates and verdicts
| Gate | Rule | Verdict | Detail |
|---|---|---|---|
| G1 · Every listed name, exactly as published, alerts | 0 misses in at least 2,995 cases (recall at least 99.9%, one-sided 95%) | fail | 3,050 of 3,053 alerted and none was served clear. 3 came back unresolved, which the rule counts against: exact lower bound 99.746%, below 99.9%. |
| G2 · Every must-alert variation alerts | 0 misses in 300 for each must cell, plus cross-list spellings and Cyrillic and Arabic script | fail | All 8 variation cells, cross-list spellings and both scripts fail. Worst: legal-form variants 54.3%, one-letter typos 77.7%. |
| G3 · Clean names rarely alert | false-alert rate upper 95% bound at most 10 per 10,000 easy clean names | fail | 213 false alerts in 9,202 resolved clean names: 2.3%, upper bound 264 per 10,000. Every company name ran but only part of the person names, which alert more often, so this understates the full set; inconclusive under G7. |
| G4 · Political exposure is kept apart from sanctions | 0 PEP-only critical; 0 PEP-and-sanctioned below critical | not evaluable | Suite F did not run in this test (quota). In the previous run (October 8) it failed: no response in either run was served as critical, so a person who is both PEP and sanctioned returned high. |
| G5 · Same answer under concurrent load | 0 must-band decisions differ between concurrency 1 and 3 | not computable | No concurrency records in this run. |
| G6 · The regression suite behaves as before | Suite H (239 earlier cases) answers as it did | not computable | Suite H is not in this run's case file, so this gate could not be evaluated. |
| G7 · Few enough answers are unresolved to read a result | at most 10% unresolved per suite, else the suite is inconclusive | inconclusive | Variations 13.5%, scripts 42.4% (mostly Chinese and Korean, which are not screened yet), clean names 32.8%. |
What failed, and why
The company legal form decided the match. The scorer treated words such as a legal form as part of a company's name, so adding or dropping one moved the similarity enough to lose the match. The change prepared for this compares names without the legal form at either end while still requiring it for an exact match.
Typos and particles inside aliases retrieved nothing. Fuzzy retrieval searched each entry's main name only. Lists publish many names as aliases, including every language version of a name, so a one-letter change inside an alias found no candidate at all. The change prepared for this adds an indexed search over aliases.
Unresolved answers. Most unresolved clean names carried a weak candidate the engine would neither clear nor alert on. That is the cautious direction, but it is workload, and it is why the clean-name result is inconclusive.
These changes were prepared on October 22, 2026 and are not measured in this report. Local measurements guided them; the next live run will measure them, and its results will be published beside these, including any regression.
Reproducing it
Every figure in this report is read from one constant transcribed from the verifier's report; intervals and pooled rates are recomputed on the page from the per-cell counts published as data. The case file is fixed and fingerprinted, and so is the raw served log (SHA-256 0ebc5782a42d25ea…). The runner needs only the public API and an account, so a bank's validator can rerun the test on its own account and compare answers case by case; contact us to arrange it. What would make this benchmark independent is a third party holding the test cases and publishing the results, as national testing laboratories do for other technologies. We would welcome that.
The benchmark page shows the same results interactively.
This report describes measurements and general regulatory context. It is not legal advice; confirm your obligations with a qualified adviser.
Limitations
- The engine measured is the engine as deployed on October 13, 2026. Changes deployed after the run (from October 17, 2026) and changes prepared on October 22 (legal forms, alias typos, identifier search, a review floor for sanctions candidates, input fail-safes) are not measured here; the next live run measures them.
- The run stopped after 29,002 of 37,543 cases when the test account's quota ran out; the last 582 were refused (HTTP 402), so 28,420 were answered, 1,057 of them screening unavailable. Suite F (PEP separation), Suite G (123 cases) and the 2,000 hard clean names never ran.
- Listed names come from two lists: OFAC (3,016) and UN (384); the listed-name suite has no EU or UK name.
- The clean names that ran are all 10,000 company names (GLEIF) but only 3,700 scored person names (ParaNames). Person names false-alert far more often (11.99% against 1.57% of resolved names), so the clean-name rate understates the full set. On the 13,700 names this run and the October 8, 2026 run both decided, false alerts rose from 197 to 213.
- An answer of political exposure only ("match") on a clean name is not counted as a false alert, as the benchmark's scorer specifies. The alert policy's text read literally counts the 134 such answers: 347 of 9,267, 3.74%. The policy text has not been updated to match its scorer.
- 546 answers were served from the API's cache rather than computed for this run.
- The pre-registration's run-identity section (its own freeze hash, plan, cache bypass) was never filled in. The rules themselves were committed on October 7, 2026, six days before the run.
- Self-administered. Verifex built the engine and the test. The figures were re-derived from the raw served log by a separate AI agent with no build context, which is a check and not independent validation; it ran inside a Verifex working session and disclosed that the product's operating rules were in its context.
- Several October fixes were developed using these test cases. A later run on the same cases measures production, but cannot show that those fixes generalise; that exposure is recorded in the benchmark's contamination log, and a fresh held-out set is needed.
- Clean names were checked against OFAC, UN, EU and UK only, while the test account's plan screened 41 lists. Some alerts counted as false may be genuine namesakes on other lists.
- Queries are mostly name-only. Real traffic often carries a date of birth or country, which can change decisions in both directions.
- Recall counts an alert, not yet that the alert names the right listed subject. That was not measured; some alerts in the altered-spelling suite may point at a different subject.
- Gates that allow zero misses are strict by design: an engine whose true miss rate is very low can still fail them by chance. We report the counts and intervals beside every verdict for that reason.
Data and code
Per-cell counts, intervals, gate verdicts and run facts are published below as JSON. The case files (which contain real designated names drawn from public lists) and the raw served log are held by Verifex; a bank's validator can ask for the test to be rerun on its own account. Publishing the full case files waits on a licence check of the list-derived names.