Skip to main content
A credible benchmark fixes the task, available evidence, population, outcome, and operating conditions before comparing models. The public-data results contain completed retrieval measurements. The task catalogue defines document, transaction, entity and risk evaluations; the frontier protocol fixes comparison conditions. No AMLOK-versus-frontier or AMLOK-versus-vendor result is reported here.

​
Benchmark tracks

​
Comparator matrix

Record the exact model identifier/revision, access date, parameters, reasoning budget, context limit, tool access, retrieval settings, output schema, pricing basis, rate limits, and retry policy. A family name alone is insufficient to reproduce a result. Limited-access models are not treated as available simply because a vendor announced them.

​
Two fair comparisons

Run a matched-evidence comparison in which every method receives the same as-of evidence and permitted calculator/retrieval tools. Separately run an end-to-end system comparison that includes each system’s normalization, graph, and retrieval. Attribute differences to the correct level. Do not give AMLOK extra sources and describe the resulting gain as a model-architecture improvement. Test prompt-only, retrieval-assisted, and tool-assisted language models separately. Log invalid outputs, refusals, timeouts, rate-limit failures, and retries. Report quality and coverage jointly. Restrict real personal data sent to hosted models to specifically authorized, contract-compatible evaluation environments.

​
Splits and analysis

Use development, selection, calibration, and final test partitions with no related-subject overlap. Include an out-of-time test and an institution-held-out test; report them separately rather than assuming either proves the other. Freeze cohort eligibility and source-availability rules before test access. Keep final labels inaccessible to prompt authors and model developers during selection. Report point estimates with uncertainty. Use paired borrower-level resampling for model comparisons; use cluster-aware or hierarchical resampling for repeated borrowers and institutions. Report both entity-weighted and institution-weighted results where useful. An interval based on one institution does not establish national generalization. Use precision-recall metrics with stated prevalence for rare events. Average precision is not an unspecified “accuracy.” Show reliability plots, Brier score, and calibration slope/intercept for probability outputs. Select thresholds on development/calibration data only. Compare operating points, not just ranking metrics. The calibration literature motivates separately testing probability reliability.

​
Sample size and multiple comparisons

Determine sample size from the primary endpoint, event prevalence, clustering, desired precision, and practical effect size. For illustration, estimating sensitivity of 0.80 within roughly ±0.05 under an independent binomial approximation requires about 246 positive outcomes; this is not 246 total loans and does not account for clustering or subgroup analysis. Use simulation-based power analysis for the actual design. Pre-register one primary endpoint per study and identify exploratory comparisons. Control multiplicity where claims rely on many tests. Publish negative results and seed variability. Do not choose only the best seed, slice, prompt, or threshold after seeing the holdout.

​
Required ablations

Bank-only versus additional source families; graph versus no graph; pretraining versus scratch; text versus structured encoding; native French labels versus translation; public context versus none; missing-source robustness; temporal truncation; weak-label versus adjudicated supervision; and narrative-only versus calculator-backed output. Hold all other conditions constant or explicitly report the differences.