Fraud detection on a public dataset: the metric that matters when 0.17% of rows are fraud
Rule-based fraud systems catch last year's fraud and drown the analysts in false positives. Machine learning is the obvious next step, and every vendor's slide shows a ROC-AUC near 0.98. We built a proof of concept — as research, on a public dataset, not for a client — to find out what that number actually means when fraud is a fraction of a percent of the data.
What we built
The dataset is the public European card-transaction set from Kaggle: real, anonymised transactions, with 0.1727% labelled fraud. We held out a test set of 56,962 rows and trained four classifiers on the rest — logistic regression as the baseline, then XGBoost, CatBoost, and LightGBM. No graph models, no real-time scoring, no investigation assistant: four models, one question.
That question was which metric to believe.
What we measured
On the held-out test set:
| Model | ROC-AUC | PR-AUC |
|---|---|---|
| XGBoost | 0.9804 | 0.8770 |
| CatBoost | 0.9768 | 0.8757 |
| Logistic regression | 0.9736 | 0.7222 |
| LightGBM | 0.9529 | 0.8518 |
Read the first column and every model looks excellent. Read the second and the picture changes: the baseline that scores 0.97 on ROC-AUC drops to 0.72 on precision-recall, and the gap between the boosted models and the baseline — small on the metric everyone quotes — is large on the one that describes the analyst's day.
The finding
ROC-AUC flatters rare-event models. When 99.83% of rows are legitimate, a model can rank almost everything correctly and still send the fraud team a queue that is mostly wrong. ROC-AUC rewards that ranking. Precision-recall AUC asks the question the team actually has — of the alerts I raise, how many are fraud, and how much fraud do I miss — and it is the number to put on the slide.
The two boosted models are close, and the choice is operational. XGBoost and CatBoost are within a rounding error of each other on both metrics. Which one ships depends on latency, tooling, and who maintains it — not on the leaderboard. (Our own README named CatBoost the winner; the notebook's numbers say XGBoost. The notebook is the source; the README is being corrected. That is the kind of discrepancy an evaluation set exists to catch.)
The baseline is worth keeping. Logistic regression at 0.72 PR-AUC is the model you deploy in week one while the boosted one earns its place, and the one the compliance team can read.
What a production system would still need
Everything this POC deliberately didn't do. Your own transactions, under your own controls, with the class balance you actually have. Features from relationships — accounts sharing devices or payout addresses — that a single-row classifier cannot see, which is where graph features earn their cost. Real-time scoring for payments, an audit trail a regulator can read, monitoring for the day fraud adapts to the model, and above all a queue tuned with the analysts who will work it: ranking and explanation decide whether a model gets used, and no offline metric measures that.
This use case sits further from the software most companies run than the others we prototype. It taught us something general anyway: pick the metric that describes the person's day, not the one that looks best on the slide.