Can AI Spot Data Leakage Before Your Model Hits 99% Accuracy?

Every data scientist knows the feeling: a model comes back with 99% accuracy, and instead of celebrating, we get suspicious. Nine times out of ten, the cause is leakage: some piece of information in the training data that would not actually be available at prediction time.

We wanted two real numbers. First, how dramatic is that accuracy jump on real data, not a toy example? Second, if we describe the pipeline to an AI model before anyone runs it, with no hint that anything is wrong, does it catch the leak, and does the answer change depending on which model we ask?

So we took four real data projects, planted a different, well-known leakage pattern into each one, trained a real model with and without the leak, and recorded the actual numbers.

Then, separately, we described each pipeline in plain terms and asked four Claude models, from fastest to strongest, what they would flag before trusting the result.

Setting Up the Experiment

Each leak takes a different route into the pipeline: a feature that mechanically defines the label, a feature built with future information, a split that ignores relational structure, and a preprocessing step done in the wrong order.

For the AI side, we tested Claude Haiku 4.5, Claude Sonnet 5, Claude Opus 5, and Claude Fable 5.1, across the speed and capability range we have access to.

We are naming exactly which models we tested and not implying a cross-vendor comparison, since that is not what we ran. Each model got a fresh session with zero memory of the others, a closed-book description of the pipeline, features, or code, and no ability to run anything. None were told a leak existed.

Case 1: The Feature That Already Knows the Answer

For this case, we are using datasets from the project Prediction of Stock Price Direction.

The task: predict whether Amazon's closing price lands higher than that same day's opening price. Here, Label equals 1 if Close is greater than Open, 0 otherwise.

Comparing Leakage and Non-Leakage Performance

Transactions table: