SyncAI.news, a Varaisys broadcasting
How to Calibrate Your Classifier’s Probabilities in Scikit-learn
SO

Shittu Olumide

· 2 min read

EngineeringKDnuggets

How to Calibrate Your Classifier’s Probabilities in Scikit-learn

Your model says 0.9. You treat it as a 90% chance, set a threshold, and build a decision on top of it. Then you check the predictions the model scored near 0.9 and find that noticeably fewer of them were right. Nothing is broken. The model ranks cases well, and it is still a poor judge of how sure it should be.

That gap between a score and a real probability is what calibration closes. In this tutorial, you will train two scikit-learn classifiers on a real dataset, expose their miscalibration with a reliability diagram and a Brier score, and fix it with Platt scaling and isotonic regression. By the end, you will know how to calibrate classifier probabilities in scikit-learn and how to pick the right method for your data.

What Calibration Means

A well-calibrated classifier lets you read predict_proba as a confidence level. In the words of the scikit-learn user guide, among the samples that get a score close to 0.8, about 80% should actually belong to the positive class.

Accuracy and the area under the receiver operating characteristic curve (ROC-AUC) cannot tell you whether that holds. Both look at labels or rankings, not at whether 0.8 means 80%. A model can score well on both and still be wildly over- or under-confident.

Some models are more prone to this than others. The guide notes that logistic regression tends to be well calibrated by itself. Gaussian naive Bayes pushes probabilities toward 0 and 1, random forests tend to avoid the extremes, and maximum-margin methods like support vector machines (SVMs) produce a sigmoid-shaped distortion. You will see two of these behaviors yourself below.

Setup and Data

Install the libraries and check your version:

pip install -U scikit-learn matplotlib numpy
python -c "import sklearn; print(sklearn.__version__)"

Create calibrate.py and start with this:

Showing the Problem

Now train two models that are known to misbehave in opposite ways. Add this to the same file:

Next come three small helpers you will reuse for every comparison:

Original source

This story was published by KDnuggets and written by Shittu Olumide. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on kdnuggets.com

Similar News