Bank prediction: looking past accuracy

The number looked good

The task was to predict whether a customer would subscribe after a bank marketing campaign.

At first, this looked like a normal classification problem. Train a few models, compare their accuracy, choose the highest one.

The problem was that most customers did not subscribe. That means a model could predict "no" almost all the time and still end up with an accuracy score that looked impressive.

Once I noticed that, accuracy stopped being a very interesting metric.

Asking a better question

The real question was not "How often is the model correct?"

It was closer to "How well can the model distinguish between customers who are likely to subscribe and customers who are not?"

That pushed me toward looking at ROC-AUC and the behaviour of the classifier across different thresholds. The final experiments reached roughly 0.949 ROC-AUC.

More importantly, I understood why that number was more informative for this particular problem than plain accuracy.

Metrics depend on the problem

This sounds obvious when written down, but it was not obvious to me when I first started doing ML.

There is no universally good metric. If the dataset is imbalanced, accuracy can hide a bad model. In another project, false negatives might matter much more than false positives. Somewhere else, probability calibration could matter more than either.

The evaluation has to reflect what the prediction is actually being used for. That lesson ended up coming back later in my medical AI work, where balanced accuracy made much more sense for staging than raw accuracy.

Why I still keep this project

This is nowhere near the most technically difficult project on my portfolio anymore.

I keep it because it marks a change in how I approached machine learning.

Before this, I mostly cared about getting the score higher. Afterwards, I started asking what the score was actually measuring.

That is a much better question.