It should be clear now that we want our model to minimize the number of false-negative mistakes because they are 5 times more expensive than false-positive mistakes. In other words, we want to maximize the recall of this model.
But there’s more!
Can we use the cost of each mistake to optimize the results of the model?
Changing the result of the model
We have a binary classification model.
Let’s imagine that the model returns a probability score attached to each answer. Whenever it says that a component is about to break, it also returns how confident on that answer it is. The complement of that probability corresponds to the opposite answer.
Can we use this probability to optimize the model?
Let’s imagine an example where the model looks at a picture and decides the result is “Negative” with a 57% probability. This would be the correct answer if we don’t want to do anything else, but we can get fancier than that:
- If the model returns “Negative,” there’s a 43% chance the result is actually “Positive,” so the potential cost of returning “Negative” is 0.43 × $5,000 = $2,150.
- If the model returns “Positive,” there’s a 57% chance the result is actually “Negative,” so the potential cost of returning “Positive” is 0.57 × $1,000 = $570.
So yes, the model thinks the result is “Negative,” but we are better off returning “Positive” if we want to reduce the potential costs of running it.
Pretty cool, huh?
I took you down a long rabbit hole to illustrate how things work, but a simpler way to have the model make final decisions is by adjusting the confidence threshold according to the cost of every mistake.
This technique is part of a family of approaches known as Cost-Sensitive Learning, and it’s instrumental in real-world scenarios where money is an important factor.
Here is an article
going deeper into cost-sensitive learning, specifically for imbalanced datasets.
Spend some time thinking about this, and I promise it will change the way you approach many of the problems you’ll face out there.