Think about a city. Any city.
You probably know a neighborhood where police cars are more common than traffic lights. Usually, impoverished areas of the city. Usually, areas your mom taught you not to visit.
The police are there because there’s a lot of crime. They aren’t patrolling my street; they are laser-focused on those neighborhoods.
Let’s now build a machine learning model that uses all crime information to predict where the following arrests will happen. Not surprisingly, heavily patrolled neighborhoods report more crimes. This influences the model’s predictions, which will redirect more police there, which will lead to more arrests, which will make the model double down on the same areas.
An infinite loop that we can’t get out of.
Crime may increase elsewhere, but it doesn’t matter. No police, no arrests. Our model is entirely biased towards specific neighborhoods, and it’s incapable of breaking the cycle that it helped create.
Our company started selling ten new products.
We create a quick marketing campaign to promote all of them. Even better, we build a machine learning model that uses the sale results to predict which products will perform best and distribute the marketing budget.
After a week, three of the products rose to the top. The model starts allocating more budget to advertise them, which causes even more sales of those three products. As the other seven receive less funding, it’s hard for them to get any traction.
Sort of a self-fulling prophecy. A biased model stands in the way of the unlucky seven products. Their faith has been sealed.
Hiring the best of the best
Everyone wants to hire the best talent out there. How about building a model that predicts what type of people is more likely to be successful?
Unfortunately, technology is a male-dominated field. Just by having a larger population, it’s likely that more men dominate the list of successful employees. This isn’t even counting any biases that we may introduce when defining the criteria to consider somebody “successful.”
Little by little, the model absorbs the status quo and recommends more of the same. What do you think will happen?
Positive feedback loops
Unfortunately, I didn’t make up any of these stories. They are real examples of a phenomenon called “positive feedback loops.” Funny enough, none of these seem to offer anything positive at all, but you already know about our track history naming things.
We are making progress
, but there’s still a lot of work to do in the machine learning community to find solutions to these problems. We need to keep working on our understanding of biases and fairness in our models and specifically avoiding or mitigating their consequences.
A long way to go still, but everything starts with you, your process, and how serious you are about putting these issues front and center on everything you do.