Product thinking for Machine Learning

Or why understanding math shouldn't be the skill product folks bring to the ML table

Machine Learning is a powerful tool to supercharge your product - whether you’re building recommender systems and personalisation modules, or automating repeated processes, incorporating some ML in your product will offer great rewards.

Machine Learning is also a black box for most product folks - it’s often perceived as too complicated or too technical, to the point that it gains the reputation of being some kind of voodoo magic. I’ve been asked many times to ‘do ML magic’ in a product to make it better. But it’s not that complicated.

For a product person, ML is all about understanding probabilities and getting creative with them. Think of an ML product as a probability generator - you ask it a question, and it gives you an answer with a probability. If you ask - will my user be interested in buying a toothbrush today, ML models will tell you something like - yes, 60% chance. If you’re interested in using rather than building these models, this is all you need to understand. This basic concept extrapolates to user segments too - you could ask an ML model to create a segment of users with a high chance of engaging with a specific post, or listening to a genre of music, or being interested in a new product line. The model will group users with highest probabilities together and give you a segment.

Bear in mind that this is a naive explanation of ML, and serves more as a mental model than a definition. Sophisticated mathematics drives most ML, and actually getting to a well performing model comes with tons of challenges. I’ve also excluded Natural Language Processing entirely from this naive simplification - stuff like translation ML works on different paradigms. But all of that is irrelevant for building a product. If you’re only interested in consuming the output, all you need to understand is probabilities. Your job as a product person is to exploit these probabilities creatively for the best outcome for your users.

I keep highlighting creativity, and it seems counterintuitive for a topic so logical and mathematical as Machine Learning. But I’ll demonstrate why you need to be creative with this example:

Let’s say you’re trying to predict, in an e-commerce marketplace, if a customer is going to cancel an order. You put your best data scientists to the task, and they come up with a prediction model. Let’s say they came up with a model that was right 100% of the time. As the product person, you would probably cancel the order before it ships to save on operating costs.

Now if they came up with a model that was right 90% of the time - you would probably call up these customers who are 90% likely to cancel and make sure they really want the product before shipping. The cost of calling is much less than the cost of ship<>return. Why couldn’t you do this for all customers then? Because the cost of calling everyone becomes an expense - if you have a return rate of 10%, then you’re calling 90% of your customers for no reason! Having a prediction made you selective about who you call and maximised your ROI.

What if you got a model that was right 60% of the time? You want something that’s not as expensive as calling, but still catches some of the orders about to be cancelled before shipping. Something like an email or notification to confirm shipping date with a visible cancellation button might be a solution. It gives a gentle nudge to potential customers who are likely to cancel. Why can’t you do this for everyone? You’re adding an extra step of customer effort - checking an email - which breaks the solid UX you’ve built so far. You’re also encouraging buyers remorse and forming a ‘cancel later’ habit.

Let’s take this to the extreme. What if you get a model that was right 20% of the time? You would probably use it as an input to your recommendation logic, making sure that customers don’t get recommended products they are likely to return. Your interventions by correctness of prediction look like this:

What we’ve discussed so far, the ‘model is right x% of the time’, is called precision. Another helpful concept to be aware of is recall. While precision tells you how many times a prediction is right, recall tells you how many such predictions have been made. When data is insufficient or when decision making is not clear, an ML model will not give you an opinion. The number of times this happens defines the recall of the model. Simply put, recall is the % cases for which an ML model has an opinion. Consider the 90% precision ML model - let’s assume it has a recall of 10%. This means, you will have a prediction for 10 out of 100 users, and of those 10 predictions, 9 will be right.

As a general rule of thumb, the higher the precision, the lower the recall and vice versa. This makes logical sense - the more certain you are about an opinion, the fewer the opinions you will have. A standard precision/ recall tradeoff looks like this orange line:

As you can see, at close to 100% precision, recall is less than 10%. A model with a high precision and a high recall is the holy grail. Data Scientists will try their best to give you this.

As a product person, you have very little influence on the precision AND recall of a model. However you still have a choice. Every model performs at a constant precision + recall, but you are free to pick this combination. This allows us to use the same model differently for different use cases.

In our e-commerce example, instead of picking just one approach, you can pick multiple. For users where precision is > 90% but recall is <10%, we can decide calling to confirm order is the best approach. This will give you some cost savings - you will avoid shipping 10% of the orders that are bound to be returned. Next, for customers where precision >60%, let’s assume recall is at 50% and decide to send an email or notification. These two approaches will catch 50% of the orders you would have returned. Finally, with a 20% precision and an 80% recall, you would use the model as an input to your recommender systems. You have now made one or another product intervention for 80% of the users who are likely to cancel after an order is shipped. The setup looks like this:

There you have it! By using precision and recall smartly, you’ve reduced the number of returns post shipment by a hypothetical 80%. Factor in all the things that would go wrong, all the times your guess doesn’t work, or your customer doesn’t react the way you expect, or bugs, or the randomness of the universe. Realistically, you would have about a 30-40% reduction in cancellations post shipment. This in itself is a large number, but consider that you do this at minimal additional cost to business, and minimal damage to customer experience.

So you see, we’ve only used two mathematical concepts for our entire journey - probabilities and percentages. We use a lot of product thinking, out of the box solutioning, and creativity! The most important role of a product person in a team is to define the problem and come up with creative ways to exploit the ML model to its maximum potential value. The next time you wish you had a way to predict an outcome, go forth and give ML a shot.

If you haven’t already, consider subscribing to this newsletter for more product stuff!