AI Product Success: A Blend of Metrics 📊🍸
Deciphering success in AI products is a multifaceted challenge. No solitary metric holds the complete solution. Instead, success emerges from a harmonious fusion of diverse metrics working together. When determining the readiness of a product or feature for launch, it is vital to consider all metrics strategically. There is no universal recipe that fits every scenario.
AI Product Metric Mixology 🍹
An ML model alone does not solve user pain points.
The motivation behind launching a feature or product lies in addressing specific user pain points. Merely relying on an ML model is not sufficient. The model must be seamlessly integrated into an experiential framework—a means for users to tangibly encounter the benefits of this intelligent technology. Consider, for instance, a recommendation widget on Spotify that suggests new songs to enrich playlists or a smart matching feature on Tinder that connects individuals based on i.e. shared hobbies. The fusion of the model with the user experience is the key to unlocking the true potential of this smart technology.
The complete picture: A combination of metrics
To truly understand an AI feature, relying on a single metric falls short. Instead, a holistic view emerges when multiple metrics intertwine. By examining a combination of metrics, we gain deeper insights and a comprehensive understanding of the AI feature's performance and impact.
AI Success = AI Proxy Metrics + Product Health Metrics + System Health Metrics
AI Proxy Metrics: Navigating the Indirect Impact
As an AI Product Manager, you may not have direct control over AI Proxy Metrics, but their significance cannot be overlooked. These metrics play a pivotal role in assessing trade-offs and making strategic decisions. They serve as a yardstick for evaluating the effectiveness of the underlying model.
System Health Metrics: Acknowledge, Understand, Act
While you may not hold direct responsibility for System Health Metrics, being informed about them is essential. These metrics pertain to the performance of the overall feature in the face of millions of users. Being aware of how the system behaves under such loads helps in ensuring its robustness and scalability.
Product Health Metrics: Your Realm of Influence
Product Health Metrics are squarely within your domain of responsibility. These metrics encompass vital aspects such as Engagement, Retention, Satisfaction, and more. As a PM, you are likely well-acquainted with these metrics and actively work towards optimizing them to enhance the success of your AI product.
What are Proxy Metrics?
In the realm of Machine Learning, Proxy Metrics play a significant role in gauging model accuracy. Referred to as "proxy" because they measure the performance of the model itself, these metrics differ from the ultimate goal of the product or feature.
Proxy Metrics only a part of measuring success of an AI product.
What are some examples of Proxy Metrics?
Proxy Metrics encompass various categories, with Accuracy and Precision being the most commonly employed ones. Additionally, Speed and Scalability metrics hold significant importance, especially as they directly impact user experience.
As an example, consider a feature that distinguishes between spam and non-spam emails in your inbox. When measuring the accuracy of this classification (focusing solely on the classification itself, not the entire experience), the metric of interest is the % of correct classifications made by the system, or in other words, what % of time the system gets things right.
True positive: When a prediction is correctly classified as spam
True negative: When a prediction is correctly classified as not spam
False positive: When a prediction is incorrectly classified as spam
False negative: When a prediction is incorrectly classified as not spam
What other Proxy Metrics should I know of?
Other AI proxy metrics include: objective function, mean absolute errors (MAE), root mean square error (RMSE) & specificity average (SSA). I won’t be covering these in my newsletter, but if you want to learn more I’m hosting an Advanced AI/ML Course for PMs on Maven very soon!
By considering all types of metrics above you will be able to get a somewhat complete picture of your AI feature’s success.
Stay tuned for next week’s post around goal setting for AI Products that is directly related to metrics.