The Values to Metrics Toolkit
A card deck for evaluating recommender systems on more than just accuracy.
By Sanne Vrijenhoek, Sara Spaargaren, and Daphne de Vries
What is it for?
Most recommender systems are only trained and evaluated on how good they are at predicting clicks and engagement. But how do you know whether the recommender system also achieves its societal goals, such as fairness, novelty or diversity? What do such terms mean, and what would you even need to measure?
The Values to Metrics toolkit facilitates collaboration between departments in an organization to bridge the gap between high-level concepts and implementable metrics.
Discuss
First, participants discuss what objectives the recommender needs to fulfill: what needs to be recommended, for whom and why.
Design metrics
Then, they design metrics that express whether the objectives have been achieved: what needs to be measured, where, and what value you expect out of these measurements.
Examples & Context
The discussions are supported by cards that provide examples and context as identified in our own research.
Design process
Starting from a slide deck on the lack of alignment of recommender system evaluations with principles beyond just growth, PhD researcher Sanne Vrijenhoek and lab manager Sara Spaargaren worked with Daphne de Vries of Bureau Merkwaardig to turn ideas from research into a more approachable and directly applicable tool.
As informed by Sanne’s PhD research on the role of AI in news recommender systems and their diversity, the set of cards became a playful, accessible medium for effective communication of core values that should be continuously considered in developing metrics for content recommender systems.
“This toolkit helps translate the needs of journalists and editors to the technical side of news distribution”
– Sanne Vrijenhoek
In short, the toolkit does not prescribe how to design or evaluate your recommender system, but provides a framework to define your own goals and strategies, and create understanding and support for those within different departments of the organization.
Try it for yourself!
The set of cards is freely available for download as a PDF. Instructions for use & implementation included!
Contributors: Kornelija Gruodyte, Lien Michiels, Savvina Daniil, Pascal Wiggers, Maarike Harbers, Natali Helberger