The Values to Metrics Toolkit
A card deck for evaluating recommender systems on more than just accuracy.
By Sanne Vrijenhoek, Sara Spaargaren, and Daphne de Vries
What is it for?
Most recommender systems are only trained and evaluated on how good they are at predicting clicks and engagement. But how do you know whether the recommender system also achieves its societal goals, such as fairness, novelty or diversity? What do such terms mean, and what would you even need to measure?
The Values to Metrics toolkit facilitates collaboration between departments in an organization to bridge the gap between abstract concepts and implementable metrics.
How does it work?
Following the step-by-step structure, participants first discuss what objectives the recommender needs to fulfill: what needs to be recommended, for whom, and why. Then, they design metrics that express whether the objective has been achieved: what needs to be measured, where, and what value is expected out of these measurements. For each step, the discussions are supported by cards that provide context and examples identified through our own research.
Design process
Starting from a slide deck on the lack of alignment of recommender system evaluations with principles beyond just growth, PhD researcher Sanne Vrijenhoek and lab manager Sara Spaargaren worked with Daphne de Vries of Bureau Merkwaardig to develop a practical toolkit. The research was translated into an accessible workshop format, supported by a card system, information design and visual identity.
As informed by Sanne’s PhD research on the role of AI in news recommender systems and their diversity, the toolkit became a playful, accessible medium for effective communication of core values that should be continuously considered in developing metrics for content recommender systems. It is typically run as a 2–3 hour workshop with 4–8 participants across editorial, legal, and technical roles.
“This toolkit helps translate the needs of journalists and editors to the technical side of news distribution”
– Sanne Vrijenhoek
In short, the toolkit does not prescribe how to design or evaluate your recommender system, but provides a framework to define your own goals and strategies, and create understanding and support for those within different departments of the organization.
Team
Sanne Vrijenhoek
Sanne is a tenure track researcher at the Netherlands Institute for Mathematics and Computer Science and the AI, Media & Democracy Lab. Her main research focus is on translating normative notions of diversity into concrete concepts that can be used to inform recommender system design.
Sara Spaargaren
Sara is the manager of the AI, Media & Democracy Lab and the central point in our network of researchers, partners and stakeholders. She is the organizing force behind projects like this one, where she provided oversight from start to finish and valuable practical input.
Daphne de Vries
At Bureau Merkwaardig, Daphne specializes in translating complex research into accessible design solutions. In close collaboration with Sanne and Sara, she helped transform the research into a structured, card-based toolkit, making the methodology accessible, intuitive and easy to facilitate.
Try it for yourself!
The full toolkit is freely available as a PDF, including facilitation instructions.
The complete card deck, ready to print
How to run the workshop, step by step
Necessary for the workshop
Contributors: Kornelija Gruodyte, Lien Michiels, Savvina Daniil, Pascal Wiggers, Maarike Harbers, Natali Helberger
This project is supported by the Impact Fund for Research of the Amsterdam Law School (University of Amsterdam).
This work is licensed under CC BY-NC-ND 4.0 
