The Values to Metrics Toolkit
A card-based workshop tool that helps news organizations translate journalistic and democratic values into measurable design choices for recommender systems.
By Sanne Vrijenhoek, Sara Spaargaren, and Daphne de Vries
What is it for?
Recommender systems increasingly decide which news stories people get to see. Yet these systems are almost always trained and evaluated on one thing: how well they predict clicks and engagement. But engagement isn’t the only thing that matters to a news organization. What about fairness, diversity, or editorial independence, the values organizations actually want to stand for? What do such terms mean in practice, and how would you even measure them?
The Values to Metrics toolkit helps journalists, researchers, legal teams and engineers have that conversation together, and bridges the gap between abstract values and concrete, implementable metrics.
How does it work?
Following a step-by-step structure, participants first discuss what objectives the recommender needs to fulfill: what needs to be recommended, for whom, and why. Then, they design metrics that express whether the objective has been achieved: what needs to be measured, where, and what value is expected out of these measurements. At each step, the discussions are supported by cards that provide context and examples identified through our own research. For more details and examples, see the Facilitator’s Guide.
The workshop is ideally run as a 2–3 hour workshop with 3–10 participants with different roles, ranging from domain experts, to technical and legal teams, to strategic decision makers.
In short, the toolkit does not prescribe how to design or evaluate your recommender system, but provides a framework to define your own goals and strategies, and create understanding and support for those within different departments of the organization.
Background
Researcher Sanne Vrijenhoek and lab manager Sara Spaargaren worked with Daphne de Vries of Bureau Merkwaardig to develop a practical toolkit that translates research carried out at the AI, Media & Democracy Lab into an accessible workshop format, supported by a card system, information design and visual identity.
The toolkit became a playful, accessible medium for effective communication of core values that should be continuously considered in developing metrics for content recommender systems.
Team
Sanne Vrijenhoek
Sanne is a tenure track researcher at the Netherlands Institute for Mathematics and Computer Science and the AI, Media & Democracy Lab. Her main research focus is on translating normative notions of diversity into concrete concepts that can be used to inform recommender system design.
Sara Spaargaren
Sara is the manager of the AI, Media & Democracy Lab and the central point in our network of researchers, partners and stakeholders. She is the organizing force behind projects like this one, where she provided oversight from start to finish and valuable practical input.
Daphne de Vries
At Bureau Merkwaardig, Daphne specializes in translating complex research into accessible design solutions. In close collaboration with Sanne and Sara, she helped transform the research into a structured, card-based toolkit, making the methodology accessible, intuitive and easy to facilitate.
Interested in a session?
If your organization is interested in having a workshop session facilitated by one of our researchers, reach out to us via email or contact Sanne Vrijenhoek directly.
Try it for yourself!
The full toolkit is freely available as a PDF, including facilitation instructions.
The complete card deck, ready to print
How to run the workshop, step by step
Necessary for the workshop
Contributors: Kornelija Gruodyte, Lien Michiels, Savvina Daniil, Pascal Wiggers, Maarike Harbers, Natali Helberger
This project is supported by the Impact Fund for Research of the Amsterdam Law School (University of Amsterdam).
This work is licensed under CC BY-NC-ND 4.0 
