“Meet New Methods” is a series of interdisciplinary sessions hosted by the AI, Media & Democracy Lab at the Institute of Advanced Studies (IAS). It aims to multiply frames of reference and encourages the exchange of methodological approaches by connecting researchers of various disciplines. The intersection and combinations of cross-disciplinary methodologies allow for critical perspectives and thinking outside the box, opening doors to advance research in the field of AI.
In the third installment of “Meet New Methods”, we invited IAS fellow Jolien Francken and our own PostDoc Kimon Kieslich to discuss the methodologies of their respective research. Jolien Francken is concerned with the measurement problem, which arises both within disciplines and in interdisciplinary research. It describes the problem that arises when defining and translating key concepts and terms across various contexts, which becomes especially apparent in research that crosses disciplinary lines. Situated in Medicine, Cognitive Neuroscience, and Philosophy, Jolien aims to smooth out interpretational and translational bumps created by the measurement problem. Kimon Kieslich uses various methodologic approaches in his research and thus has encountered the measurement problem, although he might not define it as such. He discusses how to approach translating discipline-specific methodologies to fit interdisciplinary research questions in his projects, with specific regard to fairness of AI.
The Measurement Problem
Jolien Francken begins with one of many definitions of measurement: the correct assignment of numbers to physical variables, or the assignment of values to categories in a systematic way. She stresses that, even though measurement is fundamental to modern science, it is no easy feat to conceptualise it. We need to as what we are measuring, how we are measuring, and why we are measuring. This seems natural enough when attempting to measure the height of an individual to track their growth over the years but becomes complicated when the value changes from a numerical answer to a categorical answer, which arises when attempting to measure intangible things such as happiness or inequality. Are the objects of measurement observable? Are they social properties, or mental? These questions are especially relevant when considering the impact of measurement on classification, ideas of normativity, and authority. Jolien points out that it is crucial to consider measurement in specific contexts, as it is “only right for a certain purpose”.
Measuring Algorithmic Fairness
In his research on algorithmic fairness, Kimon encountered the very complications of measurement that Jolien studies. Kimon laid out the many facets and dimensions of fairness in decision making systems which could be classified between the categories of factual fairness and perceived fairness. Factual fairness is made up of the coding behind the system, while perceived fairness describes the conceptions about the system and its outcome. But fairness can be conceptualised in various ways, and received differently by individuals. Kimon stresses that context is highly relevant for assessments of fairness, and that there is no mechanical way to decide which model is the “fairest”; it is up to humans to judge. Overall, there is a conceptual and theoretical need for clarification and clear definitions applicable to different cases, which is benefitial for specificity, but makes comparison of empirical findings difficult.
The problem of measurement arises across disciplines and, as both Jolien and Kimon have illustrated, is crucial to address and conceptualise accordingly. We express our gratitude to the speakers for joining us for this session of Meet New Methods, and look forward to the upcoming installments.
About the Speakers
Jolien Francken is an Assistant Professor in Philosophy of Mind and Neuroscience and a fellow at IAS. She obtained her PhD ad the Donders Institute for Brain, Cognition and Behavior in Nijmegen, and worked as a post-doctoral researcher and senior lecturer in Neurophilosophy at the University of Amsterdam, the Conscious Brain Lab, and Institute for Interdisciplinary Studies. Her research aims to bridge gaps left by the measurement problem in cognitive neuroscience and advance the translation of its experimental findings.
Kimon Kieslich is a postdoctoral researcher and member of the AI, Media and Democracy Lab. He is involved in the Anticipating AI impact in a Diverse Society project at the Institute for Information Law at the University of Amsterdam, with a current research focus on developing methods for assessing the societal impact of AI systems and regulations. His PhD project at the Institute for Communication Studies at the University of Hohenheim dealt with the public perception of AI ethics.