Although technologies for Augmented and Virtual Realities and immersive experiences like 360° videos have developed at increasing speeds in recent years, evaluation methods for these technologies are lagging behind. How do affective VR experiences influence our emotions? How can we use empirical methods to test whether a VR system is immersive? A team of researchers from the AI, Media and Democracy Lab, the Centrum Wiskunde en Informatica (CWI), and various other international institutions have developed new and suitable methodologies to approach these questions.
Most current assessment methods are conceptualised for 2D videos, but new approaches are necessary for emerging technologies such as 360° VR videos. They provide virtual and immersive experiences, occupying the entire vision of the viewer and allowing them to freely rotate their head to focus on objects of interest. This creates a more immersive experience and evokes a wider range of emotions than desktop videos, but complicates Quality of Experience (QoE) measuring as users either have to be ripped out of the immersion or delay reporting until after the experience. Developing tools to adequately address the QoE is necessary to optimise VR technologies and identify sources of user inconveniences such as simulator sickness.
In Subjective Evaluation of Visual Quality and Simulator Sickness of Short 360° Videos, the research team considered a cross-lab study involving 10 VR labs and over 300 participants in order to evaluate the accuracy of existing subjective ratings methodologies for VR video quality. They found that when it came to 360° videos, Absolute Category Ratings (ACR) and Degradation Category Ratings (DCR) were both useful metrics; they also found that quality ratings scaled down to the evaluation of 10-second video clips presented without or without audio content. These results have informed the ITU-T P.919 recommendation for evaluation metrics in the field of VR.
In CEAP-360 VR, researchers introduced the Continuous Physiological and Behavioral Emotion Annotation Dataset for 360 Videos, or CEAP-360VR—an approach that allows them to examine continuous emotional responses to affective VR videos without relying on distracting self-reports, which oftentimes interfere with immersion. The CEAP-360VR methodology collects behavioural and physiological information about a user, as well as momentary self-reports, discrete emotional ratings, and motion sickness indicators. The indicators the researchers used yielded consistent results across different users and metrics. Researchers hope that the annotated dataset provided by CEAP-360VR can be used to improve the quality of emotion-recognition algorithms, which sometimes struggle to account for dynamic changes in emotional expression.
