The use of mixed reality technology for the objective assessment of clinical skills: a validation study
File(s)s12909-022-03701-3.pdf (1.69 MB)
Published version
Author(s)
Type
Journal Article
Abstract
Background
Mixed Reality technology may provide many advantages over traditional teaching methods. Despite its potential, the technology has yet to be used for the formal assessment of clinical competency. This study sought to collect validity evidence and assess the feasibility of using the HoloLens 2 mixed reality headset for the conduct and augmentation of Objective Structured Clinical Examinations (OSCEs).
Methods
A prospective cohort study was conducted to compare the assessment of undergraduate medical students undertaking OSCEs via HoloLens 2 live (HLL) and recorded (HLR), and gold-standard in-person (IP) methods. An augmented mixed reality scenario was also assessed.
Results
13 undergraduate participants completed a total of 65 OSCE stations. Overall inter-modality correlation was 0.81 (p=0.01), 0.98 (p=0.01) and 0.82 (p=0.01) for IP vs. HLL, HLL vs. HLR and IP vs. HLR respectively. Skill based correlations for IP vs. HLR were assessed for history taking (0.82, p=0.01), clinical examination (0.81, p=0.01), procedural (0.88, p=0.01) and clinical skills (0.92, p=0.01), and assessment of a virtual mixed reality patient (0.74, p=0.01). The HoloLens device was deemed to be usable and practical (Standard Usability Scale (SUS) score = 51.5), and the technology was thought to deliver greater flexibility and convenience, and have the potential to expand and enhance assessment opportunities.
Conclusions
HoloLens 2 is comparable to traditional in-person examination of undergraduate medical students for both live and recorded assessments, and therefore is a valid and robust method for objectively assessing performance. The technology is in its infancy, and users need to develop confidence in its usability and reliability as an assessment tool. However, the potential to integrate additional functionality including holographic content, automated tracking and data analysis, and to facilitate remote assessment may allow the technology to enhance, expand and standardise examinations across a range of educational contexts.
Mixed Reality technology may provide many advantages over traditional teaching methods. Despite its potential, the technology has yet to be used for the formal assessment of clinical competency. This study sought to collect validity evidence and assess the feasibility of using the HoloLens 2 mixed reality headset for the conduct and augmentation of Objective Structured Clinical Examinations (OSCEs).
Methods
A prospective cohort study was conducted to compare the assessment of undergraduate medical students undertaking OSCEs via HoloLens 2 live (HLL) and recorded (HLR), and gold-standard in-person (IP) methods. An augmented mixed reality scenario was also assessed.
Results
13 undergraduate participants completed a total of 65 OSCE stations. Overall inter-modality correlation was 0.81 (p=0.01), 0.98 (p=0.01) and 0.82 (p=0.01) for IP vs. HLL, HLL vs. HLR and IP vs. HLR respectively. Skill based correlations for IP vs. HLR were assessed for history taking (0.82, p=0.01), clinical examination (0.81, p=0.01), procedural (0.88, p=0.01) and clinical skills (0.92, p=0.01), and assessment of a virtual mixed reality patient (0.74, p=0.01). The HoloLens device was deemed to be usable and practical (Standard Usability Scale (SUS) score = 51.5), and the technology was thought to deliver greater flexibility and convenience, and have the potential to expand and enhance assessment opportunities.
Conclusions
HoloLens 2 is comparable to traditional in-person examination of undergraduate medical students for both live and recorded assessments, and therefore is a valid and robust method for objectively assessing performance. The technology is in its infancy, and users need to develop confidence in its usability and reliability as an assessment tool. However, the potential to integrate additional functionality including holographic content, automated tracking and data analysis, and to facilitate remote assessment may allow the technology to enhance, expand and standardise examinations across a range of educational contexts.
Date Issued
2022-08-23
Date Acceptance
2022-08-16
Citation
BMC Medical Education, 2022, 22
ISSN
1472-6920
Publisher
BioMed Central
Journal / Book Title
BMC Medical Education
Volume
22
Copyright Statement
© The Author(s) 2022. This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
License URL
Sponsor
Defence Science and Technology Laboratory (DSTL)
Grant Number
DSTLX1000150232
Subjects
Social Sciences
Education & Educational Research
Education, Scientific Disciplines
Medical education
Assessment
Mixed reality technology
Innovation
HOLOLENS
Assessment
Innovation
Medical education
Mixed reality technology
Augmented Reality
Clinical Competence
Educational Measurement
Humans
Prospective Studies
Reproducibility of Results
Students, Medical
Technology
Humans
Prospective Studies
Reproducibility of Results
Educational Measurement
Clinical Competence
Students, Medical
Technology
Augmented Reality
Medical Informatics
1117 Public Health and Health Services
1302 Curriculum and Pedagogy
Publication Status
Published
Article Number
ARTN 639