IRUS Total

Very-short-answer questions: reliability, discrimination and acceptability

File Description SizeFormat 
Very short answer questions reliability, discrimination and acceptability.pdfAccepted version513.02 kBAdobe PDFView/Open
Title: Very-short-answer questions: reliability, discrimination and acceptability
Authors: Sam, AH
Field, SM
Collares, CF
Van der Vleuten, CPM
Wass, VJ
Melville, C
Harris, J
Meeran, K
Item Type: Journal Article
Abstract: Context Single‐best‐answer questions (SBAQs) have been widely used to test knowledge because they are easy to mark and demonstrate high reliability. However, SBAQs have been criticised for being subject to cueing. Objectives We used a novel assessment tool that facilitates efficient marking of open‐ended very‐short‐answer questions (VSAQs). We compared VSAQs with SBAQs with regard to reliability, discrimination and student performance, and evaluated the acceptability of VSAQs. Methods Medical students were randomised to sit a 60‐question assessment administered in either VSAQ and then SBAQ format (Group 1, n = 155) or the reverse (Group 2, n = 144). The VSAQs were delivered on a tablet; responses were computer‐marked and subsequently reviewed by two examiners. The standard error of measurement (SEM) across the ability spectrum was estimated using item response theory. Results The review of machine‐marked questions took an average of 1 minute, 36 seconds per question for all students. The VSAQs had high reliability (alpha: 0.91), a significantly lower SEM than the SBAQs (p < 0.001) and higher mean item–total point biserial correlations (p < 0.001). The VSAQ scores were significantly lower than the SBAQ scores (p < 0.001). The difference in scores between VSAQs and SBAQs was attenuated in Group 2. Although 80.4% of students found the VSAQs more difficult, 69.2% found them more authentic. Conclusions The VSAQ format demonstrated high reliability and discrimination and items were perceived as more authentic. The SBAQ format was associated with significant cueing. The present results suggest the VSAQ format has a higher degree of validity.
Issue Date: 1-Apr-2018
Date of Acceptance: 7-Nov-2017
URI: http://hdl.handle.net/10044/1/58945
DOI: https://dx.doi.org/10.1111/medu.13504
ISSN: 0308-0110
Publisher: WILEY
Start Page: 447
End Page: 455
Journal / Book Title: MEDICAL EDUCATION
Volume: 52
Issue: 4
Copyright Statement: © 2018 John Wiley & Sons Ltd and The Association for the Study of Medical Education. This is the accepted version of the following article: Sam, A. H., Field, S. M., Collares, C. F., Vleuten, C. P., Wass, V. J., Melville, C. , Harris, J. and Meeran, K. (2018), Very‐short‐answer questions: reliability, discrimination and acceptability. Med Educ, 52: 447-455, which has been published in final form at https://dx.doi.org/10.1111/medu.13504
Keywords: Social Sciences
Science & Technology
Life Sciences & Biomedicine
Education, Scientific Disciplines
Health Care Sciences & Services
Education & Educational Research
13 Education
11 Medical And Health Sciences
17 Psychology And Cognitive Sciences
Medical Informatics
Publication Status: Published
Online Publication Date: 2018-02-01
Appears in Collections:Department of Medicine (up to 2019)