Standard setting Very Short Answer Questions (VSAQs) relative to Single Best Answer Questions (SBAQs): does having access to the answers make a difference?
File(s)
Published version
Author(s)
Sam, Seyed Amir
Millar, Kate
Westacott, Rachel
Melville, Colin R
Brown, Celia A
Type
Journal Article
Abstract
Background
We investigated whether question format and access to the correct answers affect the pass mark set by standard-setters on written examinations.
Methods
Trained educators used the Angoff method to standard set two 50-item tests with identical vignettes, one in a single best answer question (SBAQ) format (with five answer options) and the other in a very short answer question (VSAQ) format (requiring free text responses). Half the participants had access to the correct answers and half did not. The data for each group were analysed to determine if the question format or having access to the answers affected the pass mark set.
Results
A lower pass mark was set for the VSAQ test than the SBAQ test by the standard setters who had access to the answers (median difference of 13.85 percentage points, Z=-2.82, p=0.002). Comparable pass marks were set for the SBAQ test by standard setters with and without access to the correct answers (60.65% and 60.90% respectively). A lower pass mark was set for the VSAQ test when participants had access to the correct answers (difference in medians -13.75 percentage points, Z=2.46, p=0.014).
Conclusions
When given access to the potential correct answers, standard setters appear to appreciate the increased difficulty of VSAQs compared to SBAQs.
We investigated whether question format and access to the correct answers affect the pass mark set by standard-setters on written examinations.
Methods
Trained educators used the Angoff method to standard set two 50-item tests with identical vignettes, one in a single best answer question (SBAQ) format (with five answer options) and the other in a very short answer question (VSAQ) format (requiring free text responses). Half the participants had access to the correct answers and half did not. The data for each group were analysed to determine if the question format or having access to the answers affected the pass mark set.
Results
A lower pass mark was set for the VSAQ test than the SBAQ test by the standard setters who had access to the answers (median difference of 13.85 percentage points, Z=-2.82, p=0.002). Comparable pass marks were set for the SBAQ test by standard setters with and without access to the correct answers (60.65% and 60.90% respectively). A lower pass mark was set for the VSAQ test when participants had access to the correct answers (difference in medians -13.75 percentage points, Z=2.46, p=0.014).
Conclusions
When given access to the potential correct answers, standard setters appear to appreciate the increased difficulty of VSAQs compared to SBAQs.
Date Issued
2022-08-23
Date Acceptance
2022-08-09
Citation
BMC Medical Education, 2022, 22
ISSN
1472-6920
Publisher
BioMed Central
Journal / Book Title
BMC Medical Education
Volume
22
Copyright Statement
© The Author(s) 2022. This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
License URL
Subjects
Social Sciences
Education & Educational Research
Education, Scientific Disciplines
Assessment
Undergraduate
Standard setting
ASSESSMENTS
Assessment
Standard setting
Undergraduate
Educational Measurement
Humans
Humans
Educational Measurement
Medical Informatics
1117 Public Health and Health Services
1302 Curriculum and Pedagogy
Publication Status
Published
Article Number
ARTN 640