Massive Open Online Course (MOOC) evaluation methods: A systematic review
File(s)
Published version
Author(s)
Type
Journal Article
Abstract
Background: Massive open online courses (MOOCs) have the potential for broad education impact due to many learners undertaking these courses. Despite their reach, there is a lack of knowledge about which methods are used for evaluating these courses.
Objective: This review aims to identify current MOOC evaluation methods in order to inform future study designs.
Methods: We systematically searched the following databases: (1) SCOPUS; (2) Education Resources Information Center (ERIC); (3) IEEE Xplore; (4) Medline/PubMed; (5) Web of Science; (6) British Education Index and (7) Google Scholar search engine for studies from January 2008 until October 2018. Two reviewers independently screened abstracts and titles of the studies. Published studies in English that evaluated MOOCs were included. The study design of the evaluations, the underlying motivation for the evaluation studies, data collection and data analysis methods were quantitatively and qualitatively analyzed. The quality of the included studies was appraised using the Cochrane Collaboration Risk of Bias Tool for RCTs, the NIH - National Heart, Lung and Blood Institute quality assessment tool for cohort observational studies, and for “Before-After (Pre-Post) Studies With No Control Group”.
Results: The initial search resulted in 3275 studies, and 33 eligible studies were included in this review. Studies mostly had a cross-sectional design evaluating one version of a MOOC. We found that studies mostly had a learner-focused, teaching-focused or platform-focused motivation to evaluate the MOOC. The most used data collection methods were surveys, learning management system data and quiz grades and the most used data analysis methods were descriptive and inferential statistics. The methods for evaluating the outcomes of these courses were diverse and unstructured. Most studies with cross-sectional design had a low-quality assessment, whereas randomized controlled trials and quasi-experimental studies received higher quality assessment.
Conclusions: MOOC evaluation data collection and data analysis methods should be determined carefully based on the aim of the evaluation. The MOOC evaluations are subject to bias, which could be reduced by using pre-MOOC measures for comparison or by controlling for confounding variables. Future MOOC evaluations should consider using more diverse data sources and data analysis methods.
Objective: This review aims to identify current MOOC evaluation methods in order to inform future study designs.
Methods: We systematically searched the following databases: (1) SCOPUS; (2) Education Resources Information Center (ERIC); (3) IEEE Xplore; (4) Medline/PubMed; (5) Web of Science; (6) British Education Index and (7) Google Scholar search engine for studies from January 2008 until October 2018. Two reviewers independently screened abstracts and titles of the studies. Published studies in English that evaluated MOOCs were included. The study design of the evaluations, the underlying motivation for the evaluation studies, data collection and data analysis methods were quantitatively and qualitatively analyzed. The quality of the included studies was appraised using the Cochrane Collaboration Risk of Bias Tool for RCTs, the NIH - National Heart, Lung and Blood Institute quality assessment tool for cohort observational studies, and for “Before-After (Pre-Post) Studies With No Control Group”.
Results: The initial search resulted in 3275 studies, and 33 eligible studies were included in this review. Studies mostly had a cross-sectional design evaluating one version of a MOOC. We found that studies mostly had a learner-focused, teaching-focused or platform-focused motivation to evaluate the MOOC. The most used data collection methods were surveys, learning management system data and quiz grades and the most used data analysis methods were descriptive and inferential statistics. The methods for evaluating the outcomes of these courses were diverse and unstructured. Most studies with cross-sectional design had a low-quality assessment, whereas randomized controlled trials and quasi-experimental studies received higher quality assessment.
Conclusions: MOOC evaluation data collection and data analysis methods should be determined carefully based on the aim of the evaluation. The MOOC evaluations are subject to bias, which could be reduced by using pre-MOOC measures for comparison or by controlling for confounding variables. Future MOOC evaluations should consider using more diverse data sources and data analysis methods.
Date Issued
2020-04-27
Date Acceptance
2020-01-22
Citation
Journal of Medical Internet Research, 2020, 22 (4), pp.1-14
ISSN
1438-8871
Publisher
JMIR Publications
Start Page
1
End Page
14
Journal / Book Title
Journal of Medical Internet Research
Volume
22
Issue
4
Copyright Statement
©Abrar Alturkistani, Ching Lam, Kimberley Foley, Terese Stenfors, Elizabeth R Blum, Michelle Helena Van Velthoven, Edward
Meinert. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 27.04.2020. This is an open-access
article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/),
which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the
Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication
on http://www.jmir.org/, as well as this copyright and license information must be included.
Meinert. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 27.04.2020. This is an open-access
article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/),
which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the
Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication
on http://www.jmir.org/, as well as this copyright and license information must be included.
Sponsor
European Institute of Innovation and Technology
Identifier
https://www.jmir.org/2020/4/e13851/
Subjects
Science & Technology
Life Sciences & Biomedicine
Health Care Sciences & Services
Medical Informatics
online learning
learning
computer-assisted instruction
COURSE MOOC
LEARNERS
STUDENTS
EXPERIENCE
QUALITY
TOOL
computer-assisted instruction
learning
online learning
08 Information and Computing Sciences
11 Medical and Health Sciences
17 Psychology and Cognitive Sciences
Medical Informatics
Publication Status
Published
Date Publish Online
2020-04-27