ERICA: expert guidance in validating crowd answers
File(s)erica_validating_crowd_answers_SIGIR_DEMO_2014.pdf (1.33 MB)
Accepted version
Author(s)
Hung, NQV
Thang, DC
Weidlich, M
Aberer, K
Type
Conference Paper
Abstract
Crowdsourcing became an essential tool for a broad range of Web applications. Yet, the wide-ranging levels of expertise of crowd workers as well as the presence of faulty workers call for quality control of the crowdsourcing result. To this end, many crowdsourcing platforms feature a post-processing phase, in which crowd answers are validated by experts. This approach incurs high costs though, since expert input is a scarce resource. To support the expert in the validation process, we present a tool for ExpeRt guidance In validating Crowd Answers (ERICA). It allows us to guide the expert's work by collecting input on the most problematic cases, thereby achieving a set of high quality answers even if the expert does not validate the complete answer set. The tool also supports the task requester in selecting the most cost-efficient allocation of the budget between the expert and the crowd.
Date Issued
2015-08-09
Date Acceptance
2015-08-09
Citation
Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2015, pp.1037-1038
ISBN
9781450336215
Publisher
Association for Computing Machinery
Start Page
1037
End Page
1038
Journal / Book Title
Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval
Copyright Statement
© ACM 2015. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, http://dx.doi.org/10.1145/2766462.2767866
Source
Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval
Publication Status
Published
Start Date
2015-08-09
Finish Date
2015-08-13
Coverage Spatial
Santiago, Chile