Reliability of crowdsourcing as a method for collecting emotions labels on pictures; BMC Research Notes; Vol. 12
| Parent link: | BMC Research Notes Vol. 12.— 2019.— [715, 6 p.] |
|---|---|
| Autor principal: | |
| Autor Corporativo: | |
| Outros Autores: | , |
| Resumo: | Title screen Objective In this paper we study if and under what conditions crowdsourcing can be used as a reliable method for collecting high-quality emotion labels on pictures. To this end, we run a set of crowdsourcing experiments on the widely used IAPS dataset, using the Self-Assessment Manikin (SAM) emotion collection instrument, in order to rate pictures on valence, arousal and dominance, and explore the consistency of crowdsourced results across multiple runs (reliability) and the level of agreement with the gold labels (quality). In doing so, we explored the impact of targeting populations of different level of reputation (and cost) and collecting varying numbers of ratings per picture. Results The results tell us that crowdsourcing can be a reliable method, reaching excellent levels of reliability and agreement with only 3 ratings per picture for valence and 8 per arousal, with only marginal difference between target populations. Results for dominance were very poor, echoing previous studies on the data collection instrument used. We also observed that specific types of content generate diverging opinions in participants (leading to higher variability or multimodal distributions), which remain consistent across pictures of the same theme. These can inform the data collection and exploitation of crowdsourced emotion datasets. |
| Idioma: | inglês |
| Publicado em: |
2019
|
| Assuntos: | |
| Acesso em linha: | https://doi.org/10.1186/s13104-019-4764-4 |
| Formato: | Recurso Electrónico Capítulo de Livro |
| KOHA link: | https://koha.lib.tpu.ru/cgi-bin/koha/opac-detail.pl?biblionumber=664077 |
MARC
| LEADER | 00000naa0a2200000 4500 | ||
|---|---|---|---|
| 001 | 664077 | ||
| 005 | 20250425141352.0 | ||
| 035 | |a (RuTPU)RU\TPU\network\35261 | ||
| 090 | |a 664077 | ||
| 100 | |a 20210326d2019 k||y0rusy50 ba | ||
| 101 | 0 | |a eng | |
| 102 | |a GB | ||
| 135 | |a drcn ---uucaa | ||
| 181 | 0 | |a i | |
| 182 | 0 | |a b | |
| 200 | 1 | |a Reliability of crowdsourcing as a method for collecting emotions labels on pictures |f O. E. Korovina, G. M. A. Baez, F. Casati | |
| 203 | |a Text |c electronic | ||
| 300 | |a Title screen | ||
| 320 | |a [References: 15 tit.] | ||
| 330 | |a Objective In this paper we study if and under what conditions crowdsourcing can be used as a reliable method for collecting high-quality emotion labels on pictures. To this end, we run a set of crowdsourcing experiments on the widely used IAPS dataset, using the Self-Assessment Manikin (SAM) emotion collection instrument, in order to rate pictures on valence, arousal and dominance, and explore the consistency of crowdsourced results across multiple runs (reliability) and the level of agreement with the gold labels (quality). In doing so, we explored the impact of targeting populations of different level of reputation (and cost) and collecting varying numbers of ratings per picture. Results The results tell us that crowdsourcing can be a reliable method, reaching excellent levels of reliability and agreement with only 3 ratings per picture for valence and 8 per arousal, with only marginal difference between target populations. Results for dominance were very poor, echoing previous studies on the data collection instrument used. We also observed that specific types of content generate diverging opinions in participants (leading to higher variability or multimodal distributions), which remain consistent across pictures of the same theme. These can inform the data collection and exploitation of crowdsourced emotion datasets. | ||
| 461 | |t BMC Research Notes | ||
| 463 | |t Vol. 12 |v [715, 6 p.] |d 2019 | ||
| 610 | 1 | |a электронный ресурс | |
| 610 | 1 | |a труды учёных ТПУ | |
| 610 | 1 | |a crowdsourcing emotions | |
| 610 | 1 | |a empirical study | |
| 610 | 1 | |a rating behavior | |
| 610 | 1 | |a reliability | |
| 610 | 1 | |a краудсорсинг | |
| 700 | 1 | |a Korovina |b O. E. |g Olga Evgenjevna | |
| 701 | 1 | |a Baez |b G. M. A. |g Gonzalez Markos Antonio | |
| 701 | 1 | |a Casati |b F. |c Italian economist and Professor at the University of Trento (Italy) |c Professor of Tomsk Polytechnic University, candidate of technical Sciences |f 1971- |g Fabio |3 (RuTPU)RU\TPU\pers\39820 | |
| 712 | 0 | 2 | |a Национальный исследовательский Томский политехнический университет |b Институт социально-гуманитарных технологий |b Кафедра экономики |b Международная научно-образовательная лаборатория технологий улучшения благополучия пожилых людей |3 (RuTPU)RU\TPU\col\21257 |
| 801 | 0 | |a RU |b 63413507 |c 20210326 |g RCR | |
| 850 | |a 63413507 | ||
| 856 | 4 | |u https://doi.org/10.1186/s13104-019-4764-4 | |
| 942 | |c CF | ||