CUED Publications database

Ambiguity Helps: Classification with Disagreements in Crowdsourced Annotations

Sharmanska, V and Hernandez-Lobato, D and Hernandez-Lobato, JM and Quadrianto, N (2016) Ambiguity Helps: Classification with Disagreements in Crowdsourced Annotations. In: UNSPECIFIED pp. 2194-2202..

Full text not available from this repository.


© 2016 IEEE. Imagine we show an image to a person and ask her/him to decide whether the scene in the image is warm or not warm, and whether it is easy or not to spot a squirrel in the image. For exactly the same image, the answers to those questions are likely to differ from person to person. This is because the task is inherently ambiguous. Such an ambiguous, therefore challenging, task is pushing the boundary of computer vision in showing what can and can not be learned from visual data. Crowdsourcing has been invaluable for collecting annotations. This is particularly so for a task that goes beyond a clear-cut dichotomy as multiple human judgments per image are needed to reach a consensus. This paper makes conceptual and technical contributions. On the conceptual side, we define disagreements among annotators as privileged information about the data instance. On the technical side, we propose a framework to incorporate annotation disagreements into the classifiers. The proposed framework is simple, relatively fast, and outperforms classifiers that do not take into account the disagreements, especially if tested on high confidence annotations.

Item Type: Conference or Workshop Item (UNSPECIFIED)
Divisions: Div F > Computational and Biological Learning
Depositing User: Cron Job
Date Deposited: 17 Jul 2017 19:29
Last Modified: 15 Sep 2020 05:05
DOI: 10.1109/CVPR.2016.241