Collaborative Annotation for Reliable Natural Language Processing

Glossary

Annotation:

covers both the process of adding a note on a source signal and the whole set of notes or each note that results from this process, without a priori presuming what would be the nature of the source (text, video, images, etc.), the semantic content of the note (numbered note, value chosen in a reference list or free text), its position (global or local) or its objective (evaluation, characterization, simple comment).

Crowdsourcing:

“the act of outsourcing tasks, traditionally performed by an employee or contractor, to an undefined, large group of people or community (a crowd), through an open call” [Wikipedia, December 2, 2010].

Game With A Purpose:

an entertaining setting – a gamified interface or a full-fledged game – in which voluntary participants produce data that require human knowledge and learning capabilities.

Markables:

segments of the source signal that could be annotated: for example, all of the tokens for POS annotation, but only the noun phrases for named entity annotation.

Microworking:

(micro) remunerated crowdsourcing, in which the tasks are decomposed and simplified so as to be carried out by workers, (micro) paid by the (micro) task.

Prevalence:

a category is prevalent when it is dominant, i.e. when it is used more than the others, without the proportion being very precisely defined.

Requester:

a person (or group of persons) proposing tasks on Amazon Mechanical Turk.

Turker:

a worker on Amazon Mechanical Turk.