Collaborative Annotation for Reliable Natural Language Processing

Appendix(Some) Annotation Tools

Keeping track of all the annotation tools is a job for Sisyphus. The list presented here is, therefore, far from exhaustive. It also contains deprecated tools or tools that were used only once, because they present interesting features that we want to expose. Finally, we have done our best to keep this list up-to-date, but some tools evolve so rapidly that it is difficult to keep track of changes; so if one of them seems interesting to you, do not hesitate to check its evolution on the provided website.¹

We chose not to use the traditional comparative tabular presentation, as it does not allow for the representation of the different underlying principles behind each tool. We prefer to detail each of them in its context of creation and usage.

The organization of the inventory that we propose here relies on a typology that we deduced from our tests.² Thus, section A.1 presents generic tools, some of which were born so whereas others evolved toward from a specific annotation task. The following section logically introduces task-oriented tools. The annotation platforms, which aim more directly at applying NLP tools on corpora, are presented in section A.3. Finally, section A.4 details annotation campaign management tools. Within each section, the tools are presented in historical order, from the oldest to the newest.

A.1. Generic tools

The majority of the tools presented here were not initially designed to be generic, but were created for a specific ...