Natural Language Annotation for Machine Learning

Chapter 9. Revising and Reporting

Finally, we’re at the “R” of the MATTER cycle—revising your project. Of course, you’ve probably been revising your project all along, as you worked your way through the MAMA cycle, and refit your algorithms through the training, testing, and evaluation stages of machine learning. However, while making adjustments at each step of the way, you may have focused only on the steps at hand, so in the first part of this chapter, we are going to take a step back and examine some of the “big picture” items that you may want to reconsider about your project. To that end, we’ll discuss:

• Corpus modification

• Model and specs

• Annotation task and annotators

• Algorithm implementation

In the second part of the chapter we will discuss what information you should include about your task when you are writing papers, giving presentations, or just putting together a website so that people can learn about your project. Creating annotated corpora and leveraging those corpora into good machine learning (ML) algorithms are difficult tasks, and because so many variables affect the outcome of a project, the more open you are about the choices you made, the more other people will be able to learn based on your example. Some of the aspects of your project you need to consider reporting on are:

• Corpus size, content, and creation

• Annotation methods and annotator qualifications

• ML modifications and training adjustments

• Revisions to your project, both implemented and planned

Revising Your ...