Errata

current: "As for the Fleiss's Kappa score...well, that's definitely one that would have to be revisited. But why, and what should be changed to get better agreement? Looking at the chart
again, we can see that there's a lot of variation in all of the columns?while Annotators
A, B, and D all have nearly the same number of positive reviews, those are the only values
that come anywhere close to grouping. Annotator E is so far off from everyone else that
if this were a real task, we?d be wondering if she got hold of a completely different set of
guidelines!"


corrected: "As for the Fleiss's Kappa score... well, that's definitely one that would havee to be revisited. Looking at the chart again, we can see that there's a lot of variation in all of the columns--in face, none of the reviews seem to have any real sense of agreement. There are a lot of factors that can influence a crowdsourced project, such as not being able to train annotators, not being able to ensure that the annotators meet certain guidelines (such as native language), or just sheer online mischief; see Chapter 12 for discussion of some of the most common platforms and pitfalls. Of course, our example was made up, but agreement scores like these definitely mean that you need to review your annotation guidelines, and probably your dataset as well."

paragraph 1:

current: "...which represents each annotator's agreement with other annotators, compared to all possible agreement values. As before, a is the number of annotations per annotator, k is the number of categories, c is the current category, and i is the current annotator."


corrected: "... which represents the annotator's agreement per review compared to all possible agreement values. As before, a is the number of annotations per review, k is the number of categories, c is the current category, and i is the current review."


--------------------------------------------

paragraph 2:

current: " ...moderating the output by the number of total annotations by each annotator. So for Annotator A, we would calculate this:"

corrected: "...moderating the output by the number of total annotations for each review. So for Review 1, we would calculate this:"

-------------------------------------

first line of equation after paragraph 2:

current: "P(Annotator 1)=..."

corrected: "P(Review 1) = ..."

-------------------------------------------

paragraph 3:

current: "Annotator row"

corrected: "Review row"

---------------------------------------

paragraph 4:

Change Pi to have P with a subscript i;

last sentence: change "number of annotators" to "number of reviews"

paragraph 1:

current: "In this table, the categories are across the top, the annotators are down the side, and the content of each cell represents how many times that annotator assigned that tag to a
document."

corrected: "In this table, the categories are across the top, the movie review documents are down the side, and the content of each cell represents how many times that an annotator assigned each category to each review."

--------------------------------------

paragraph 2, last sentence:

current: "In the following equation, A is the number of annotators, a is the number of
annotations per annotator, k is the number of categories, and i represents the current
annotator:"

corrected: "In the following equation, A is the number of reviews, a is the number of annotations per review, k is the number of categories, and i represents the current table cell."

---------------------------------------------------

paragraph 3, first and second sentences:

current:"...will equal the sum of the values in its column divided by the number of annotators times the number of annotations each annotator created. The second part..."

corrected: "... will equal the sum of the values in its column divided by the number of reviews (5) times the number of annotations per review (250). The second version..."

All instances of the table should replace the leftmost column's "Annotator A, B, C, D, E" with "Review 1, 2, 3, 4, 5"

The description of the use of Fleiss's kappa was unclear, and the problem is incorrectly described. The paragraph should read:

The table used to represent annotator values for Fleiss's Kappa, rather than having one axis per annotator, has one axis for the possible values an annotator could assign, and the other axis for each of the items being annotated. The contents of the cells show how many annotators assigned each category to each item. Note that Fleiss's Kappa does not assume that all items are annotated by the <i>same</i> annotators, but it does assume that all items are annotated the same number of times. Just so we can look at some bigger numbers, let's assume we redid our movie review annotation task as a crowdsourcing project (see Chapter 12). Instead of having 250 movie reviews annotated by 2 people, let's say that we had 5 movie reviews annotated as positive, neutral, or negative by 250 people each. These annotations would be represented like this:

"Predicted Labeling" and "Gold Labeling" should be put together and centered. As they are now, each word seem to be referred to one column/row in particular. This lead to a bit of initial confusion.

Note from the Author or Editor:
yes, each of these labels should span the rows/columns that they are labeling

In the printed, black and white version, the colors of "training" and "dev-test" are very similar.

I suggest to use geometric patterns.

Note from the Author or Editor:
True; the PDF version has the image in color, where the differences are much clearer. Thank you for pointing this out.