In the previous chapter, I indicated that monitoring plays a central role in managing algorithms. This is surprisingly tricky. As Ron DeLegge II put it nicely, “99 percent of all statistics only tell 49 percent of the story.”1 As a result, a lot of rubbish is said and done because of meaningless numbers showing up in some report. Even if no bad intentions are involved, a poorly calculated or interpreted number can seriously mislead you. This chapter is a comprehensive review of how best to monitor algorithms for biases from a user’s perspective.
I have tried ...