
42 Text Mining and Visualization: Case Studies Using Open-Source Tools
FIGURE 2.1: Observed variation for the word “the” for consecutive 5,000-word windows
within the novel Moby Dick.
statistic that captured these. This summary could then be used as a feature vector for an
author to determine if the author exhibited a variation that was unique.
Returning to the Zipf-Mandelbrot distribution from Section 2.3.3, one possibility is to
use this as a way to fit the distribution to the observed probabilities and use the values
of the parameters A, B, C of Equation 2.7 as a proxy for the deviation from randomness.
This makes sense because the rank–frequency ...