Text Mining and Visualization

Mining Search Logs for Usage Patterns 163

So at this point it may be prudent to review our choice of features; after all we have just

six so far that were selected in a relatively informal manner. To be maximally useful (e.g.,

indicative of a latent variable such as behaviour type), we would want them to be relatively

independent, i.e. uncorrelated with each other. But if we run a Pearson correlation across

the ones above we find the opposite: most of them are actually highly correlated, particularly

pairs such as All Actions and Duration. In this instance, these features may be “drowning

out” interesting patterns that we might otherwise see. Therefore, ...