Enterprise Search, 2nd Edition

Chapter 15. Search Evaluation

Very rarely is the performance of an enterprise application measured in any detail. Invariably, these applications support a task or workflow and the efficiency is assessed in terms of whether the overall task could be accomplished. These applications are usually built on SQL or a similar database, and methods of optimizing these databases in performance terms are well understood and generally well implemented.

Search is different. Every user at every visit is at a different stage in undertaking a task, and somehow the search application has to deliver what is needed on a highly consistent basis. Recall and precision are both defined in terms of relevance, and yet the “relevance” of a piece of retrieved information is totally dependent on the personal context of the user. Some users might have an inadequate understanding of a concept and so may create an inappropriate query or not be able to judge the relevance of the results presented to them.

Far too many presentations at conferences focus on “improving relevance” but then go no further in considering the difference between recall and precision. Often the emphasis is on precision, seeing the document required on the first page of results, because some other application (typically an intranet) has failed to deliver the information being sought.

Search assessment needs to be undertaken on a carefully considered basis. Ad hoc analysis of search logs will give neither short-term nor long-term benefits. ...