Search Patterns

Chapter 5. Engines of Discovery

"Let your hook always be cast. In the pool where you least expect it, there will be fish.”

—Ovid

Serendipity refers to an accidental yet desirable discovery. It plays a surprising role. Alexander Fleming’s failure to disinfect cultures of bacteria before leaving for vacation led to the discovery of penicillin. Archimedes figured out how to measure the volume of irregularly shaped objects by taking a bath. And Columbus found America by sailing for India. LSD, Uranus, Viagra, safety glass, infrared radiation, microwave ovens, inkjet printers, Corn Flakes, and chocolate chip cookies are all accidental discoveries. It’s amazing what we find while searching for something else. In fact, search and serendipity often travel together. Discovery requires that we move beyond what we know. Ironically, the most frustrating journeys can lead to the best and least expected destinations. As the dictionary evangelist Erin McKean remarked, “Serendipity is when you find things you weren’t looking for because finding what you are looking for is so damned difficult.”

This reveals a tension in search between relevance and interestingness. The most interesting (and valuable) results aren’t always the most relevant. Often, the best answers lie just beyond the edges of what we know to seek. In this ambiguity lies the promised land of personalization, collaborative filtering, recommender systems, and discovery engines. But software algorithms take us only so far. We must also ...