System Design on AWS

Chapter 13. Designing a Web Crawler and Search Engine

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 15th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

You planned a get together with your loved ones in the holiday season. You love cooking and decided to cook all the food by yourself, but you don’t have the recipes for the dishes you wish to prepare. What is the best possible resolution here? You could ask your friends if they have the recipes, or go looking through cookbooks, but a very simple yet effective solution is using Google search. Google looks across the internet and finds you the best results on how to prepare a specific meal. How does Google go through such a vast sea of information and find the perfect answer? In this chapter, we’ll try to figure out this answer by digging into the architecture of such search systems.

At a high level, the entire system consists of two subsystems, a web crawler and a search engine. A web crawler is essentially a software responsible for crawling the web. The content on the internet ...