In this chapter, we will be addressing the problem of building a web crawler. Web crawlers are important, for example, in the domain of indexing and searching the web.
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month, and much more.