The Web Bot
As anyone who has spent any time at all online can tell you, there is a lot of information available on the Internet. According to Google’s indexes, as of 2013 there are 4.04 billion web pages in existence. (See Figure 5-1.) Sure, a lot of those pages are probably cat pictures and pornography, but there are also hundreds of millions of pages with information on them. Useful information. It has been said that every piece of information that has been digitized exists somewhere on the Internet. It just has to be found—not an easy task when the Internet looks something like Figure 5-1.
Figure 5-1. Visual map of the Internet ...