O'Reilly logo

Webbots, Spiders, and Screen Scrapers by Michael Schrenk

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

How the Search-Ranking Webbot Works

Our search-ranking webbot uses the process detailed in Figure 11-4 to determine the ranking of a website using a specific search term. These are the steps:

  1. Initialize variables for use, including the search criteria and the subject web page.

  2. Fetch the subject web page from the search engine using the search term.

  3. Parse the organic search results from the advertisement and navigation text.

  4. Determine whether or not the desired website appears in this page's search results.

    1. If the desired website is not found, keep looking deeper into the search results until the desired web page is found or the maximum number of attempts has been used.

    2. If the desired website is found, record the ranking.

  5. Report the results.

Figure 11-4. Search-ranking ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required