Python Web Scraping - Second Edition

Sitemap crawler

For our first simple crawler, we will use the sitemap discovered in the example website's robots.txt to download all the web pages. To parse the sitemap, we will use a simple regular expression to extract URLs within the <loc> tags.

We will need to update our code to handle encoding conversions as our current download function simply returns bytes. Note that a more robust parsing approach called CSS selectors will be introduced in the next chapter. Here is our first example crawler:

import redef download(url, user_agent='wswp', num_retries=2, charset='utf-8'):     print('Downloading:', url)     request = urllib.request.Request(url)     request.add_header('User-agent', user_agent)    try:         resp = urllib.request.urlopen(request) cs = resp.headers.get_content_charset() ...