Python Web Scraping - Second Edition

Implementing a multithreaded crawler

Fortunately, Python makes threading relatively straightforward. This means we can keep a similar queuing structure to the link crawler developed in Chapter 1, Introduction to Web Scraping, but start the crawl loop in multiple threads to download these links in parallel. Here is a modified version of the start of the link crawler with the crawl loop moved into a function:

import time import threading ...SLEEP_TIME = 1 def threaded_crawler(..., max_threads=10, scraper_callback=None):     ...    def process_queue():         while crawl_queue:             ... 

Here is the remainder of the threaded_crawler function to start process_queue in multiple threads and wait until they have completed:

threads = []  while threads or crawl_queue: ...