To show a more complex example that you can take and adapt to your own needs, the Pipeline demo from the Pipeline folder implements a simple web spider. This code accepts a URL and retrieves all of the pages on that website.
This project is not meant to be a fully functional web spider application, but a relatively simple demo. It may not work correctly on all sites. It may also cause you to be temporarily locked out of accessing the site that you are trying to crawl as it can generate a big amount of http requests, which may trigger security measures on the website.
This example is significantly more complicated than the pipeline concept I have discussed so far. It extracts data (URLs) from the retrieved pages and feeds them back ...
