Python Penetration Testing Cookbook

How to do it...

We can go through some simple usage of Scrapy's interactive shell. The steps are as follows:

1. Open up a Terminal window and type the following command:

$ Scrapy shell http://books.toscrape.com/  

After loading the Scrapy shell, it will open up an interface to interact with the response object as follows:

2. We can use this interface to debug the selectors for the response object:

>>> response.xpath('//ol/li/article')  

This will print the selector output. With this, we can create and test the extraction rules for spiders.

3. We can also open the Scrapy shell from the code for debugging errors in extraction rules. For that, we can ...