How to do it...

Let's create a simple spider with the Scrapy:

  1. For creating a new spider project, open the Terminal and go to the folder for our spider:
$ mkdir new-spider
$ cd new-spider  
  1. Then run the following command to create a new spider project with scrapy:
$ scrapy startproject books  

This will create a project with the name books and some useful files for creating the crawler. Now you have a folder structure, as shown in the following screenshot:

  1. Now we can create a crawler with the following command:
$ scrapy genspider home books.toscrape.com  

This will generate the code for the spider with the name home, as we are planning to ...

Get Python Penetration Testing Cookbook now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.