How to do it...

Let's create a simple spider with the Scrapy:

  1. For creating a new spider project, open the Terminal and go to the folder for our spider:
$ mkdir new-spider
$ cd new-spider  
  1. Then run the following command to create a new spider project with scrapy:
$ scrapy startproject books  

This will create a project with the name books and some useful files for creating the crawler. Now you have a folder structure, as shown in the following screenshot:

  1. Now we can create a crawler with the following command:
$ scrapy genspider home  

This will generate the code for the spider with the name home, as we are planning to ...

Get Python Penetration Testing Cookbook now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.