Learning Python Networking - Second Edition

Creating a project with Scrapy

Before starting with Scrapy, you have to start a project where you want to store your code. To create a project with Scrapy, you have to execute the command from the console:

scrapy startproject helloProject

This command will create a helloProject directory with the following contents:

helloProject/ scrapy.cfg # deploy configuration file helloProject/ # project's Python module, you'll import your code from here     __init__.py     items.py # project items file     pipelines.py # project pipelines file     settings.py # project settings file     spiders/ # a directory where you'll later put your spiders         __init__.py

Each project consists of the following:

• items.py: We define the elements to extract
• spiders: The heart of the project, ...