Python Social Media Analytics

Creating spiders

Although a full tutorial for the Scrapy framework is beyond the scope of the chapter, we will show you how to create a simple and efficient spider.

The first important thing to understand about Scrapy is the project structure.

Source: https://doc.scrapy.org/en/latest/intro/tutorial.html

Directories tutorial will be replaced by your project name. In the first level, we store Scrapy configuration file. All the options are given by default. Usually, for simple scraping, there is no need to change the parameters. In the second level, we find items.py, pipelines.py, and settings.py. Items.py defines the type of elements we want ...