Learning Python Networking - Second Edition

Scrapy item class

Scrapy provides the item class to define the output data format. Item objects are containers used to collect the extracted data and specify metadata for the field used to characterize that data. For more details, see https://doc.scrapy.org/en/1.5/topics/items.html.

Create a file named MyItem.py and add the following code into it:

import scrapyfrom scrapy.loader.processors import TakeFirstclass MyItem(scrapy.Item):    # define the fields for your item here like:    name = scrapy.Field(output_processor=TakeFirst(),)

The next step is to describe how the information can be extracted using XPath expressions so that Scrapy can differentiate it from the rest of the HTML code on the page of each book.

To start the crawling process, it ...