Learning Python Networking - Second Edition

Pipelines items and export formats

The items pipelines could be called the channels or pipes of the items. They are elements of Scrapy and the information that arrives to them are Items that have been previously obtained and processed by some spider. They are classes in themselves that have a simple objective—to re-process the item that arrives to them, being able to reject it for some reasons or let it pass through this channel.

The typical uses of pipelines are as follows:

• Cleaning data in HTML
• Validation of scraped data checking that the items contain certain fields
• Checking duplicate items
• Storage of the data in a database

For each element that is obtained, it is sent to the corresponding pipeline, which will process it either to save ...