Learning Python Networking - Second Edition

Executing Scrapy

With Scrapy, we can collect the information and save it in a file in one of the supported formats (XML, JSON, or CSV), or even directly in a database using a pipeline. In this case, we are executing the scrapy command passing as argument the JSON format:

$ scrapy crawl <crawler_name> -o items.json -t json

The last parameters indicate that the extracted data is stored in a file called items.json and that the exporter uses for JSON format. It can be done in the same way to export to CSV and XML formats.

The option -o items.csv provides as a parameter the name of the output file that will contain the data you have extracted. You can also extract information as JSON format by using the -t json option.

With the -t csv option, ...