O'Reilly logo

Learning Scrapy by Dimitrios Kouzis-Loukas

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Summary

In this chapter, we dived a bit deeper into the internals of Scrapy spiders. We used FormRequest to log in, passed variables around with meta of Request/Response, used relevant XPaths and Selectors, used .csv files as sources, and much more.

We are now ready to see how we can deploy our spiders in the Scrapinghub cloud in the brief Chapter 6, Deploying to Scrapinghub before we move onto reviewing the wealth of Scrapy settings in Chapter 7, Configuration and Management.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required