In Chapter 3, Basic Crawling,
we focused on how to extract information from pages and store them into
Items. What we learned covers the most common Scrapy use cases, and it should be enough to get you up and running. In this chapter, we will examine more specialized cases in order to become more familiar with the two most important Scrapy classes—
Response—the two R's on the UR2IM scraping model we presented in Chapter 3, Basic Crawling.
Quite often, you may find yourself wanting to extract data from websites that have a login mechanism. In the most common case, a website will require you to provide a username and a password in order to log in. We are going to use the example that ...