Web Scraping with Python, 3rd Edition

Chapter 8. Scrapy

Chapter 7 presented some techniques and patterns for building large, scalable, and (most important!) maintainable web crawlers. Although this is easy enough to do by hand, many libraries, frameworks, and even GUI-based tools will do this for you or at least try to make your life a little easier.

Since its release in 2008, Scrapy has quickly grown into the largest and best-maintained web scraping framework in Python. It is currently maintained by Zyte (formerly Scrapinghub).

One of the challenges of writing web crawlers is that you’re often performing the same tasks again and again: find all links on a page, evaluate the difference between internal and external links, and go to new pages. These basic patterns are useful to know and to be able to write from scratch, but the Scrapy library handles many of these details for you.

Of course, Scrapy isn’t a mind reader. You still need to define page templates, give it locations to start scraping from, and define URL patterns for the pages that you’re looking for. But in these cases, it provides a clean framework to keep your code organized.

Installing Scrapy

Scrapy offers the tool for download from its website, as well as instructions for installing Scrapy with third-party installation managers such as pip.

Because of its relatively large size and complexity, Scrapy is not usually a framework that can be installed in the traditional way with:

$ pip install Scrapy

Note that I say “usually” because, though it is theoretically ...