Automate the Boring Stuff with Python, 2nd Edition

12WEB SCRAPING

In those rare, terrifying moments when I’m without Wi-Fi, I realize just how much of what I do on the computer is really what I do on the internet. Out of sheer habit I’ll find myself trying to check email, read friends’ Twitter feeds, or answer the question, “Did Kurtwood Smith have any major roles before he was in the original 1987 RoboCop?”¹

Since so much work on a computer involves going on the internet, it’d be great if your programs could get online. Web scraping is the term for using a program to download and process content from the web. For example, Google runs many web scraping programs to index web pages for its search ...