Web Scraping with Python

Preface

To those who have not developed the skill, computer programming can seem like a kind of magic. If programming is magic, then web scraping is wizardry; that is, the application of magic for particularly impressive and useful—yet surprisingly effortless—feats. 

In fact, in my years as a software engineer, I’ve found that very few programming practices capture the excitement of both programmers and laymen alike quite like web scraping. The ability to write a simple bot that collects data and streams it down a terminal or stores it in a database, while not difficult, never fails to provide a certain thrill and sense of possibility, no matter how many times you might have done it before. 

It’s unfortunate that when I speak to other programmers about web scraping, there’s a lot of misunderstanding and confusion about the practice. Some people aren’t sure if it’s legal (it is), or how to handle the modern Web, with all its JavaScript, multimedia, and cookies. Some get confused about the distinction between APIs and web scrapers.

This book seeks to put an end to many of these common questions and misconceptions about web scraping, while providing a comprehensive guide to most common web-scraping tasks.

Beginning in Chapter 1, I’ll provide code samples periodically to demonstrate concepts. These code samples are in the public domain, and can be used with or without attribution (although acknowledgment is always appreciated). All code samples also will be available on GitHub for ...