Web Scraping with Python, 2nd Edition

Chapter 1. Your First Web Scraper

Once you start web scraping, you start to appreciate all the little things that browsers do for you. The web, without a layer of HTML formatting, CSS styling, JavaScript execution, and image rendering, can look a little intimidating at first, but in this chapter, as well as the next one, we’ll cover how to format and interpret data without the help of a browser.

This chapter starts with the basics of sending a GET request (a request to fetch, or “get,” the content of a web page) to a web server for a specific page, reading the HTML output from that page, and doing some simple data extraction in order to isolate the content that you are looking for.

Connecting

If you haven’t spent much time in networking or network security, the mechanics of the internet might seem a little mysterious. You don’t want to think about what, exactly, the network is doing every time you open a browser and go to http://google.com, and, these days, you don’t have to. In fact, I would argue that it’s fantastic that computer interfaces have advanced to the point where most people who use the internet don’t have the faintest idea about how it works.

However, web scraping requires stripping away some of this shroud of interface—not just at the browser level (how it interprets all of this HTML, CSS, and JavaScript), but occasionally at the level of the network connection.

To give you an idea of the infrastructure required to get information to your browser, let’s use the ...