R for Data Science, 2nd Edition

Chapter 24. Web Scraping

Introduction

This chapter introduces you to the basics of web scraping with rvest. Web scraping is a useful tool for extracting data from web pages. Some websites will offer an API, a set of structured HTTP requests that return data as JSON, which you handle using the techniques from Chapter 23. Where possible, you should use the API,¹ because typically it will give you more reliable data. Unfortunately, however, programming with web APIs is out of scope for this book. Instead, we are teaching scraping, a technique that works whether or not a site provides an API.

In this chapter, we’ll first discuss the ethics and legalities of scraping before we dive into the basics of HTML. You’ll then learn the basics of CSS selectors to locate specific elements on the page and how to use rvest functions to get data from text and attributes out of HTML and into R. We’ll then discuss some techniques to figure out what CSS selector you need for the page you’re scraping, before finishing up with a couple of case studies and a brief discussion of dynamic websites.

Prerequisites

In this chapter, we’ll focus on tools provided by rvest. rvest is a member of the tidyverse but is not a core member, so you’ll need to load it explicitly. We’ll also load the full tidyverse since we’ll find it generally useful working with the data we’ve scraped.

library(tidyverse)
library(rvest)

Scraping Ethics and Legalities

Before we get started discussing the code you’ll need to perform ...