R for Data Science, 2nd Edition

Chapter 22. Arrow

Introduction

CSV files are designed to be easily read by humans. They’re a good interchange format because they’re simple and they can be read by every tool under the sun. But CSV files aren’t efficient: you have to do quite a lot of work to read the data into R. In this chapter, you’ll learn about a powerful alternative: the parquet format, an open standards–based format widely used by big data systems.

We’ll pair parquet files with Apache Arrow, a multilanguage toolbox designed for efficient analysis and transport of large datasets. We’ll use Apache Arrow via the arrow package, which provides a dplyr backend allowing you to analyze larger-than-memory datasets using familiar dplyr syntax. As an additional benefit, arrow is extremely fast; you’ll see some examples later in the chapter.

Both arrow and dbplyr provide dplyr backends, so you might wonder when to use each. In many cases, the choice is made for you, as in the data is already in a database or in parquet files, and you’ll want to work with it as is. But if you’re starting with your own data (perhaps CSV files), you can either load it into a database or convert it to parquet. In general, it’s hard to know what will work best, so in the early stages of your analysis, we encourage you to try both and pick the one that works the best for you.

(A big thanks to Danielle Navarro who contributed the initial version of this chapter.)

Prerequisites

In this chapter, we’ll continue to use the tidyverse, particularly ...