Mastering Large Datasets with Python

Chapter 11. Large datasets in the cloud with Amazon Web Services and S3

This chapter covers

• Understanding distributed object storage in the cloud
• Using the AWS web interface to set up buckets and upload objects
• Working with the boto3 library to upload data to an S3 bucket

In chapters 7–10, we saw the power of the distributed frameworks in Hadoop and Spark. These frameworks can take advantage of clusters of computers to parallelize massive data processing tasks and complete them in short order. Most of us, however, don’t have access to physical compute clusters.

In contrast, we can all get access to compute clusters from cloud service providers such as Amazon, Microsoft, and Google. These cloud providers have platforms that we can use for ...