Learn Python by Building Data Science Applications

Writing to an S3 bucket

S3 buckets and similar blob storage services have proven to be a great tool. Given the small price and ease of use, they are arguably the best solution for shared data exchange in the cloud. Here, we won't go in depth on S3 and Amazon Web Services (AWS) in general. Instead, we'll show how to modify your existing pipelines to redirect them to S3. Aside from data being in the cloud, this has the benefit of a shared state—if another computer or user tries to run the pipeline, they will find that the data is there already.

Let's assume you have registered as an AWS customer and have an S3 bucket. To work with buckets, luigi uses the boto3 package, an official library for AWS-related operations, built by Amazon. This means ...