Automating the generation of clusters with cfncluster

As we just learned, we can spawn machines using the web interface, but it quickly becomes tedious and error prone. Fortunately, Amazon has an API. This means that we can write scripts that perform all the operations we discussed earlier, automatically. Even better, others have already developed tools that can be used to mechanize and automate many of the processes you want to perform with AWS.

Amazon themselves provide many command-line tools for their own infrastructure. For cluster provision, the tool is called cfncluster. If you are using conda, you can install it with:

$ conda install cfncluster

You can run this from your local machine: it will use the Amazon API.

The first step is ...

Get Building Machine Learning Systems with Python - Third Edition now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.