Building Machine Learning Systems with Python - Third Edition

Using jug to break up your pipeline into tasks

Often, we have a simple pipeline: we preprocess the initial data, compute features, and then call a machine learning algorithm with the resulting features.

Jug is a package developed by Luis Pedro Coelho, one of the authors of this book. It's open source (using the liberal MIT license) and can be useful in many areas, but was designed specifically around data analysis problems. It simultaneously solves several problems, for example:

• It can memoize results to disk (or a database), which means that if you ask it to compute something you have already computed before, the result is instead read from disk.
• It can use multiple cores or even multiple computers on a cluster. Jug was also designed to ...