O'Reilly logo

Python Data Analysis by Ivan Idris

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Speeding up embarrassingly parallel for loops with Joblib

Joblib is a Python library created by the developers of scikit-learn. Its main mission is to improve the performance of long-running Python functions. Joblib achieves the improvements through caching and parallelization using multiprocessing or threading under the hood. Install Joblib as follows:

$ pip install joblib
$ pip freeze|grep joblib
joblib==0.8.2

We will reuse the code from the previous example only changing the parallel() function. Refer to the joblib_demo.py file in this book's code bundle:

def parallel(nprocs): start = timeit.default_timer() Parallel(nprocs)(delayed(simulate)(i) for i in xrange(10, 50)) end = timeit.default_timer() - start print nprocs, "Parallel time", end return ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required