Advanced Analytics with Spark, 2nd Edition

Chapter 11. Analyzing Neuroimaging Data with PySpark and Thunder

Uri Laserson

We are not interested in the fact that the brain has the consistency of cold porridge.

Alan Turing

Advances in imaging equipment and automation have led to a glut of data on the function of the brain. While past experiments might have generated time series data from only a handful of electrodes in the brain or a small number of static images of brain slices, technologies today can sample brain activity from a large number of neurons in a large region while organisms are actively behaving. Indeed, the BRAIN Initiative is a government-funded initiative with the lofty technology development goals of enabling, for example, simultaneous recording of the electrical activity of every neuron of the mouse brain over an extended period of time. While breakthroughs in measurement technology are certainly necessary, the amount of data generated will create completely new paradigms for biology.

In this chapter, we will introduce the PySpark API for interacting with Spark through Python, as well as the Thunder project, which is developed on top of PySpark for processing large amounts of time series data in general and neuroimaging data in particular. PySpark is a particularly flexible tool for exploratory big data analysis because it integrates well with the rest of the PyData ecosystem, including matplotlib for visualization, and even IPython Notebook (Jupyter) for “executable documents.”

We will marshal these ...