Advanced Analytics with PySpark

Chapter 9. Analyzing Genomics Data and the BDG Project

The advent of next-generation DNA sequencing (NGS) technology has rapidly transformed the life sciences into a data-driven field. However, making the best use of this data is butting up against a traditional computational ecosystem that builds on difficult-to-use, low-level primitives for distributed computing and a jungle of semistructured text-based file formats.

This chapter will serve two primary purposes. First, we introduce a set of popular serialization and file formats (Avro and Parquet) that simplify many problems in data management. These serialization technologies enable us to convert data into compact, machine-friendly binary representations. This helps with movement of data across networks and helps with cross-compatibility across programming languages. Although we will use data serialization techniques with genomics data, the concepts will be useful whenever processing large amounts of data.

Second, we show how to perform typical genomics tasks in the PySpark ecosystem. Specifically, we’ll use PySpark and the open source ADAM library to manipulate large quantities of genomics data and process data from multiple sources to create a dataset for predicting transcription factor (TF) binding sites. For this, we will join genome annotations from the ENCODE dataset. This chapter will serve as a tutorial to the ADAM project, which comprises a set of genomics-specific Avro schemas, PySpark-based APIs, and command-line ...