June 2022
Beginner to intermediate
233 pages
6h 28m
English
Content preview from Advanced Analytics with PySpark
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Chapter 9. Analyzing Genomics Data and the BDG Project
The advent of next-generation DNA sequencing (NGS) technology has rapidly transformed the life sciences into a data-driven field. However, making the best use of this data is butting up against a traditional computational ecosystem that builds on difficult-to-use, low-level primitives for distributed computing and a jungle of semistructured text-based file formats.
This chapter will serve two primary purposes. First, we introduce a set of popular serialization and file formats (Avro and Parquet) that simplify many problems in data management. These serialization technologies enable us to convert data into compact, machine-friendly binary representations. This helps with movement of data across networks and helps with cross-compatibility across programming languages. Although we will use data serialization techniques with genomics data, the concepts will be useful whenever processing large amounts of data.
Second, we show how to perform typical genomics tasks in the PySpark ecosystem. Specifically, we’ll use PySpark and the open source ADAM library to manipulate large quantities of genomics data and process data from multiple sources to create a dataset for predicting transcription factor (TF) binding sites. For this, we will join genome annotations from the ENCODE dataset. This chapter will serve as a tutorial to the ADAM project, which comprises a set of genomics-specific Avro schemas, PySpark-based APIs, and command-line ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.