Bioinformatics with Python Cookbook - Second Edition

Using high-performance data formats – HDF5

VCF processing is very slow: if you do an empty for loop over a big VCF file, it can easily take days just to parse it. This is because text parsing is very demanding. Alternatively, using NumPy arrays is fast, but you are limited to whatever fits in memory. There are several alternatives to deal with both of these problems (and we will explore more than one in this chapter). Here, we will consider representing our data in HDF5 format.

We will use an existing HDF5 file that was exported from a VCF file and do some basic extraction of data.