Skip to Main Content
Python and HDF5
book

Python and HDF5

by Andrew Collette
November 2013
Intermediate to advanced content levelIntermediate to advanced
148 pages
3h 21m
English
O'Reilly Media, Inc.
Content preview from Python and HDF5

Chapter 4. How Chunking and Compression Can Help You

So far we have avoided talking about exactly how the data you write is stored on disk. Some of the most interesting features in HDF5, including per-dataset compression, are tied up in the details of how data is arranged on disk.

Before we get down to the nuts and bolts, there’s a more fundamental issue we have to discuss: how multidimensional arrays are actually handled in Python and HDF5.

Contiguous Storage

Let’s suppose we have a four-element NumPy array of strings:

>>> a = np.array([ ["A","B"], ["C","D"] ])
>>> print a
[['A' 'B']
 ['C' 'D']]

Mathematically, this is a two-dimensional object. It has two axes, and can be indexed using a pair of numbers in the range 0 to 1:

>>> a[1,1]
'D'

However, there’s no such thing as “two-dimensional” computer memory, at least not in common use. The elements are actually stored in a one-dimensional buffer:

'A' 'B' 'C' 'D'

This is called contiguous storage, because all the elements of the array, whether it’s stored on disk or in memory, are stored one after another. NumPy uses a simple set of rules to turn an indexing expression into the appropriate offset into this one-dimensional buffer. In this case, indexing along the first axis advances us into the buffer in steps (strides, in NumPy lingo) of 2, while indexing along the second axis advances us in steps of 1.

For example, the indexing expression a[0,1] is handled as follows:

offset = 2*0 + 1*1 -> 1
buffer[offset] -> value "B"

Note

You might notice that ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

Python One-Liners

Python One-Liners

Christian Mayer
Scaling Python with Dask

Scaling Python with Dask

Holden Karau, Mika Kimmins
Robust Python

Robust Python

Patrick Viafore
Python Cookbook

Python Cookbook

Alex Martelli, David Ascher

Publisher Resources

ISBN: 9781491944981Errata Page