Skip to Main Content
Python and HDF5
book

Python and HDF5

by Andrew Collette
November 2013
Intermediate to advanced content levelIntermediate to advanced
148 pages
3h 21m
English
O'Reilly Media, Inc.
Content preview from Python and HDF5

Chapter 2. Getting Started

HDF5 Basics

Before we jump into Python code examples, it’s useful to take a few minutes to address how HDF5 itself is organized. Figure 2-1 shows a cartoon of the various logical layers involved when using HDF5. Layers shaded in blue are internal to the library itself; layers in green represent software that uses HDF5.

Most client code, including the Python packages h5py and PyTables, uses the native C API (HDF5 is itself written in C). As we saw in the introduction, the HDF5 data model consists of three main public abstractions: datasets (see Chapter 3), groups (see Chapter 5), and attributes (see Chapter 6)in addition to a system to represent types. The C API (and Python code on top of it) is designed to manipulate these objects.

HDF5 uses a variety of internal data structures to represent groups, datasets, and attributes. For example, groups have their entries indexed using structures called “B-trees,” which make retrieving and creating group members very fast, even when hundreds of thousands of objects are stored in a group (see How Groups Are Actually Stored). You’ll generally only care about these data structures when it comes to performance considerations. For example, when using chunked storage (see Chapter 4), it’s important to understand how data is actually organized on disk.

The next two layers have to do with how your data makes its way onto disk. HDF5 objects all live in a 1D logical address space, like in a regular file. However, there’s an ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

Python One-Liners

Python One-Liners

Christian Mayer
Scaling Python with Dask

Scaling Python with Dask

Holden Karau, Mika Kimmins
Robust Python

Robust Python

Patrick Viafore
Python Cookbook

Python Cookbook

Alex Martelli, David Ascher

Publisher Resources

ISBN: 9781491944981Errata Page