November 2013
Intermediate to advanced
148 pages
3h 21m
English
Content preview from Python and HDF5Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Chapter 9. Concurrency: Parallel HDF5, Threading, and Multiprocessing
Over the past 10 years or so, parallel code has become crucial to scientific programming. Nearly every modern computer has at least two cores; dedicated workstations are readily available with 12 or more. Plunging hardware prices have made 100-core clusters feasible even for small research groups.
As a rapid development language, and one with easy access to C and
FORTRAN libraries, Python is increasingly being used as a top-level “glue”
language for such platforms.
Scientific programs written in Python can leverage existing “heavy-lifting”
libraries written in C or FORTRAN using any number of mechanisms, from
ctypes to Cython to the built-in NumPy routines.
This chapter discusses the various mechanisms in Python for writing parallel code, and how they interact with HDF5.
Python Parallel Basics
Broadly speaking, there are three ways to do concurrent programming in Python:
threads, the multiprocessing module, and finally by using bindings for the
Message Passing Interface (MPI).
Thread-based code is fine for GUIs and applications that call into external libraries that don’t tie up the Python interpreter. As we’ll see in a moment, you can’t use more than one core’s worth of time when running a pure-Python program. There’s also no performance advantage on the HDF5 side to using threads, since the HDF5 library serializes all calls.
multiprocessing is a more recent built-in module available with Python, which provides support ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.