book

Inside the Message Passing Interface

Name: Inside the Message Passing Interface
Author: Alexander Supalov
ISBN: 9781501506789

by Alexander Supalov

September 2018

Intermediate to advanced

384 pages

10h 47m

English

De Gruyter

Read now

Unlock full access

Title Page
Copyright
Dedication
Acknowledgments
Contents
Preface
Introduction
Chapter 1: Overview
1.1 Parallel Computer1.1.1 Intraprocessor Parallelism1.1.2 Interprocessor Parallelism1.1.3 Exercises1.2 MPI Standard1.2.1 MPI History1.2.2 Related Standards1.2.3 Exercises1.3 MPI Subsetting1.3.1 Motivation1.3.2 Sensible MPI Subsets1.3.3 Exercises1.4 Fine Points1.4.1 Programming1.4.2 Code Management1.4.3 Building1.4.4 Testing and Debugging1.4.5 Benchmarking1.4.6 Presentation1.4.7 Optimization1.4.8 Issue Tracking1.4.9 Exercises
Chapter 2: Shared Memory
2.1 Subset Definition2.1.1 Exercises2.2 Communication Basics2.2.1 Intraprocess Communication2.2.2 Interprocess Communication2.2.3 Ping-Pong Pattern Revisited2.2.4 Exercises2.3 Subset Implementation: Blocking Communication2.3.1 Design Decisions2.3.2 Start-up and Termination2.3.3 Blocking Point-to-Point Communication2.3.4 Testing2.3.5 Benchmarking2.3.6 Optimization2.3.7 Exercises2.4 Conclusions
Chapter 3: Sockets
3.1 Subset Definition3.1.1 Exercises3.2 Communication Basics3.2.1 Intranode Communication3.2.2 Internode Communication3.2.3 Exercises3.3 Initial Subset Implementation: Start-up and Blocking Communication3.3.1 Design Decisions3.3.2 Start-up and Termination3.3.3 Point-to-Point Communication3.3.4 Testing3.3.5 Benchmarking3.3.6 Exercises3.4 Subset Reimplementation: Nonblocking Communication3.4.1 Redesign Directions3.4.2 Start-up and Termination3.4.3 Point-to-Point Communication3.4.4 Testing3.4.5 Benchmarking3.4.6 Optimization3.4.7 Exercises3.5 Conclusions

Chapter 4: Extensions
4.1 Subset Definition4.1.1 Exercises4.2 Subset Basics4.2.1 Communicator Management4.2.2 Datatypes4.2.3 Collective Operations4.2.4 Error Handling4.2.5 Language Bindings4.2.6 Interdependencies and Steps4.2.7 Exercise4.3 Initial Subset Implementation: Start-up, Data Transfer, and Error Handling4.3.1 Design Decisions4.3.2 Code Refactoring4.3.3 Initialization and Termination4.3.4 Error Handling4.3.5 Process Start-up and Data Transfer4.3.6 Testing4.3.7 Benchmarking4.3.8 Exercises4.4 Subset Extension 1: Communicators, Datatypes, and Collectives4.4.1 Extension Directions4.4.2 Communicator Management4.4.3 Datatypes4.4.4 Point-to-Point Communication4.4.5 Collective Operations4.4.6 Testing4.4.7 Benchmarking4.5 Subset Extension 2: Communicators and Collectives Revisited4.5.1 Extension Directions4.5.2 Communicator Management Revisited4.5.3 Collective Operations Revisited4.5.4 Testing4.5.5 Benchmarking4.5.6 Exercises4.6 Subset Completion: Language Bindings4.6.1 Language Bindings4.6.2 Testing4.6.3 Benchmarking4.6.4 Exercises4.7 Conclusions
Chapter 5: Optimization
5.1 Unifabric Optimization5.1.1 Design Decisions5.1.2 Code Refactoring5.1.3 Start-up and Termination5.1.4 Connection Establishment5.1.5 Point-to-Point Communication5.1.6 Collective Operations5.1.7 Exercises5.2 Multifabric Optimization5.2.1 Design Decisions5.2.2 Code Refactoring5.2.3 Multifabric Device5.2.4 Progress Engine Revisited5.2.5 Start-up and Termination5.2.6 Connection Establishment5.2.7 Point-to-Point Communication5.2.8 Collective Operations5.2.9 Exercises5.3 Conclusions
Chapter 6: Tuning
6.1 Overview6.1.1 Default Settings6.1.2 Static Settings6.1.3 Dynamic Settings6.1.4 Exercises6.2 Static Tuning6.2.1 Design Decisions6.2.2 Parameter Control6.2.3 Process Remapping6.2.4 Exercises6.3 Dynamic Tuning6.3.1 Design Decisions6.3.2 Parameter Control6.3.3 Process Migration6.3.4 Closed Loop Tuning6.3.5 Exercises6.4 Conclusions
Chapter 7: And the Rest
7.1 MPI-1 Features7.1.1 Communication Modes7.1.2 Wildcards7.1.3 Derived Datatypes and Heterogeneity7.1.4 Profiling Interface7.1.5 Other Features7.1.6 Exercises7.2 MPI-2 Features7.2.1 Dynamic Process Management7.2.2 One-Sided Communication7.2.3 Thread Support7.2.4 File I/O7.2.5 Other Features7.2.6 Exercises7.3 MPI-3 Features7.3.1 Fast One-Sided Communication7.3.2 Long Counts7.3.3 Nonblocking Collectives7.3.4 Other Features7.3.5 Exercises7.4 Conclusions
Chapter 8: A Look Ahead
8.1 Selected MPI-4 Features (En Route)8.1.1 Fault Tolerance8.1.2 Threads as MPI Processes8.1.3 Exercises8.2 MPI Limitations8.2.1 Dense Process Rank Sequence8.2.2 Reliable and Pairwise Ordered Data Transfer8.2.3 Message Progress Rules8.2.4 Exercises8.3 Beyond MPI8.3.1 Scalability, Mobility, and IoT Challenges8.3.2 Frequency Wall and Memory Lag8.3.3 Programmability Barrier and Stack Crawling8.3.4 Exercises8.4 Conclusions
Bibliography
Appendix A: Solutions
A.1 OverviewA.2 Shared MemoryA.3 SocketsA.4 ExtensionsA.5 OptimizationA.6 TuningA.7 And the RestA.8 A Look Ahead
Index

Overview

A hands-on guide to writing a Message Passing Interface, this book takes the reader on a tour across major MPI implementations, best optimization techniques, application relevant usage hints, and a historical retrospective of the MPI world, all based on a quarter of a century spent inside MPI. Readers will learn to write MPI implementations from scratch, and to design and optimize communication mechanisms using pragmatic subsetting as the guiding principle. Inside the Message Passing Interface also covers MPI quirks and tricks to achieve best performance.

Dr. Alexander Supalov created the Intel Cluster Tools product line, including the Intel MP Library that he designed and led between 2003 and 2015. He invented the common MPICH ABI and also guided Intel efforts in the MPI Forum during the development of the MPI-2.1, MPI-2.2, and MPI-3 standards. Before that, Alexander designed new finite-element mesh-generation methods, contributing to the PARMACS and PARASOL interfaces, and developed the first full MPI-2 and IMPI implementations in the world. He graduated from the Moscow Institute of Physics and Technology in 1990, and earned his PhD in applied mathematics at the Institute of Numerical Mathematics of the Russian Academy of Sciences in 1995. Alexander holds 26 patents (more pending worldwide).

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Real-Time Agility: The Harmony/ESW Method for Real-Time and Embedded Systems Development

Publisher Resources

ISBN: 9781501506789

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills