Inside the Message Passing Interface

Inside the Message Passing Interface

by Alexander Supalov
Released September 2018
Publisher(s): De Gruyter
ISBN: 9781501506789

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

A hands-on guide to writing a Message Passing Interface, this book takes the reader on a tour across major MPI implementations, best optimization techniques, application relevant usage hints, and a historical retrospective of the MPI world, all based on a quarter of a century spent inside MPI. Readers will learn to write MPI implementations from scratch, and to design and optimize communication mechanisms using pragmatic subsetting as the guiding principle. Inside the Message Passing Interface also covers MPI quirks and tricks to achieve best performance.

Dr. Alexander Supalov created the Intel Cluster Tools product line, including the Intel MP Library that he designed and led between 2003 and 2015. He invented the common MPICH ABI and also guided Intel efforts in the MPI Forum during the development of the MPI-2.1, MPI-2.2, and MPI-3 standards. Before that, Alexander designed new finite-element mesh-generation methods, contributing to the PARMACS and PARASOL interfaces, and developed the first full MPI-2 and IMPI implementations in the world. He graduated from the Moscow Institute of Physics and Technology in 1990, and earned his PhD in applied mathematics at the Institute of Numerical Mathematics of the Russian Academy of Sciences in 1995. Alexander holds 26 patents (more pending worldwide).

Table of contents

  1. Title Page
  2. Copyright
  3. Dedication
  4. Acknowledgments
  5. Contents
  6. Preface
  7. Introduction
  8. Chapter 1: Overview
    1. 1.1 Parallel Computer
      1. 1.1.1 Intraprocessor Parallelism
      2. 1.1.2 Interprocessor Parallelism
      3. 1.1.3 Exercises
    2. 1.2 MPI Standard
      1. 1.2.1 MPI History
      2. 1.2.2 Related Standards
      3. 1.2.3 Exercises
    3. 1.3 MPI Subsetting
      1. 1.3.1 Motivation
      2. 1.3.2 Sensible MPI Subsets
      3. 1.3.3 Exercises
    4. 1.4 Fine Points
      1. 1.4.1 Programming
      2. 1.4.2 Code Management
      3. 1.4.3 Building
      4. 1.4.4 Testing and Debugging
      5. 1.4.5 Benchmarking
      6. 1.4.6 Presentation
      7. 1.4.7 Optimization
      8. 1.4.8 Issue Tracking
      9. 1.4.9 Exercises
  9. Chapter 2: Shared Memory
    1. 2.1 Subset Definition
      1. 2.1.1 Exercises
    2. 2.2 Communication Basics
      1. 2.2.1 Intraprocess Communication
      2. 2.2.2 Interprocess Communication
      3. 2.2.3 Ping-Pong Pattern Revisited
      4. 2.2.4 Exercises
    3. 2.3 Subset Implementation: Blocking Communication
      1. 2.3.1 Design Decisions
      2. 2.3.2 Start-up and Termination
      3. 2.3.3 Blocking Point-to-Point Communication
      4. 2.3.4 Testing
      5. 2.3.5 Benchmarking
      6. 2.3.6 Optimization
      7. 2.3.7 Exercises
    4. 2.4 Conclusions
  10. Chapter 3: Sockets
    1. 3.1 Subset Definition
      1. 3.1.1 Exercises
    2. 3.2 Communication Basics
      1. 3.2.1 Intranode Communication
      2. 3.2.2 Internode Communication
      3. 3.2.3 Exercises
    3. 3.3 Initial Subset Implementation: Start-up and Blocking Communication
      1. 3.3.1 Design Decisions
      2. 3.3.2 Start-up and Termination
      3. 3.3.3 Point-to-Point Communication
      4. 3.3.4 Testing
      5. 3.3.5 Benchmarking
      6. 3.3.6 Exercises
    4. 3.4 Subset Reimplementation: Nonblocking Communication
      1. 3.4.1 Redesign Directions
      2. 3.4.2 Start-up and Termination
      3. 3.4.3 Point-to-Point Communication
      4. 3.4.4 Testing
      5. 3.4.5 Benchmarking
      6. 3.4.6 Optimization
      7. 3.4.7 Exercises
    5. 3.5 Conclusions
  11. Chapter 4: Extensions
    1. 4.1 Subset Definition
      1. 4.1.1 Exercises
    2. 4.2 Subset Basics
      1. 4.2.1 Communicator Management
      2. 4.2.2 Datatypes
      3. 4.2.3 Collective Operations
      4. 4.2.4 Error Handling
      5. 4.2.5 Language Bindings
      6. 4.2.6 Interdependencies and Steps
      7. 4.2.7 Exercise
    3. 4.3 Initial Subset Implementation: Start-up, Data Transfer, and Error Handling
      1. 4.3.1 Design Decisions
      2. 4.3.2 Code Refactoring
      3. 4.3.3 Initialization and Termination
      4. 4.3.4 Error Handling
      5. 4.3.5 Process Start-up and Data Transfer
      6. 4.3.6 Testing
      7. 4.3.7 Benchmarking
      8. 4.3.8 Exercises
    4. 4.4 Subset Extension 1: Communicators, Datatypes, and Collectives
      1. 4.4.1 Extension Directions
      2. 4.4.2 Communicator Management
      3. 4.4.3 Datatypes
      4. 4.4.4 Point-to-Point Communication
      5. 4.4.5 Collective Operations
      6. 4.4.6 Testing
      7. 4.4.7 Benchmarking
    5. 4.5 Subset Extension 2: Communicators and Collectives Revisited
      1. 4.5.1 Extension Directions
      2. 4.5.2 Communicator Management Revisited
      3. 4.5.3 Collective Operations Revisited
      4. 4.5.4 Testing
      5. 4.5.5 Benchmarking
      6. 4.5.6 Exercises
    6. 4.6 Subset Completion: Language Bindings
      1. 4.6.1 Language Bindings
      2. 4.6.2 Testing
      3. 4.6.3 Benchmarking
      4. 4.6.4 Exercises
    7. 4.7 Conclusions
  12. Chapter 5: Optimization
    1. 5.1 Unifabric Optimization
      1. 5.1.1 Design Decisions
      2. 5.1.2 Code Refactoring
      3. 5.1.3 Start-up and Termination
      4. 5.1.4 Connection Establishment
      5. 5.1.5 Point-to-Point Communication
      6. 5.1.6 Collective Operations
      7. 5.1.7 Exercises
    2. 5.2 Multifabric Optimization
      1. 5.2.1 Design Decisions
      2. 5.2.2 Code Refactoring
      3. 5.2.3 Multifabric Device
      4. 5.2.4 Progress Engine Revisited
      5. 5.2.5 Start-up and Termination
      6. 5.2.6 Connection Establishment
      7. 5.2.7 Point-to-Point Communication
      8. 5.2.8 Collective Operations
      9. 5.2.9 Exercises
    3. 5.3 Conclusions
  13. Chapter 6: Tuning
    1. 6.1 Overview
      1. 6.1.1 Default Settings
      2. 6.1.2 Static Settings
      3. 6.1.3 Dynamic Settings
      4. 6.1.4 Exercises
    2. 6.2 Static Tuning
      1. 6.2.1 Design Decisions
      2. 6.2.2 Parameter Control
      3. 6.2.3 Process Remapping
      4. 6.2.4 Exercises
    3. 6.3 Dynamic Tuning
      1. 6.3.1 Design Decisions
      2. 6.3.2 Parameter Control
      3. 6.3.3 Process Migration
      4. 6.3.4 Closed Loop Tuning
      5. 6.3.5 Exercises
    4. 6.4 Conclusions
  14. Chapter 7: And the Rest
    1. 7.1 MPI-1 Features
      1. 7.1.1 Communication Modes
      2. 7.1.2 Wildcards
      3. 7.1.3 Derived Datatypes and Heterogeneity
      4. 7.1.4 Profiling Interface
      5. 7.1.5 Other Features
      6. 7.1.6 Exercises
    2. 7.2 MPI-2 Features
      1. 7.2.1 Dynamic Process Management
      2. 7.2.2 One-Sided Communication
      3. 7.2.3 Thread Support
      4. 7.2.4 File I/O
      5. 7.2.5 Other Features
      6. 7.2.6 Exercises
    3. 7.3 MPI-3 Features
      1. 7.3.1 Fast One-Sided Communication
      2. 7.3.2 Long Counts
      3. 7.3.3 Nonblocking Collectives
      4. 7.3.4 Other Features
      5. 7.3.5 Exercises
    4. 7.4 Conclusions
  15. Chapter 8: A Look Ahead
    1. 8.1 Selected MPI-4 Features (En Route)
      1. 8.1.1 Fault Tolerance
      2. 8.1.2 Threads as MPI Processes
      3. 8.1.3 Exercises
    2. 8.2 MPI Limitations
      1. 8.2.1 Dense Process Rank Sequence
      2. 8.2.2 Reliable and Pairwise Ordered Data Transfer
      3. 8.2.3 Message Progress Rules
      4. 8.2.4 Exercises
    3. 8.3 Beyond MPI
      1. 8.3.1 Scalability, Mobility, and IoT Challenges
      2. 8.3.2 Frequency Wall and Memory Lag
      3. 8.3.3 Programmability Barrier and Stack Crawling
      4. 8.3.4 Exercises
    4. 8.4 Conclusions
  16. Bibliography
  17. Appendix A: Solutions
    1. A.1 Overview
    2. A.2 Shared Memory
    3. A.3 Sockets
    4. A.4 Extensions
    5. A.5 Optimization
    6. A.6 Tuning
    7. A.7 And the Rest
    8. A.8 A Look Ahead
  18. Index

Product information

  • Title: Inside the Message Passing Interface
  • Author(s): Alexander Supalov
  • Release date: September 2018
  • Publisher(s): De Gruyter
  • ISBN: 9781501506789