book

HPC, Big Data, and AI Convergence Towards Exascale

Name: HPC, Big Data, and AI Convergence Towards Exascale
ISBN: 9781000485172

by Olivier Terzo, Jan Martinovič

January 2022

Intermediate to advanced

322 pages

English

CRC Press

Read now

Unlock full access

Cover
Half-Title Page
Title Page
Copyright Page
Contents
Foreword by Anders Dam Jensen – EuroHPC
Foreword by Jean-Pierre Panziera – ETP4HPC
Preface by Vít Vondrák – IT4Innovations
Preface by Marco Mezzalama – Links Foundation
Acknowledgments

Editors
Contributors
1 Toward the Convergence of High-Performance Computing, Cloud, and Big Data Domains
1.1 Introduction1.1.1 History of Cloud Computing1.1.2 History of HPC1.1.3 Evolution of Big Data1.1.4 Evolution of Big Data Storage and Tools1.2 Exploiting Convergence1.2.1 CYBELE Project1.2.2 DeepHealth Project1.2.3 EVOLVE Project1.2.4 LEXIS ProjectAcknowledgmentReferences
2 The LEXIS Platform for Distributed Workflow Execution and Data Management
2.1 Motivation2.2 Architecture (Codesign) and Interfaces2.3 Security2.4 Accounting and Billing2.5 Easy Access to HPC/Cloud through a Specialized Web Portal2.6 Market Analysis2.6.1 LEXIS Project ImpactAcknowledgmentReferences
3 Enabling the HPC and Artificial Intelligence Cross-Stack Convergence at the Exascale Level
3.1 Introduction3.2 The Rise of Convergent Infrastructures3.3 The ACROSS Approach to the HPC, Big Data, and AI Convergence3.3.1 Heterogeneous Infrastructural Support3.3.2 The Management of the Convergent Platform3.4 Related Works3.5 ConclusionsAcknowledgmentNotesBibliography
4 Data System and Data Management in a Federation of HPC/Cloud Centers
4.1 Introduction: Data Federation of European HPC/Cloud Centers4.2 Requirements on the LEXIS DDI4.2.1 Unified Data Access4.2.2 Usage and Federation of Diverse Data Backend Systems4.2.3 Reliability and Redundancy4.2.4 AAI Support4.2.5 APIs4.2.6 State-of-the-art Research Data Management4.3 Federation via a DDI Based on iRODS4.3.1 Relevant Basic Properties of iRODS4.3.2 iRODS HA Setup4.3.3 iRODS Zones Federation across Centers and Data Movement4.3.4 Storage Tiering and Underlying Data Storage4.3.5 Logical Structure of the DDI4.4 Hardware4.4.1 Storage Systems for HPC and Infrastructure-as-a-Service- Cloud Clusters4.4.2 Storage Systems Dedicated to LEXIS4.4.3 HPC–Cloud-Storage Interconnect and Data Node/Burst Buffer Concept4.4.3.1 SBF (Smart Bunch of Flash)4.4.3.2 SBB4.5 Unified Access to the Platform Based on an AAI4.5.1 LEXIS Identity and Access Management (IAM) Solution, SSO, and AAI4.5.2 Platform Services vs. AAI: Separation of Concerns4.5.3 LEXIS DDI and IAM/AAI System4.6 Data Management via APIs4.6.1 Data Search, Upload, and Download APIs4.6.2 Staging API4.6.3 Replication and PID Assignment API4.6.4 Helper APIs4.6.5 Compression/Decompression/Encryption/Decryption API4.7 Integration with EUDAT Services4.7.1 EUDAT B2HANDLE4.7.2 EUDAT B2SAFE4.7.3 EUDAT B2STAGE4.8 ConclusionAcknowledgmentReferences
5 Distributed HPC Resources Orchestration for Supporting Large-Scale Workflow Execution
5.1 Introduction5.2 Federated Execution Platforms5.3 WMSs and Implementation in LEXIS5.3.1 Dynamic Workflow Orchestration5.3.2 Resource Management Metrics5.4 Workflow Data Management5.5 LEXIS Pilot Use Cases and Orchestration5.6 Related Works5.7 ConclusionAcknowledgmentNotesBibliography
6 Advanced Engineering Platform Supporting CFD Simulations of Aeronautical Engine Critical Parts
6.1 Introduction: Background and LEXIS Aeronautics Pilot6.2 Engineering Case Studies in the LEXIS Aeronautics Pilot6.3 The Turbomachinery Case Study6.3.1 Engineering Context6.3.2 Digital Technology Deployment6.3.2.1 Application Workflow6.3.2.2 Main Application Software and HW Resources6.3.3 First Results6.3.4 Benefit–Cost Analysis of HW Acceleration6.3.5 Next Steps6.4 The Rotating Parts Case Study6.4.1 Engineering Context6.4.2 Digital Technology Deployment6.4.2.1 Application Workflow6.4.2.2 Main Application Software and HW Resources6.4.3 First Results6.4.3.1 SPH Liquid-Phase Simulation6.4.3.2 SPH Gas-Phase Simulation6.4.4 Next Steps6.5 Final remarksAcknowledgmentNotesReferences
7 Event-Driven, Time-Constrained Workflows: An Earthquake and Tsunami Pilot
7.1 Introduction7.2 Event-Driven, Time-Constrained Workflows7.2.1 Requirements7.2.2 Background7.2.3 Overall View of the Workflow7.3 Workflow Components7.3.1 Shakemap and Exposure Dataset7.3.2 Tsunami Simulations7.3.3 SEM7.4 Technological Layers7.4.1 Technology Layer 1: Orchestration7.4.2 Technology Layer 2: Heterogeneous Compute7.4.3 Technology Layer 3: Data7.5 ConclusionAcknowledgmentNoteReferences
8 Exploitation of Multiple Model Layers within LEXIS Weather and Climate Pilot: An HPC-Based Approach
8.1 Introduction: Background and Driving Forces8.2 The Weather and Climate Pilot8.3 Observational Data8.4 LEXIS DDI and Weather and Climate Data API8.5 LEXIS Orchestration System8.6 Weather and Climate Pilot Workflows8.6.1 WRF–ERDS Workflow Examples8.7 ConclusionAcknowledgmentReferences
9 Data Convergence for High-Performance Cloud
9.1 Introduction9.2 Motivations9.3 Design and Implementation9.4 Karvdash9.5 DataShim9.5.1 Overview9.5.2 Dataset Custom Resource Definition9.5.3 DatasetInternal Custom Resource Definition9.5.4 DataShim Operator and Admission Controller9.5.5 Caching Plugin9.5.6 Objects Caching on CEPH9.5.7 Ceph-Based Caching Plugin Implementation9.5.8 Evaluation of the Ceph-Based Caching Plugin9.6 H39.6.1 Overview9.6.2 Data and Metadata Organization9.6.3 The H3 Ecosystem9.7 Integration9.8 Related Work9.9 ConclusionsNoteReferences
10 The DeepHealth HPC Infrastructure: Leveraging Heterogenous HPC and Cloud-Computing Infrastructures for IA-Based Medical Solutions
10.1 Introduction10.2 The Parallel Execution of EDDL Operations10.2.1 COMPSs10.2.2 StreamFlow10.3 Cloud Infrastructures10.3.1 Hybrid Cloud10.3.2 Parallel Execution on Cloud Environments10.3.2.1 Parallel Cloud Execution Based on COMPSs10.3.2.2 Parallel Cloud Execution Based on StreamFlow10.4 Acceleration Devices: GPU and FPGAs10.4.1 FPGA Acceleration10.4.1.1 The DeepHealth FPGA Infrastructure10.4.1.2 An Optimized FPGA Board Design for DL10.4.1.3 FPGA-Based Algorithms10.4.2 Many Core and GPU Acceleration10.5 ConclusionsNotes
11 Applications of AI and HPC in the Health Domain
11.1 Introduction11.2 AI and HPC in the Health Domain in 202011.3 DeepHealth Concept11.4 DeepHealth Use Cases11.5 Use of HPC and Cloud in Medical Pilots11.5.1 UC2 – UNITOPatho11.5.2 UC3 – UNITOBrain11.5.3 UC4 – Chest11.5.4 UC5 – UNITO Deep Image Annotation11.5.5 UC12 –Skin Cancer Melanoma Detection11.6 DeepHealth Value Proposition11.7 ConclusionsNotes
12 CYBELE: On the Convergence of HPC, Big Data Services, and AI Technologies
12.1 Introduction: Background and Driving Forces12.2 Identified Gaps: Motivating the CYBELE Vision12.3 Materializing the Solution: Convergence of HPC, Big Data, and AI12.3.1 Data and Infrastructure Access Security Layer12.3.2 Embedded Experiments Composition Layer12.3.3 Parallel and Distributed Execution Management Layer12.3.4 Data Services Layer12.3.5 Visualization and Reporting Layer12.4 Key Takeaways and ConclusionsNoteReferences
13 CYBELE:: A Hybrid Architecture of HPC and Big Data for AI Applications in Agriculture
13.1 Introduction: Vision and Challenges13.2 Background13.2.1 AI in Big Data Analytics on Cloud13.2.2 AI on HPC Systems13.3 Hybrid Big Data and HPC Resource for AI Applications in CYBELE13.4 Parallelization and Deployment of AI Applications on HPC Systems13.4.1 Pilot Soybean Farming13.4.1.1 Pilot Description13.4.1.2 Application Parallelization for HPC Systems13.4.2 Pilot Wheat Ear13.4.2.1 Pilot Description13.4.2.2 Application Parallelization for HPC Systems13.5 Performance Evaluation for Pilot Soybean Farming and Pilot Wheat Ear13.6 Discussion13.7 Conclusion Remarks and Future WorksAcknowledgmentsNotesReferences
14 European Processor Initiative: Europe’s Approach to Exascale Computing
14.1 Introduction14.2 European Processor Initiative14.2.1 Global Technical Panstream14.2.2 GPP Stream14.2.3 Accelerator Stream14.2.4 Automotive Stream14.3 ConclusionAcknowledgmentBibliography
Index

Content preview from HPC, Big Data, and AI Convergence Towards Exascale

Foreword

The convergence of HPC, big data, and AI on their trajectory toward exascale is very close to the heart of ETP4HPC, the European High-Performance Computing Technology Platform. We began work on this in 2017 by comparing the system stacks of HPC and big data and identifying the commonalities and deviations that needed to be addressed. This collaboration grew into an alliance with Europe’s Big Data Value Association, which continued for a number of years and the work of which was present in two issues of our HPC Strategic Research Agenda (SRA).

At present, most of the key players of the European Digital Technology ecosystem are part of the TransContinuum Initiative (TCI), which is led by our team. It is a forum where long- and medium-term ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Using HPC for Computational Fluid Dynamics

Publisher Resources

ISBN: 9781000485172

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

HPC, Big Data, and AI Convergence Towards Exascale

by Olivier Terzo, Jan Martinovič

Foreword

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.