Scaling Machine Learning with Spark

Book description

Get up to speed on Apache Spark, the popular engine for large-scale data processing, including machine learning and analytics. If you're looking to expand your skill set or advance your career in scalable machine learning with MLlib, distributed PyTorch, and distributed TensorFlow, this practical guide is for you. Using Spark as your main data processing platform, you'll discover several open source technologies designed and built for enriching Spark's ML capabilities.

Scaling Machine Learning with Spark examines various technologies for building end-to-end distributed ML workflows based on the Apache Spark ecosystem with Spark MLlib, MLFlow, TensorFlow, PyTorch, and Petastorm. This book shows you when to use each technology and why. If you're a data scientist working with machine learning, you'll learn how to:

  • Build practical distributed machine learning workflows, including feature engineering and data formats
  • Extend deep learning functionalities beyond Spark by bridging into distributed TensorFlow and PyTorch
  • Manage your machine learning experiment lifecycle with MLFlow
  • Use Petastorm as a storage layer for bridging data from Spark into TensorFlow and PyTorch
  • Use machine learning terminology to understand distribution strategies

Publisher resources

View/Submit Errata

Table of contents

  1. 1. Managing the ML Experiments Lifecycle with MLFlow
    1. What Is MLflow
      1. Software Components of MLFlow Platform
    2. Use of MLflow in Different Organizations
    3. Logic Components of MLflow
      1. MLflow Tracking
      2. MLflow Projects
      3. MLflow Models
      4. MLflow Registry
    4. Understanding MLFlow Tracking Servers for Large Scale
    5. ML Experiments Anti-Pattern
    6. Summary
  2. 2. Data Ingestion, Preprocessing, and Data Statistics
    1. Data Ingestion with Spark
      1. Working with images
      2. Working with Tabular Data
    2. Preprocessing Data
      1. Preprocessing vs Processing data
      2. Why Preprocess the Data?
      3. Data Structures
      4. MLlib Data Types
      5. Preprocessing with MLlib Transformers
      6. Working with Text Data
      7. Preprocessing Images Data
      8. Save the data and avoid small files problem
    3. Descriptive Statistics: Getting a Feel for the Data
      1. Calculating Statistics
      2. Descriptive Statistics with Spark Summarizer
      3. Correlation
      4. Pearson Correlation
      5. Spearman Correlation
    4. Summary
  3. 3. Feature Engineering
    1. Features and Their Impact on Models
    2. The MLlib Tools
      1. Extractors
      2. Selectors
    3. Image Featurization Process
      1. Understanding Image Manipulation
      2. Extracting Features Leveraging Spark APIs
    4. Text Featurization Process
      1. Extracting Features
    5. Enriching the Dataset
    6. Summary
  4. 4. Training Models with Spark MLlib
    1. Algorithms
    2. Supervised Machine Learning
      1. Classification
      2. Indexed labels and vectors
      3. Regression
    3. Unsupervised Machine Learning
      1. Frequent Pattern Mining
    4. Evaluating
      1. Evaluator
      2. Supervised Evaluator
      3. Unsupervised Evaluator
    5. Hyperparameters and Tuning Experiments
      1. Simple Split of Train-Test Sets
      2. Cross Validation: A Better Way to Test Your Models
    6. ML Pipelines
      1. Constructing
      2. How Does Split Work with Pipeline API?
    7. Persistence
    8. Summary
  5. 5. Bridging Spark and Deep Learning Frameworks
    1. Two Clusters Approach
    2. Dedicated Data Access Layer
      1. Selecting a Data Access Layer
      2. Can Alluxio fit the bill?
      3. What about cache? Is it critical for training ML?
    3. What Is Petastorm?
      1. Petastrom SparkDatasetConverter
      2. Petastorm as a Parquet store
    4. Spark Hydrogen Project
      1. Barrier Execution Mode
      2. Accelerator-Aware Scheduling
    5. Introducing Horovod’s Estimator API
    6. Summary
  6. About the Author

Product information

  • Title: Scaling Machine Learning with Spark
  • Author(s): Adi Polak
  • Release date: February 2023
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781098106829