TinyML

Table of contents

1. Preface
   1. Conventions Used in This Book
   2. Using Code Examples
   3. O’Reilly Online Learning
   4. How to Contact Us
   5. Acknowledgments
2. 1. Introduction
   1. Embedded Devices
   2. Changing Landscape
3. 2. Getting Started
   1. Who Is This Book Aimed At?
   2. What Hardware Do You Need?
   3. What Software Do You Need?
   4. What Do We Hope You’ll Learn?
4. 3. Getting Up to Speed on Machine Learning
   1. What Machine Learning Actually Is
   2. The Deep Learning Workflow
      1. Decide on a Goal
      2. Collect a Dataset
      3. Design a Model Architecture
      4. Train the Model
      5. Convert the Model
      6. Run Inference
      7. Evaluate and Troubleshoot
   3. Wrapping Up
5. 4. The “Hello World” of TinyML: Building and Training a Model
   1. What We’re Building
   2. Our Machine Learning Toolchain
      1. Python and Jupyter Notebooks
      2. Google Colaboratory
      3. TensorFlow and Keras
   3. Building Our Model
      1. Importing Dependencies
      2. Generating Data
      3. Splitting the Data
      4. Defining a Basic Model
   4. Training Our Model
      1. Training Metrics
      2. Graphing the History
      3. Improving Our Model
      4. Testing
   5. Converting the Model for TensorFlow Lite
      1. Converting to a C File
   6. Wrapping Up
6. 5. The “Hello World” of TinyML: Building an Application
   1. Walking Through the Tests
      1. Including Dependencies
      2. Setting Up the Test
      3. Getting Ready to Log Data
      4. Mapping Our Model
      5. Creating an AllOpsResolver
      6. Defining a Tensor Arena
      7. Creating an Interpreter
      8. Inspecting the Input Tensor
      9. Running Inference on an Input
      10. Reading the Output
      11. Running the Tests
   2. Project File Structure
   3. Walking Through the Source
      1. Starting with main_functions.cc
      2. Handling Output with output_handler.cc
      3. Wrapping Up main_functions.cc
      4. Understanding main.cc
      5. Running Our Application
   4. Wrapping Up
7. 6. The “Hello World” of TinyML: Deploying to Microcontrollers
   1. What Exactly Is a Microcontroller?
   2. Arduino
      1. Handling Output on Arduino
      2. Running the Example
      3. Making Your Own Changes
   3. SparkFun Edge
      1. Handling Output on SparkFun Edge
      2. Running the Example
      3. Testing the Program
      4. Viewing Debug Data
      5. Making Your Own Changes
   4. ST Microelectronics STM32F746G Discovery Kit
      1. Handling Output on STM32F746G
      2. Running the Example
      3. Making Your Own Changes
   5. Wrapping Up
8. 7. Wake-Word Detection: Building an Application
   1. What We’re Building
   2. Application Architecture
      1. Introducing Our Model
      2. All the Moving Parts
   3. Walking Through the Tests
      1. The Basic Flow
      2. The Audio Provider
      3. The Feature Provider
      4. The Command Recognizer
      5. The Command Responder
   4. Listening for Wake Words
      1. Running Our Application
   5. Deploying to Microcontrollers
      1. Arduino
      2. SparkFun Edge
      3. ST Microelectronics STM32F746G Discovery Kit
   6. Wrapping Up
9. 8. Wake-Word Detection: Training a Model
   1. Training Our New Model
      1. Training in Colab
   2. Using the Model in Our Project
      1. Replacing the Model
      2. Updating the Labels
      3. Updating command_responder.cc
      4. Other Ways to Run the Scripts
   3. How the Model Works
      1. Visualizing the Inputs
      2. How Does Feature Generation Work?
      3. Understanding the Model Architecture
      4. Understanding the Model Output
   4. Training with Your Own Data
      1. The Speech Commands Dataset
      2. Training on Your Own Dataset
      3. How to Record Your Own Audio
      4. Data Augmentation
      5. Model Architectures
   5. Wrapping Up
10. 9. Person Detection: Building an Application
    1. What We’re Building
    2. Application Architecture
       1. Introducing Our Model
       2. All the Moving Parts
    3. Walking Through the Tests
       1. The Basic Flow
       2. The Image Provider
       3. The Detection Responder
    4. Detecting People
    5. Deploying to Microcontrollers
       1. Arduino
       2. SparkFun Edge
    6. Wrapping Up
11. 10. Person Detection: Training a Model
    1. Picking a Machine
    2. Setting Up a Google Cloud Platform Instance
    3. Training Framework Choice
    4. Building the Dataset
    5. Training the Model
    6. TensorBoard
    7. Evaluating the Model
    8. Exporting the Model to TensorFlow Lite
       1. Exporting to a GraphDef Protobuf File
       2. Freezing the Weights
       3. Quantizing and Converting to TensorFlow Lite
       4. Converting to a C Source File
    9. Training for Other Categories
    10. Understanding the Architecture
    11. Wrapping Up
12. 11. Magic Wand: Building an Application
    1. What We’re Building
    2. Application Architecture
       1. Introducing Our Model
       2. All the Moving Parts
    3. Walking Through the Tests
       1. The Basic Flow
       2. The Accelerometer Handler
       3. The Gesture Predictor
       4. The Output Handler
    4. Detecting Gestures
    5. Deploying to Microcontrollers
       1. Arduino
       2. SparkFun Edge
    6. Wrapping Up
13. 12. Magic Wand: Training a Model
    1. Training a Model
       1. Training in Colab
       2. Other Ways to Run the Scripts
    2. How the Model Works
       1. Visualizing the Input
       2. Understanding the Model Architecture
    3. Training with Your Own Data
       1. Capturing Data
       2. Modifying the Training Scripts
       3. Training
       4. Using the New Model
    4. Wrapping Up
       1. Learning Machine Learning
       2. What’s Next
14. 13. TensorFlow Lite for Microcontrollers
    1. What Is TensorFlow Lite for Microcontrollers?
       1. TensorFlow
       2. TensorFlow Lite
       3. TensorFlow Lite for Microcontrollers
       4. Requirements
       5. Why Is the Model Interpreted?
       6. Project Generation
    2. Build Systems
       1. Specializing Code
       2. Makefiles
       3. Writing Tests
    3. Supporting a New Hardware Platform
       1. Printing to a Log
       2. Implementing DebugLog()
       3. Running All the Targets
       4. Integrating with the Makefile Build
    4. Supporting a New IDE or Build System
    5. Integrating Code Changes Between Projects and Repositories
    6. Contributing Back to Open Source
    7. Supporting New Hardware Accelerators
    8. Understanding the File Format
       1. FlatBuffers
    9. Porting TensorFlow Lite Mobile Ops to Micro
       1. Separate the Reference Code
       2. Create a Micro Copy of the Operator
       3. Port the Test to the Micro Framework
       4. Build a Bazel Test
       5. Add Your Op to AllOpsResolver
       6. Build a Makefile Test
    10. Wrapping Up
15. 14. Designing Your Own TinyML Applications
    1. The Design Process
    2. Do You Need a Microcontroller, or Would a Larger Device Work?
    3. Understanding What’s Possible
    4. Follow in Someone Else’s Footsteps
    5. Find Some Similar Models to Train
    6. Look at the Data
    7. Wizard of Oz-ing
    8. Get It Working on the Desktop First
16. 15. Optimizing Latency
    1. First Make Sure It Matters
    2. Hardware Changes
    3. Model Improvements
       1. Estimating Model Latency
       2. How to Speed Up Your Model
    4. Quantization
    5. Product Design
    6. Code Optimizations
       1. Performance Profiling
    7. Optimizing Operations
       1. Look for Implementations That Are Already Optimized
       2. Write Your Own Optimized Implementation
       3. Taking Advantage of Hardware Features
       4. Accelerators and Coprocessors
    8. Contributing Back to Open Source
    9. Wrapping Up
17. 16. Optimizing Energy Usage
    1. Developing Intuition
       1. Typical Component Power Usage
       2. Hardware Choice
    2. Measuring Real Power Usage
    3. Estimating Power Usage for a Model
    4. Improving Power Usage
       1. Duty Cycling
       2. Cascading Design
    5. Wrapping Up
18. 17. Optimizing Model and Binary Size
    1. Understanding Your System’s Limits
    2. Estimating Memory Usage
       1. Flash Usage
       2. RAM Usage
    3. Ballpark Figures for Model Accuracy and Size on Different Problems
       1. Speech Wake-Word Model
       2. Accelerometer Predictive Maintenance Model
       3. Person Presence Detection
    4. Model Choice
    5. Reducing the Size of Your Executable
       1. Measuring Code Size
       2. How Much Space Is Tensorflow Lite for Microcontrollers Taking?
       3. OpResolver
       4. Understanding the Size of Individual Functions
       5. Framework Constants
    6. Truly Tiny Models
    7. Wrapping Up
19. 18. Debugging
    1. Accuracy Loss Between Training and Deployment
       1. Preprocessing Differences
       2. Debugging Preprocessing
       3. On-Device Evaluation
    2. Numerical Differences
       1. Are the Differences a Problem?
       2. Establish a Metric
       3. Compare Against a Baseline
       4. Swap Out Implementations
    3. Mysterious Crashes and Hangs
       1. Desktop Debugging
       2. Log Tracing
       3. Shotgun Debugging
       4. Memory Corruption
    4. Wrapping Up
20. 19. Porting Models from TensorFlow to TensorFlow Lite
    1. Understand What Ops Are Needed
    2. Look at Existing Op Coverage in Tensorflow Lite
    3. Move Preprocessing and Postprocessing into Application Code
    4. Implement Required Ops if Necessary
    5. Optimize Ops
    6. Wrapping Up
21. 20. Privacy, Security, and Deployment
    1. Privacy
       1. The Privacy Design Document
       2. Using a PDD
    2. Security
       1. Protecting Models
    3. Deployment
       1. Moving from a Development Board to a Product
    4. Wrapping Up
22. 21. Learning More
    1. The TinyML Foundation
    2. SIG Micro
    3. The TensorFlow Website
    4. Other Frameworks
    5. Twitter
    6. Friends of TinyML
    7. Wrapping Up
23. A. Using and Generating an Arduino Library Zip
24. B. Capturing Audio on Arduino
25. Index