Mobile Deep Learning with TensorFlow Lite, ML Kit and Flutter

Mobile Deep Learning with TensorFlow Lite, ML Kit and Flutter

by Anubhav Singh, Rimjhim Bhadani
Released April 2020
Publisher(s): Packt Publishing
ISBN: 9781789611212

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Learn how to deploy effective deep learning solutions on cross-platform applications built using TensorFlow Lite, ML Kit, and Flutter

Key Features

  • Work through projects covering mobile vision, style transfer, speech processing, and multimedia processing
  • Cover interesting deep learning solutions for mobile
  • Build your confidence in training models, performance tuning, memory optimization, and neural network deployment through every project

Book Description

Deep learning is rapidly becoming the most popular topic in the mobile app industry. This book introduces trending deep learning concepts and their use cases with an industrial and application-focused approach. You will cover a range of projects covering tasks such as mobile vision, facial recognition, smart artificial intelligence assistant, augmented reality, and more.

With the help of eight projects, you will learn how to integrate deep learning processes into mobile platforms, iOS, and Android. This will help you to transform deep learning features into robust mobile apps efficiently. You'll get hands-on experience of selecting the right deep learning architectures and optimizing mobile deep learning models while following an application oriented-approach to deep learning on native mobile apps. We will later cover various pre-trained and custom-built deep learning model-based APIs such as machine learning (ML) Kit through Firebase. Further on, the book will take you through examples of creating custom deep learning models with TensorFlow Lite. Each project will demonstrate how to integrate deep learning libraries into your mobile apps, right from preparing the model through to deployment.

By the end of this book, you'll have mastered the skills to build and deploy deep learning mobile applications on both iOS and Android.

What you will learn

  • Create your own customized chatbot by extending the functionality of Google Assistant
  • Improve learning accuracy with the help of features available on mobile devices
  • Perform visual recognition tasks using image processing
  • Use augmented reality to generate captions for a camera feed
  • Authenticate users and create a mechanism to identify rare and suspicious user interactions
  • Develop a chess engine based on deep reinforcement learning
  • Explore the concepts and methods involved in rolling out production-ready deep learning iOS and Android applications

Who this book is for

This book is for data scientists, deep learning and computer vision engineers, and natural language processing (NLP) engineers who want to build smart mobile apps using deep learning methods. You will also find this book useful if you want to improve your mobile app's user interface (UI) by harnessing the potential of deep learning. Basic knowledge of neural networks and coding experience in Python will be beneficial to get started with this book.

Table of contents

  1. Title Page
  2. Copyright and Credits
    1. Mobile Deep Learning with TensorFlow Lite, ML Kit and Flutter
  3. About Packt
    1. Why subscribe?
  4. Contributors
    1. About the authors
    2. About the reviewer
    3. Packt is searching for authors like you
  5. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
      1. Download the example code files
      2. Download the color images
      3. Conventions used
    4. Get in touch
      1. Reviews
  6. Introduction to Deep Learning for Mobile
    1. Growth of AI-powered mobile devices
      1. Changes in hardware to support AI
        1. Why do mobile devices need to have AI chips?
        2. Improved user experience with AI on mobile devices
          1. Personalization
          2. Virtual assistants
          3. Facial recognition
          4. AI-powered cameras
          5. Predictive text
      2. Most popular mobile applications that use AI
        1. Netflix
        2. Seeing AI
        3. Allo
        4. English Language Speech Assistant
        5. Socratic
    2. Understanding machine learning and deep learning
      1. Understanding machine learning
      2. Understanding deep learning
        1. The input layer
        2. The hidden layers
        3. The output layer
        4. The activation function
    3. Introducing some common deep learning architectures
      1. Convolutional neural networks
      2. Generative adversarial networks
      3. Recurrent neural networks
      4. Long short-term memory
    4. Introducing reinforcement learning and NLP
      1. Reinforcement learning
      2. NLP
    5. Methods of integrating AI on Android and iOS
      1. Firebase ML Kit
      2. Core ML
      3. Caffe2
      4. TensorFlow
    6. Summary
  7. Mobile Vision - Face Detection Using On-Device Models
    1. Technical requirements
    2. Introduction to image processing
      1. Understanding images
      2. Manipulating images
        1. Rotation
        2. Grayscale conversion
    3. Developing a face detection application using Flutter
      1. Adding the pub dependencies
      2. Building the application
        1. Creating the first screen
          1. Building the row title
          2. Building the row with button widgets
          3. Creating the whole user interface
        2. Creating the second screen
          1. Getting the image file
          2. Analyzing the image to detect faces
          3. Marking the detected faces
          4. Displaying the final image on the screen
        3. Creating the final MaterialApp
    4. Summary
  8. Chatbot Using Actions on Google
    1. Technical requirements
    2. Understanding the tools available for creating chatbots
      1. Wit.ai
      2. Dialogflow
        1. How does Dialogflow work? 
    3. Creating a Dialogflow account
    4. Creating a Dialogflow agent
    5. Understanding the Dialogflow Console
      1. Creating an Intent and grabbing entities
    6. Creating your first action on Google
      1. Why would you want to build an action on Google?
    7. Creating Actions on a Google project
      1. Creating an integration to the Google Assistant
    8. Implementing a Webhook
    9. Deploying a webhook to Cloud Functions for Firebase
    10. Creating an Action on Google release
    11. Creating the UI for the conversational application
      1. Creating the Text Controller
      2. Creating ChatMessage
    12. Integrating the Dialogflow agent
    13. Adding audio interactions with the assistant
      1. Adding the plugin
      2. Adding SpeechRecognition
      3. Adding the mic button
    14. Summary
  9. Recognizing Plant Species
    1. Technical requirements
    2. Introducing image classification
    3. Understanding the project architecture
    4. Introducing the Cloud Vision API
    5. Configuring the Cloud Vision API for image recognition
      1. Enabling the Cloud Vision API
      2. Creating a Cloud Vision API key
    6. Using an SDK/tools to build a model
      1. Introducing Google's Colaboratory
    7. Creating a custom TensorFlow Lite model for image recognition
    8. Creating a Flutter application
      1. Choosing between two different models
      2. Creating the second screen
        1. Creating the user interface
        2. Adding the functionality
        3. Displaying the chosen image on the screen
    9. Running image recognition
      1. Using the Cloud Vision API
      2. Using an on-device TensorFlow Lite model
      3. Updating the UI with results
    10. Summary
  10. Generating Live Captions from a Camera Feed
    1. Designing the project architecture
    2. Understanding an image caption generator
      1. Understanding the dataset
      2. Building an image caption generation model
        1. Initializing the caption dataset
        2. Preparing the caption dataset
        3. Training
        4. Testing
      3. Creating a simple click-deploy image caption generation model
    3. Understanding the camera plugin
      1. Installing the camera plugin
      2. Adding methods for persistent storage and proper execution
      3. Coding
    4. Creating a camera application
      1. Building the camera preview
    5. Generating image captions from the camera feed
    6. Creating the material app
    7. Summary
  11. Building an Artificial Intelligence Authentication System
    1. Technical requirements
    2. A simple login application
      1. Creating the UI
    3. Adding Firebase authentication
      1. Creating auth.dart
      2. Adding authentication in SignupSigninScreen
      3. Creating the main screen
      4. Creating the home screen
      5. Creating main.dart
    4. Understanding anomaly detection for authentication
    5. A custom model for authenticating users
      1. Building a model for an authentication validity check
      2. Hosting the custom authentication validation model
    6. Implementing ReCaptcha for spam protection
      1. ReCAPTCHA v2
        1. Obtaining the API key
        2. Code integration
    7. Deploying the model in Flutter
    8. Summary
  12. Speech/Multimedia Processing - Generating Music Using AI
    1. Designing the project's architecture
    2. Understanding multimedia processing
      1. Image processing
      2. Audio processing
        1. Magenta
      3. Video processing 
    3. Developing RNN-based models for music generation
      1. Creating the LSTM-based model
      2. Deploying a model using Flask
    4. Deploying an audio generation API on Android and iOS
      1. Creating the UI
      2. Adding Audio Player
      3. Deploying the model
      4. Creating the final material app
    5. Summary
  13. Reinforced Neural Network-Based Chess Engine
    1. Introduction to reinforcement learning
    2. Reinforcement learning in mobile games
    3. Exploring Google's DeepMind
      1. AlphaGo
      2. Alpha Zero
      3. Monte Carlo tree search
    4. Alpha Zero-like AI for Connect 4
      1. Creating a virtual representation of the board
      2. Allowing moves according to the game's rules
      3. The state management system
      4. Facilitating gameplay
      5. Generating sample gameplays
      6. System training
      7. Monte Carlo tree search implementation
      8. Implementing the neural network
    5. Underlying project architecture
    6. Developing a GCP-hosted REST API for the chess engine
      1. Understanding the Universal Chess Interface
      2. Deployment on GCP
        1. Request for a quota increase on GPU instances
        2. Creating a GPU instance
        3. Deploying the script
    7. Creating a simple chess UI on Android
      1. Adding dependencies to pubspec.yaml
      2. Understanding the mapping structure
      3. Placing the images of the actual pieces
      4. Making the pieces movable
    8. Integrating the chess engine API with a UI
      1. Creating the material app
    9. Summary
  14. Building an Image Super-Resolution Application
    1. Basic project architecture
    2. Understanding GANs
    3. Understanding how image super-resolution works
      1. Understanding image resolution
        1. Pixel resolution
        2. Spatial resolution
        3. Temporal resolution
        4. Spectral resolution
        5. Radiometric resolution
      2. Understanding SRGANs
    4. Creating a TensorFlow model for super-resolution
      1. Project directory structure
      2. Creating an SRGAN model for super-resolution
    5. Building the UI for the application
    6. Getting pictures from the device's local storage
    7. Hosting a TensorFlow model on DigitalOcean
      1. Creating a Flask server script
      2. Deploying the Flask script to DigitalOcean Droplet
    8. Integrating a hosted custom model on Flutter
    9. Creating the Material app
    10. Summary
  15. Road Ahead
    1. Understanding recent trends in DL on mobile applications
      1. Math solver
      2. Netflix
      3. Google Maps
      4. Tinder
      5. Snapchat
    2. Exploring the latest developments in DL on mobile devices
      1. Google's MobileNet
      2. Alibaba Mobile Neural Network
    3. Exploring current research areas for DL in mobile apps
      1. Fashion images
      2. Self-Attention Generative Adversarial Networks
      3. Image animation
    4. Summary
  16. Appendix
    1. Setting up a deep learning environment on Cloud VM
      1. Creating a GCP account and enabling billing
      2. Creating a project and GCP Compute Engine instance
      3. Configuring your VM instance to perform deep learning
        1. Installing TensorFlow on a VM
        2. Installing NLTK on a VM and downloading packages
    2. Installing Dart SDK
      1. Windows
      2. macOS
      3. Linux
    3. Installing Flutter SDK
      1. Windows
      2. macOS
      3. Linux
    4. Configuring Firebase
      1. Creating a Firebase project
      2. Configuring the Android project
      3. Configuring the iOS project
    5. Setting up VS Code
      1. Installing the Flutter and Dart plugins
      2. Validating the setup with flutter doctor
      3. Creating the first Flutter app
      4. Running the app
      5. Trying hot reload
  17. Other Books You May Enjoy
    1. Leave a review - let other readers know what you think

Product information

  • Title: Mobile Deep Learning with TensorFlow Lite, ML Kit and Flutter
  • Author(s): Anubhav Singh, Rimjhim Bhadani
  • Release date: April 2020
  • Publisher(s): Packt Publishing
  • ISBN: 9781789611212