Deep Learning and the Game of Go

Deep Learning and the Game of Go

by Kevin Ferguson, Max Pumperla
Released January 2019
Publisher(s): Manning Publications
ISBN: 9781617295324

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Deep Learning and the Game of Go teaches you how to apply the power of deep learning to complex reasoning tasks by building a Go-playing AI. After exposing you to the foundations of machine and deep learning, you'll use Python to build a bot and then teach it the rules of the game.



About the Technology

The ancient strategy game of Go is an incredible case study for AI. In 2016, a deep learning–based system shocked the Go world by defeating a world champion. Shortly after that, the upgraded AlphaGo Zero crushed the original bot by using deep reinforcement learning to master the game. Now, you can learn those same deep learning techniques by building your own Go bot!



About the Book

Deep Learning and the Game of Go introduces deep learning by teaching you to build a Go-winning bot. As you progress, you’ll apply increasingly complex training techniques and strategies using the Python deep learning library Keras. You’ll enjoy watching your bot master the game of Go, and along the way, you’ll discover how to apply your new deep learning skills to a wide range of other scenarios!



What's Inside

  • Build and teach a self-improving game AI
  • Enhance classical game AI systems with deep learning
  • Implement neural networks for deep learning


About the Reader

All you need are basic Python skills and high school–level math. No deep learning experience required.



About the Authors

Max Pumperla and Kevin Ferguson are experienced deep learning specialists skilled in distributed systems and data science. Together, Max and Kevin built the open source bot BetaGo.

We interviewed Kevin as a part of our Six Questions series. Check it out here.



Quotes
Using the game of Go as a way to teach machine learning is inspired and inspiring. Highly recommended!
- Burk Hufnagel, Daugherty Business Solutions

Brilliant introduction to the most exciting technology of our time.
- Helmut Hauschild, HSEC

Great Pythonic code that is a pleasure to read.
- Domingo Salazar, AstraZeneca

Table of contents

  1. Copyright
  2. Brief Table of Contents
  3. Table of Contents
  4. Foreword
  5. Preface
  6. Acknowledgments
  7. About this book
  8. About the authors
  9. About the cover illustration
  10. Part 1. Foundations
    1. Chapter 1. Toward deep learning: a machine-learning introduction
      1. 1.1. What is machine learning?
      2. 1.2. Machine learning by example
      3. 1.3. Deep learning
      4. 1.4. What you’ll learn in this book
      5. 1.5. Summary
    2. Chapter 2. Go as a machine-learning problem
      1. 2.1. Why games?
      2. 2.2. A lightning introduction to the game of Go
      3. 2.3. Handicaps
      4. 2.4. Where to learn more
      5. 2.5. What can we teach a machine?
      6. 2.6. How to measure your Go AI’s strength
      7. 2.7. Summary
    3. Chapter 3. Implementing your first Go bot
      1. 3.1. Representing a game of Go in Python
      2. 3.2. Capturing game state and checking for illegal moves
      3. 3.3. Ending a game
      4. 3.4. Creating your first bot: the weakest Go AI imaginable
      5. 3.5. Speeding up game play with Zobrist hashing
      6. 3.6. Playing against your bot
      7. 3.7. Summary
  11. Part 2. Machine learning and game AI
    1. Chapter 4. Playing games with tree search
      1. 4.1. Classifying games
      2. 4.2. Anticipating your opponent with minimax search
      3. 4.3. Solving tic-tac-toe: a minimax example
      4. 4.4. Reducing search space with pruning
      5. 4.5. Evaluating game states with Monte Carlo tree search
      6. 4.6. Summary
    2. Chapter 5. Getting started with neural networks
      1. 5.1. A simple use case: classifying handwritten digits
      2. 5.2. The basics of neural networks
      3. 5.3. Feed-forward networks
      4. 5.4. How good are our predictions? Loss functions and optimization
      5. 5.5. Training a neural network step-by-step in Python
      6. 5.6. Summary
    3. Chapter 6. Designing a neural network for Go data
      1. 6.1. Encoding a Go game position for neural networks
      2. 6.2. Generating tree-search games as network training data
      3. 6.3. Using the Keras deep-learning library
      4. 6.4. Analyzing space with convolutional networks
      5. 6.5. Predicting Go move probabilities
      6. 6.6. Building deeper networks with dropout and rectified linear units
      7. 6.7. Putting it all together for a stronger Go move-prediction network
      8. 6.8. Summary
    4. Chapter 7. Learning from data: a deep-learning bot
      1. 7.1. Importing Go game records
      2. 7.2. Preparing Go data for deep learning
      3. 7.3. Training a deep-learning model on human game-play data
      4. 7.4. Building more-realistic Go data encoders
      5. 7.5. Training efficiently with adaptive gradients
      6. 7.6. Running your own experiments and evaluating performance
      7. 7.7. Summary
    5. Chapter 8. Deploying bots in the wild
      1. 8.1. Creating a move-prediction agent from a deep neural network
      2. 8.2. Serving your Go bot to a web frontend
      3. 8.3. Training and deploying a Go bot in the cloud
      4. 8.4. Talking to other bots: the Go Text Protocol
      5. 8.5. Competing against other bots locally
      6. 8.6. Deploying a Go bot to an online Go server
      7. 8.7. Summary
    6. Chapter 9. Learning by practice: reinforcement learning
      1. 9.1. The reinforcement-learning cycle
      2. 9.2. What goes into experience?
      3. 9.3. Building an agent that can learn
      4. 9.4. Self-play: how a computer program practices
      5. 9.5. Summary
    7. Chapter 10. Reinforcement learning with policy gradients
      1. 10.1. How random games can identify good decisions
      2. 10.2. Modifying neural network policies with gradient descent
      3. 10.3. Tips for training with self-play
      4. 10.4. Summary
    8. Chapter 11. Reinforcement learning with value methods
      1. 11.1. Playing games with Q-learning
      2. 11.2. Q-learning with Keras
      3. 11.3. Summary
    9. Chapter 12. Reinforcement learning with actor-critic methods
      1. 12.1. Advantage tells you which decisions are important
      2. 12.2. Designing a neural network for actor-critic learning
      3. 12.3. Playing games with an actor-critic agent
      4. 12.4. Training an actor-critic agent from experience data
      5. 12.5. Summary
  12. Part 3. Greater than the sum of its parts
    1. Chapter 13. AlphaGo: Bringing it all together
      1. 13.1. Training deep neural networks for AlphaGo
      2. 13.2. Bootstrapping self-play from policy networks
      3. 13.3. Deriving a value network from self-play data
      4. 13.4. Better search with policy and value networks
      5. 13.5. Practical considerations for training your own AlphaGo
      6. 13.6. Summary
    2. Chapter 14. AlphaGo Zero: Integrating tree search with reinforcement learning
      1. 14.1. Building a neural network for tree search
      2. 14.2. Guiding tree search with a neural network
      3. 14.3. Training
      4. 14.4. Improving exploration with Dirichlet noise
      5. 14.5. Modern techniques for deeper neural networks
      6. 14.6. Exploring additional resources
      7. 14.7. Wrapping up
      8. 14.8. Summary
      9. Vectors, matrices, and beyond: a linear algebra primer
      10. Rank 3 tensors
      11. Calculus in five minutes: derivatives and finding maxima
      12. A bit of notation
      13. The backpropagation algorithm for feed-forward networks
      14. Backpropagation for sequential neural networks
      15. Backpropagation for neural networks in general
      16. Computational challenges with backpropagation
      17. Go programs
      18. Go servers
      19. Model training on AWS
      20. Hosting a bot on AWS over HTTP
      21. Registering and activating your bot at OGS
      22. Testing your OGS bot locally
      23. Deploying your OGS bot on AWS
  13. Appendix A. Mathematical foundations
  14. Appendix B. The backpropagation algorithm
  15. Appendix C. Go programs and servers
  16. Appendix D. Training and deploying bots by using Amazon Web Services
  17. Appendix E. Submitting a bot to the Online Go Server
  18. Index
  19. List of Figures
  20. List of Tables
  21. List of Listings

Product information

  • Title: Deep Learning and the Game of Go
  • Author(s): Kevin Ferguson, Max Pumperla
  • Release date: January 2019
  • Publisher(s): Manning Publications
  • ISBN: 9781617295324