book

Deep Learning and the Game of Go

Name: Deep Learning and the Game of Go
ISBN: 9781617295324

by Max Pumperla, Kevin Ferguson

January 2019

Intermediate to advanced

384 pages

13h 27m

English

Manning Publications

Audiobook available

Read now

Unlock full access

Copyright
Brief Table of Contents
Table of Contents
Foreword
Preface
Acknowledgments
About this book
About the authors
About the cover illustration
Part 1. Foundations

Chapter 1. Toward deep learning: a machine-learning introduction
1.1. What is machine learning?1.2. Machine learning by example1.3. Deep learning1.4. What you’ll learn in this book1.5. Summary
Chapter 2. Go as a machine-learning problem
2.1. Why games?2.2. A lightning introduction to the game of Go2.3. Handicaps2.4. Where to learn more2.5. What can we teach a machine?2.6. How to measure your Go AI’s strength2.7. Summary
Chapter 3. Implementing your first Go bot
3.1. Representing a game of Go in Python3.2. Capturing game state and checking for illegal moves3.3. Ending a game3.4. Creating your first bot: the weakest Go AI imaginable3.5. Speeding up game play with Zobrist hashing3.6. Playing against your bot3.7. Summary
Part 2. Machine learning and game AI
Chapter 4. Playing games with tree search
4.1. Classifying games4.2. Anticipating your opponent with minimax search4.3. Solving tic-tac-toe: a minimax example4.4. Reducing search space with pruning4.5. Evaluating game states with Monte Carlo tree search4.6. Summary
Chapter 5. Getting started with neural networks
5.1. A simple use case: classifying handwritten digits5.2. The basics of neural networks5.3. Feed-forward networks5.4. How good are our predictions? Loss functions and optimization5.5. Training a neural network step-by-step in Python5.6. Summary
Chapter 6. Designing a neural network for Go data
6.1. Encoding a Go game position for neural networks6.2. Generating tree-search games as network training data6.3. Using the Keras deep-learning library6.4. Analyzing space with convolutional networks6.5. Predicting Go move probabilities6.6. Building deeper networks with dropout and rectified linear units6.7. Putting it all together for a stronger Go move-prediction network6.8. Summary
Chapter 7. Learning from data: a deep-learning bot
7.1. Importing Go game records7.2. Preparing Go data for deep learning7.3. Training a deep-learning model on human game-play data7.4. Building more-realistic Go data encoders7.5. Training efficiently with adaptive gradients7.6. Running your own experiments and evaluating performance7.7. Summary
Chapter 8. Deploying bots in the wild
8.1. Creating a move-prediction agent from a deep neural network8.2. Serving your Go bot to a web frontend8.3. Training and deploying a Go bot in the cloud8.4. Talking to other bots: the Go Text Protocol8.5. Competing against other bots locally8.6. Deploying a Go bot to an online Go server8.7. Summary
Chapter 9. Learning by practice: reinforcement learning
9.1. The reinforcement-learning cycle9.2. What goes into experience?9.3. Building an agent that can learn9.4. Self-play: how a computer program practices9.5. Summary
Chapter 10. Reinforcement learning with policy gradients
10.1. How random games can identify good decisions10.2. Modifying neural network policies with gradient descent10.3. Tips for training with self-play10.4. Summary
Chapter 11. Reinforcement learning with value methods
11.1. Playing games with Q-learning11.2. Q-learning with Keras11.3. Summary
Chapter 12. Reinforcement learning with actor-critic methods
12.1. Advantage tells you which decisions are important12.2. Designing a neural network for actor-critic learning12.3. Playing games with an actor-critic agent12.4. Training an actor-critic agent from experience data12.5. Summary
Part 3. Greater than the sum of its parts
Chapter 13. AlphaGo: Bringing it all together
13.1. Training deep neural networks for AlphaGo13.2. Bootstrapping self-play from policy networks13.3. Deriving a value network from self-play data13.4. Better search with policy and value networks13.5. Practical considerations for training your own AlphaGo13.6. Summary
Chapter 14. AlphaGo Zero: Integrating tree search with reinforcement learning
14.1. Building a neural network for tree search14.2. Guiding tree search with a neural network14.3. Training14.4. Improving exploration with Dirichlet noise14.5. Modern techniques for deeper neural networks14.6. Exploring additional resources14.7. Wrapping up14.8. Summary
Vectors, matrices, and beyond: a linear algebra primer
Rank 3 tensorsCalculus in five minutes: derivatives and finding maximaAppendix A. Mathematical foundations
A bit of notation
The backpropagation algorithm for feed-forward networksBackpropagation for sequential neural networksBackpropagation for neural networks in generalComputational challenges with backpropagationAppendix B. The backpropagation algorithm
Go programs
Go serversAppendix C. Go programs and servers
Model training on AWS
Hosting a bot on AWS over HTTPAppendix D. Training and deploying bots by using Amazon Web Services
Registering and activating your bot at OGS
Testing your OGS bot locallyDeploying your OGS bot on AWSAppendix E. Submitting a bot to the Online Go Server
Index
List of Figures
List of Tables
List of Listings

Content preview from Deep Learning and the Game of Go

Chapter 9. Learning by practice: reinforcement learning

This chapter covers

Defining a task for reinforcement learning
Building a learning agent for games
Collecting self-play experiences for training

I’ve probably read a dozen books on Go, all written by strong pros from China, Korea, and Japan. And yet I’m just an intermediate amateur player. Why haven’t I reached the level of these legendary players? Have I forgotten their lessons? I don’t think that’s it; I can practically recite Toshiro Kageyama’s Lessons in the Fundamentals of Go (Ishi Press, 1978) by heart. Maybe I just need to read more books....

I don’t know the full recipe for becoming a top Go star, but I know at least one difference between me and Go professionals: practice. ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781617295324Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Deep Learning and the Game of Go

by Max Pumperla, Kevin Ferguson

Chapter 9. Learning by practice: reinforcement learning

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.