Anatomy of an Incident

Chapter 6. The Mayan Apocalypse: A Real-World Example

To see some of the principles we talked about in action, we’re going to dig into a real-world example of a major Google outage. We’ll go through what happened, see the scaled organizational structure in action, and show how it was resolved and how we’ve worked to learn from this incident.

For Google, the Mayan Apocalypse was not some New Age phenomenon that led to failure during the year 2012. Rather, the Mayan Apocalypse happened June 2, 2019, with a network automation tool named Maya. Maya does flag management and organizes traffic direction over one of our networking backbones, and a tiny, tiny code shift led to an entity type being consistently mis-flagged.

Around noon, we were proceeding with planned maintenance. We finalized a list of operations and configuration changes (including on Maya) to be run over a set number of servers. When this mis-flag came into conflict with our job scheduling logic, we “discovered” a new failure mode in which jobs associated with traffic direction were de-scheduled en masse. The network traffic to/from those regions then tried to fit the de-scheduled jobs into the remaining network capacity where traffic direction was still functional, but it did not succeed. The network became congested, and our systems correctly triaged the traffic overload and auto-drained larger, less latency-sensitive traffic in order to preserve smaller, latency-sensitive traffic flows.

The traffic jam had begun. ...