Like a lever, automation allows administrators to make large movements with less effort. It’s a force multiplier.
On August 11, 2016, link aggregator Reddit.com suffered an outage. It was unavailable for approximately ninety minutes and had degraded service for about another ninety minutes. In their postmortem, Reddit admins described a conflict between deliberate, manual changes and their automation platform:
First, the admins shut down their autoscaler service so that they could upgrade a ZooKeeper cluster.
Sometime into the upgrade process, the package management system detected the autoscaler was off and restarted it.
The autoscaler came back online and read the partially migrated ZooKeeper ...