Design – Build – Run: Applied Practices and Principles for Production-Ready Software Development

11.2. Typical Incidents

Many functional and technical incidents can occur in the daily running of a system. Some are due to faulty or defective software, while others are simply due to quirks and environmental conditions. For example, when a user attempts to start a particular service, it fails, but when the user tries again, it starts perfectly. A review of logs and events might not highlight the actual reason why the service stopped in the first instance, although my firm belief is that there is a legitimate reason for the incident. Computer systems have become so complicated that it's nearly impossible to understand everything that's going on under the covers all the time. That doesn't mean to say that there's not a legitimate reason for a crash; it simply means that it could be difficult to determine the reason if the logs and events don't highlight any clues. This reinforces my message that logs and events should contain as much information as possible to enable you to understand and re-create the incident. Furthermore, using the wait-retry pattern can help to alleviate a complete stop to the processing. That doesn't mean you shouldn't log the issue and/or gather as much information as possible; it simply means that some issues are accepted or anticipated, and as long as the action is retried and successful, the system will continue to function.

Some common technical issues and incidents that can occur during the day-to-day running of the system include but are not limited ...