Design – Build – Run: Applied Practices and Principles for Production-Ready Software Development

21.2. Investigative Tooling

The tooling required during an incident investigation will vary greatly. During testing phases, testers raise defects against the application. In production, incident investigation is triggered following a live service incident, such as the application raising an error event. In either case, pulling together a complete timeline of the transaction and the data being used helps to identify what happened. The following outlines some of the "tools" that can help with piecing together the timeline:

• Event log(s) — The event logs should point to the heart of the issue and contain all the relevant contextual information about the activity. This is the starting point for most incident investigations. The transaction will be executed through many servers and many components. The timeline needs to be pieced together from all the event entries. The event logs will also show what other "system" and application activity was happening at the same time, which could throw new light on the incident being investigated. A simple script or tool that can extract all the relevant log entries from the relevant servers will come in very handy during testing and live support.

• Trace file(s) — The trace files will also be distributed across multiple servers. There will probably be trace files for different processes, so it is even more important to ensure that activities can be traced across these individual files. A simple tool that I've always wanted is one that scrapes the files ...