Securing DevOps

7 Collecting and storing logs

This chapter covers

• Building the five layers of a modern logging pipeline
• Collecting logs from systems, applications, infrastructures, and third parties
• Using a message broker to pass logs from producers to consumers
• Understanding techniques to analyze logs through task-specific modules
• Learning how to store logs effectively and implement a retention policy
• Evaluating tools to access and visualize both raw logs and metrics

You probably already know that you should be collecting logs on all applications and systems, but it’s easy to wonder why, what kind, and exactly how much logging is needed. We’ll spend this chapter discussing what a modern logging pipeline looks like, and what logs should be sent to it, but ...