AWS System Administration

Chapter 9. Log Management

Despite the best efforts of system administrators everywhere, logging in the cloud can quickly become more complicated (and more expensive) than logging in a physical hardware environment. Because EC2 instances come and go dynamically, the number of instances producing log files can grow and shrink at any time. Your logging system must therefore be designed with this in mind, to ensure that it keeps up with peaks in demand when processing log files.

Another area that requires some advance planning is log storage. Running a large number of instances will produce large log files, which need to be stored somewhere. Without some advance planning, the storage requirements can grow rapidly, leading to an increase in costs.

This chapter presents some popular logging tools that can be useful in AWS environments and introduces some strategies for managing log files without breaking the bank. Logstash is used to demonstrate the concepts in this chapter, but the principles also apply to most other logging software.

Central Logging

A common solution to the problem of viewing logs from multiple machines is to set up a central logging server to which all servers in your infrastructure send their logs. The central logging server is responsible for parsing and processing log files and managing the policies for log retention.

This pattern works well within AWS, with a few caveats. It is critical to ensure that your logging system does not struggle to keep up when many instances ...