Automating System Administration with Perl, 2nd Edition

Chapter 10. Log Files

If this weren’t a book on system administration, an entire chapter on log files would seem peculiar. But system administrators have a very special relationship with log files. System administrators are expected to be like Doctor Doolittle, who could talk to the animals: able to communicate with a large menagerie of software and hardware. Much of this communication takes place through log files, so we become log file linguists. Perl can be a big help in this process.

It would be impossible to touch on all the different kinds of processing and analysis you can do with logs in a single chapter. Entire books have been devoted to just statistical analysis of this sort of data, and companies have been founded to sell products to help analyze it. However, this chapter does present some general approaches to the topic and some relevant Perl tools, to whet your appetite for more.

Reading Text Logs

Logs come in different flavors, so we need several approaches for dealing with them. The most common type of log file is one composed entirely of lines of text: popular server packages like Apache (Web), BIND (DNS), and sendmail (email) spew log text in voluminous quantities (especially in debug mode). Most logs on Unix machines look similar because they are created by a centralized logging facility known as syslog. For our purposes, we can treat files created by syslog like any other text files.

Here’s a simple Perl program to scan for the word “error” in a text-based log file: ...