Network Security Hacks

Monitor Your Logs Automatically

Use swatch to alert you to possible problems as they happen.

Automatically generated log file summaries are fine for keeping abreast of what’s happening with your systems and networks, but if you want to know about events as they happen, you’ll need to look elsewhere. One tool that can help keep you informed in real time is swatch (http://swatch.sourceforge.net), the "Simple WATCHer.”

Swatch is a highly configurable log file monitor that can watch a file for user-defined triggers and dispatch alerts in a variety of ways. It consists of a Perl program, a configuration file, and a library of actions to take when it sees a trigger in the file it is monitoring.

To install swatch, download the package, unpack it, and go into the directory that it creates. Then run these commands:

# perl Makefile.PL
# make && make install

Before swatch will build, the Date::Calc , Date::Parse, File::Tail, and Time::HiRes Perl CPAN modules must be installed. If you get an error message like the following when you run perl Makefile.PL, then you will need to install those modules:

Warning: prerequisite Date::Calc 0 not found.
Warning: prerequisite Date::Parse 0 not found.
Warning: prerequisite Time::HiRes 1.12 not found.
Writing Makefile for swatch

If you already have Perl’s CPAN module installed, simply run these commands:

# perl -MCPAN -e "install Date::Calc"
# perl -MCPAN -e "install Date::Parse"
# perl -MCPAN -e "Time::HiRes"

By default, swatch looks for its configuration in a file called .swatchrc in the current user’s home directory. This file contains regular expressions to watch for in the file that you are monitoring with swatch. If you want to use a different configuration file, tell swatch by using the -c command-line switch.

For instance, to use /etc/swatch/messages.conf to monitor /var/log/messages, you could invoke swatch like this:

# swatch -c /etc/swatch/messages.conf -t /var/log/messages

The general format for entries in this file is the following:

watchfor /<regex>/
<action1>
[action2]
[action3]
...

Alternatively, you can ignore specific log messages that match a regular expression by using the ignore keyword:

ignore /<regex>/

You can also specify multiple regular expressions by separating them with the | character.

Swatch is very configurable in what actions it can take when a string matches a regular expression. Some useful actions that you can specify in your .swatchrc are echo, write, exec, mail, pipe, and throttle.

The echo action simply prints the matching line to the console; additionally, you can specify what text mode it will use. Thus, lines can be printed to the console as bold, underlined, blinking, inverted, or colored text.

For instance, if you wanted to print a matching line in red, blinking text, you could use the following action:

echo blink,red

The write action is similar to the echo action, except it does not support text modes. It can, however, write the matching line to any specified user’s TTY:

write user:user2:...

The exec action allows you to execute any command:

exec <command>

You can use the $0 or $* variables to pass the entire matching line to the command that you execute, $1 to pass the first field in the line, $2 for the second, and so on. So, if you wanted to pass only the second and third fields from the matching line to the command mycommand, you could use an action like this:

exec "mycommand $2 $3"

The mail action is especially useful if you have an email-enabled or text messaging-capable cell phone or pager. When using the mail action, you can list as many recipient addresses as you like, in addition to specifying a subject line. Swatch will send the line that matched the regular expression to these addresses with the subject you set.

Here is the general form of the mail action:

mail addresses=address:address2:...,subject=mysubject

When using the mail action, be sure to escape the @ characters in the email addresses (i.e., @ becomes \@). If you have any spaces in the subject of the email, you should escape those as well.

In addition to the exec action, swatch can execute external commands with the pipe action as well. The only difference is that instead of passing arguments to the command, swatch will execute the command and pipe the matching line to it. To use this action, just put the pipe keyword followed by the command you want to use.

Alternatively, to increase performance, you can use the keep_open option to keep the pipe to the program open until swatch exits or needs to perform a different pipe action:

pipe mycommand,keep_open

One problem with executing commands or sending emails whenever a specific string occurs in a log message is that sometimes the same log message may be generated over and over again very rapidly. Clearly, if this were to happen, you wouldn’t want to get paged or emailed 100 times within a 10-minute period. To alleviate this problem, swatch provides the throttle action. This action lets you suppress a specific message or any message that matches a particular regular expression for a specified amount of time.

The general form of the throttle action is:

throttle h:m:s

The throttle action will throttle based on the contents of the message by default. If you would like to throttle the actions based on the regular expression that caused the match, you can add a ,use=regex to the end of your throttle statement.

Swatch is an incredibly useful tool, but it can take some work to create a good .swatchrc. The best way to figure out what to look for is to examine your log files for behavior that you want to monitor closely.