So far, we've looked at lots of ways to prevent security problems. The remainder of this chapter will look at ways to detect and investigate security breaches. We'll consider all of the various monitoring activities that you might want to use as they would be performed manually and in isolation from one another. There are both vendor-supplied and free tools to simplify and automate the process, and you may very well choose to use one of them. However, knowing what to look for and how to find it will help you to evaluate these tools and use them more effectively. The most sophisticated system watchdog package is ultimately only as good as the person reading, interpreting, and acting on the information it produces.
The fundamental prerequisite for effective system monitoring is knowing what normal is, that is, knowing how things ought to be in terms of:
General system activity levels and how they change over the course of a day and a week.
Normal activities for all the various users on the system.
The structure, attributes, and contents of the filesystem itself, key system directories, and important files.
The proper formats and settings within important system configuration files.
Some of these things can be determined from the current system configuration (and possibly by comparing it to a newly installed system). Others are a matter of familiarity and experience and must be acquired over time.
It is important to examine the password file regularly ...