Cybersecurity Ops with bash

Chapter 8. Real-Time Log Monitoring

The ability to analyze a log after an event is an important skill. It is equally important to be able to extract information from a logfile in real time to detect malicious or suspicious activity as it happens. In this chapter, we explore methods to read in log entries as they are generated, format them for output to the analyst, and generate alerts based on known indicators of compromise.

Tip

Maintenance, Monitoring, and Analysis of Audit Logs is identified as a top 20 security control by the Center for Internet Security. To learn more, visit the CIS Controls page.

Monitoring Text Logs

The most basic method to monitor a log in real time is to use the tail command’s -f option, which continuously reads a file and outputs new lines to stdout as they are added. As in previous chapters, we will use an Apache web server access log for examples, but the techniques presented can be applied to any text-based log. To monitor the Apache access log with tail:

tail -f /var/logs/apache2/access.log

Commands can be combined to provide more-advanced functionality. The output from tail can be piped into grep so only entries matching specific criteria will be output. The following example monitors the Apache access log and outputs entries matching a particular IP address:

tail -f /var/logs/apache2/access.log | grep '10.0.0.152'

Regular expressions can also be used. In this example, only entries returning an HTTP status code of 404 Page Not Found will be displayed; ...