Parsing a Web Server Log File
Problem
You want to extract from a web server log file only the information you’re interested in.
Solution
Pull apart the log file as follows:
while (<LOGFILE>) { my ($client, $identuser, $authuser, $date, $time, $tz, $method, $url, $protocol, $status, $bytes) = /^(\S+) (\S+) (\S+) \[([^:]+):(\d+:\d+:\d+) ([^\]]+) "(\S+) (.*?) (\S+)" (\S+) (\S+)$/; # ... }
Discussion
This regular expression pulls apart entries in Common Log Format, an informal standard that most web servers adhere to. The fields are:
- client
IP address or domain name of browser’s machine
- identuser
If IDENT (RFC 1413) was used, what it returned
- authuser
If username/password authentication was used, whom they logged in as
- date
Date of request (e.g., 01/Mar/1997)
- time
Time of request (e.g., 12:55:36)
- tz
Time zone (e.g., -0700)
- method
Method of request (e.g., GET, POST, or PUT)
- url
URL in request (e.g., /~user/index.html)
- protocol
HTTP/1.0 or HTTP/1.1
- status
Returned status (200 is okay, 500 is server error)
- bytes
Number of bytes returned (could be
"-"
for errors, redirects, and other non-document transfers)
Other formats include the referrer and agent information. The pattern
needs only minor changes for it to work with other log file formats.
Watch out that spaces in the URL field are not escaped. This means
that we can’t use \S*
to extract the URL.
.*
would cause the regex to match the entire
string and then backtrack until it could satisfy the rest of the
pattern. We use .*?
and anchor the pattern to the end ...
Get Perl Cookbook now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.