Skip to Main Content
Regular Expressions Cookbook, 2nd Edition
book

Regular Expressions Cookbook, 2nd Edition

by Jan Goyvaerts, Steven Levithan
August 2012
Intermediate to advanced content levelIntermediate to advanced
609 pages
19h 16m
English
O'Reilly Media, Inc.
Content preview from Regular Expressions Cookbook, 2nd Edition

Broken Links Reported in Web Logs

Problem

You have a log for your website in the Combined Log Format. You want to check the log for any errors caused by broken links on your own website.

Solution

"(?:GET|POST)(?<file>[^#?"]+)(?:[#?][^"]*)?HTTP/[0-9.]+"404↵
(?:[0-9]+|-)"(?<referrer>http://www\.yoursite\.com[^"]*)"
Regex options: None
Regex flavors: .NET, Java 7, XRegExp, PCRE 7, Perl 5.10, Ruby 1.9
"(?:GET|POST)(?P<file>[^#?"]+)(?:[#?][^"]*)?HTTP/[0-9.]+"404↵
(?:[0-9]+|-)"(?P<referrer>http://www\.yoursite\.com[^"]*)"
Regex options: None
Regex flavors: PCRE 4, Perl 5.10, Python
"(?:GET|POST)([^#?"]+)(?:[#?][^"]*)?HTTP/[0-9.]+"404↵
(?:[0-9]+|-)"(http://www\.yoursite\.com[^"]*)"
Regex options: None
Regex flavors: .NET, Java, JavaScript, PCRE, Perl, Python, Ruby

Discussion

When a visitor clicks a link on your website that points to a file on your own site that does not exist, the visitor gets a “page not found” error. Your web server will write an entry in its log that contains the file that does not exist as the requested object, status code 404, and the page that contains the broken link as the referrer. So you need to extract the requested object and the referrer from log entries that have status code 404 and a referring URL on your own website.

One way to do this would be to use your favorite programming language to write a script that implements Combined Log Format. While iterating over all the matches, check whether the “status” group captured 404 and whether the “referrer” ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

Mastering Regular Expressions, 3rd Edition

Mastering Regular Expressions, 3rd Edition

Jeffrey E.F. Friedl
Regular Expressions Cookbook

Regular Expressions Cookbook

Jan Goyvaerts, Steven Levithan

Publisher Resources

ISBN: 9781449327453Supplemental ContentErrata Page