Perl & LWP

Preface

Perl soared to popularity as a language for creating and managing web content. Perl is equally adept at consuming information on the Web. Most web sites are created for people, but quite often you want to automate tasks that involve accessing a web site in a repetitive way. Such tasks could be as simple as saying “here’s a list of URLs; I want to be emailed if any of them stop working,” or they could involve more complex processing of any number of pages. This book is about using LWP (the Library for World Wide Web in Perl) and Perl to fetch and process web pages.

For example, if you want to compare the prices of all O’Reilly books on Amazon.com and bn.com, you could look at each page yourself and keep track of the prices. Or you could write an LWP program to fetch the product pages, extract the prices, and generate a report. O’Reilly has a lot of books in print, and after reading this one, you’ll be able to write and run the program much more quickly than you could visit every catalog page.

Consider also a situation in which a particular page has links to several dozen files (images, music, and so on) that you want to download. You could download each individually, by monotonously selecting each link in your browser and choosing Save as..., or you could dash off a short LWP program that scans for URLs in that page and downloads each, unattended.

Besides extracting data from web pages, you can also automate submitting data through web forms. Whether this is a matter of uploading ...