Perl & LWP

Perl & LWP

by Sean M. Burke
Released June 2002
Publisher(s): O'Reilly Media, Inc.
ISBN: 9780596001780

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Buy on Amazon Buy on ebooks.com
Start your free trial

Book description

Perl soared to popularity as a language for creating and managing web content, but with LWP (Library for WWW in Perl), Perl is equally adept at consuming information on the Web. LWP is a suite of modules for fetching and processing web pages.The Web is a vast data source that contains everything from stock prices to movie credits, and with LWP all that data is just a few lines of code away. Anything you do on the Web, whether it's buying or selling, reading or writing, uploading or downloading, news to e-commerce, can be controlled with Perl and LWP. You can automate Web-based purchase orders as easily as you can set up a program to download MP3 files from a web site.Perl & LWP covers:

  • Understanding LWP and its design
  • Fetching and analyzing URLs
  • Extracting information from HTML using regular expressions and tokens
  • Working with the structure of HTML documents using trees
  • Setting and inspecting HTTP headers and response codes
  • Managing cookies
  • Accessing information that requires authentication
  • Extracting links
  • Cooperating with proxy caches
  • Writing web spiders (also known as robots) in a safe fashion
Perl & LWP includes many step-by-step examples that show how to apply the various techniques. Programs to extract information from the web sites of BBC News, Altavista, ABEBooks.com, and the Weather Underground, to name just a few, are explained in detail, so that you understand how and why they work.Perl programmers who want to automate and mine the web can pick up this book and be immediately productive. Written by a contributor to LWP, and with a foreword by one of LWP's creators, Perl & LWP is the authoritative guide to this powerful and popular toolkit.

Table of contents

  1. A Note Regarding Supplemental Files
  2. Foreword
  3. Preface
    1. Audience for This Book
    2. Structure of This Book
    3. Order of Chapters
    4. Important Standards Documents
    5. Conventions Used in This Book
    6. Comments & Questions
    7. Acknowledgments
  4. 1. Introduction to Web Automation
    1. 1.1. The Web as Data Source
      1. 1.1.1. Screen Scraping
      2. 1.1.2. Brittleness
      3. 1.1.3. Web Services
    2. 1.2. History of LWP
    3. 1.3. Installing LWP
      1. 1.3.1. Installing LWP from the CPAN Shell
        1. 1.3.1.1. Configuring
        2. 1.3.1.2. Obtaining help
        3. 1.3.1.3. Installing LWP
      2. 1.3.2. Installing LWP Manually
        1. 1.3.2.1. Download distributions
        2. 1.3.2.2. Unpack and configure
        3. 1.3.2.3. Make, test, and install
    4. 1.4. Words of Caution
      1. 1.4.1. Network and Server Load
      2. 1.4.2. Copyright
      3. 1.4.3. Acceptable Use
    5. 1.5. LWP in Action
      1. 1.5.1. The Object-Oriented Interface
      2. 1.5.2. Forms
      3. 1.5.3. Parsing HTML
      4. 1.5.4. Authentication
  5. 2. Web Basics
    1. 2.1. URLs
    2. 2.2. An HTTP Transaction
      1. 2.2.1. Request
      2. 2.2.2. Response
    3. 2.3. LWP::Simple
      1. 2.3.1. Basic Document Fetch
      2. 2.3.2. Fetch and Store
      3. 2.3.3. Fetch and Print
      4. 2.3.4. Previewing with HEAD
    4. 2.4. Fetching Documents Without LWP::Simple
    5. 2.5. Example: AltaVista
    6. 2.6. HTTP POST
    7. 2.7. Example: Babelfish
  6. 3. The LWP Class Model
    1. 3.1. The Basic Classes
    2. 3.2. Programming with LWP Classes
    3. 3.3. Inside the do_GET and do_POST Functions
    4. 3.4. User Agents
      1. 3.4.1. Connection Parameters
      2. 3.4.2. Request Parameters
      3. 3.4.3. Protocols
      4. 3.4.4. Redirection
      5. 3.4.5. Authentication
      6. 3.4.6. Proxies
      7. 3.4.7. Request Methods
        1. 3.4.7.1. Saving response content to a file
        2. 3.4.7.2. Sending response content to a callback
        3. 3.4.7.3. Mirroring a URL to a file
      8. 3.4.8. Advanced Methods
    5. 3.5. HTTP::Response Objects
      1. 3.5.1. Status Line
      2. 3.5.2. Content
      3. 3.5.3. Headers
      4. 3.5.4. Expiration Times
      5. 3.5.5. Base for Relative URLs
      6. 3.5.6. Debugging
    6. 3.6. LWP Classes: Behind the Scenes
  7. 4. URLs
    1. 4.1. Parsing URLs
      1. 4.1.1. Constructors
      2. 4.1.2. Output
      3. 4.1.3. Comparison
      4. 4.1.4. Components of a URL
      5. 4.1.5. Queries
    2. 4.2. Relative URLs
    3. 4.3. Converting Absolute URLs to Relative
    4. 4.4. Converting Relative URLs to Absolute
  8. 5. Forms
    1. 5.1. Elements of an HTML Form
    2. 5.2. LWP and GET Requests
      1. 5.2.1. GETting Fixed URLs
      2. 5.2.2. GETting a query_form( ) URL
    3. 5.3. Automating Form Analysis
    4. 5.4. Idiosyncrasies of HTML Forms
      1. 5.4.1. Hidden Elements
      2. 5.4.2. Text Elements
      3. 5.4.3. Password Elements
      4. 5.4.4. Checkboxes
      5. 5.4.5. Radio Buttons
      6. 5.4.6. Submit Buttons
      7. 5.4.7. Image Buttons
      8. 5.4.8. Reset Buttons
      9. 5.4.9. File Selection Elements
      10. 5.4.10. Textarea Elements
      11. 5.4.11. Select Elements and Option Elements
    5. 5.5. POST Example: License Plates
      1. 5.5.1. The Form
      2. 5.5.2. Use formpairs.pl
      3. 5.5.3. Translating This into LWP
    6. 5.6. POST Example: ABEBooks.com
      1. 5.6.1. The Form
      2. 5.6.2. Translating This into LWP
      3. 5.6.3. Adding Features
      4. 5.6.4. Generalizing the Program
    7. 5.7. File Uploads
    8. 5.8. Limits on Forms
  9. 6. Simple HTML Processing with Regular Expressions
    1. 6.1. Automating Data Extraction
    2. 6.2. Regular Expression Techniques
      1. 6.2.1. Anchor Your Match
      2. 6.2.2. Whitespace
      3. 6.2.3. Embedded Newlines
      4. 6.2.4. Minimal and Greedy Matches
      5. 6.2.5. Capture
      6. 6.2.6. Repeated Matches
      7. 6.2.7. Develop from Components
      8. 6.2.8. Use Multiple Steps
    3. 6.3. Troubleshooting
    4. 6.4. When Regular Expressions Aren’t Enough
    5. 6.5. Example: Extracting Linksfrom a Bookmark File
    6. 6.6. Example: Extracting Linksfrom Arbitrary HTML
    7. 6.7. Example: Extracting Temperatures from Weather Underground
  10. 7. HTML Processing with Tokens
    1. 7.1. HTML as Tokens
    2. 7.2. Basic HTML::TokeParser Use
      1. 7.2.1. Start-Tag Tokens
      2. 7.2.2. End-Tag Tokens
      3. 7.2.3. Text Tokens
      4. 7.2.4. Comment Tokens
      5. 7.2.5. Markup Declaration Tokens
      6. 7.2.6. Processing Instruction Tokens
    3. 7.3. Individual Tokens
      1. 7.3.1. Checking Image Tags
      2. 7.3.2. HTML Filters
    4. 7.4. Token Sequences
      1. 7.4.1. Example: BBC Headlines
      2. 7.4.2. Translating the Problem into Code
      3. 7.4.3. Bundling into a Program
    5. 7.5. More HTML::TokeParser Methods
      1. 7.5.1. The get_text( ) Method
      2. 7.5.2. The get_text( ) Method with Parameters
      3. 7.5.3. The get_trimmed_text( ) Method
      4. 7.5.4. The get_tag( ) Method
        1. 7.5.4.1. Start-tags
        2. 7.5.4.2. End-tags
      5. 7.5.5. The get_tag( ) Method with Parameters
    6. 7.6. Using Extracted Text
  11. 8. Tokenizing Walkthrough
    1. 8.1. The Problem
    2. 8.2. Getting the Data
    3. 8.3. Inspecting the HTML
    4. 8.4. First Code
    5. 8.5. Narrowing In
    6. 8.6. Rewrite for Features
      1. 8.6.1. Debuggability
      2. 8.6.2. Images and Applets
      3. 8.6.3. Link Text
      4. 8.6.4. Live Data
    7. 8.7. Alternatives
  12. 9. HTML Processing with Trees
    1. 9.1. Introduction to Trees
    2. 9.2. HTML::TreeBuilder
      1. 9.2.1. Constructors
      2. 9.2.2. Parse Options
      3. 9.2.3. Parsing
      4. 9.2.4. Cleanup
    3. 9.3. Processing
      1. 9.3.1. Methods for Searching the Tree
      2. 9.3.2. Attributes of a Node
      3. 9.3.3. Traversing
    4. 9.4. Example: BBC News
    5. 9.5. Example: Fresh Air
  13. 10. Modifying HTML with Trees
    1. 10.1. Changing Attributes
      1. 10.1.1. Whitespace
      2. 10.1.2. Other HTML Options
    2. 10.2. Deleting Images
    3. 10.3. Detaching and Reattaching
      1. 10.3.1. The detach_content( ) Method
      2. 10.3.2. Constraints
    4. 10.4. Attaching in Another Tree
      1. 10.4.1. Retaining Comments
      2. 10.4.2. Accessing Comments
      3. 10.4.3. Attaching Content
    5. 10.5. Creating New Elements
      1. 10.5.1. Literals
      2. 10.5.2. New Nodes from Lists
  14. 11. Cookies, Authentication,and Advanced Requests
    1. 11.1. Cookies
      1. 11.1.1. Enabling Cookies
      2. 11.1.2. Loading Cookies from a File
      3. 11.1.3. Saving Cookies to a File
      4. 11.1.4. Cookies and the New York Times Site
    2. 11.2. Adding Extra Request Header Lines
      1. 11.2.1. Pretending to Be Netscape
      2. 11.2.2. Referer
    3. 11.3. Authentication
      1. 11.3.1. Comparing Cookies with Basic Authentication
      2. 11.3.2. Authenticating via LWP
      3. 11.3.3. Security
    4. 11.4. An HTTP Authentication Example:The Unicode Mailing Archive
  15. 12. Spiders
    1. 12.1. Types of Web-Querying Programs
    2. 12.2. A User Agent for Robots
    3. 12.3. Example: A Link-Checking Spider
      1. 12.3.1. The Basic Spider Logic
      2. 12.3.2. Overall Design in the Spider
      3. 12.3.3. HEAD Response Processing
      4. 12.3.4. Redirects
      5. 12.3.5. Link Extraction
      6. 12.3.6. Fleshing Out the URL Scheduling
      7. 12.3.7. The Rest of the Code
    4. 12.4. Ideas for Further Expansion
  16. A. LWP Modules
  17. B. HTTP Status Codes
    1. B.1. 100s: Informational
    2. B.2. 200s: Successful
    3. B.3. 300s: Redirection
    4. B.4. 400s: Client Errors
    5. B.5. 500s: Server Errors
  18. C. Common MIME Types
  19. D. Language Tags
  20. E. Common Content Encodings
  21. F. ASCII Table
  22. G. User’s View of Object-Oriented Modules
    1. G.1. A User’s View of Object-Oriented Modules
    2. G.2. Modules and Their Functional Interfaces
    3. G.3. Modules with Object-Oriented Interfaces
    4. G.4. What Can You Do with Objects?
    5. G.5. What’s in an Object?
    6. G.6. What Is an Object Value?
    7. G.7. So Why Do Some Modules Use Objects?
    8. G.8. The Gory Details
  23. Index
  24. Colophon
  25. Copyright

Product information

  • Title: Perl & LWP
  • Author(s): Sean M. Burke
  • Release date: June 2002
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9780596001780