Perl & LWP

Chapter 4. URLs

Now that you’ve seen how LWP models HTTP requests and responses, let’s study the facilities it provides for working with URLs. A URL tells you how to get to something: “use HTTP with this host and request this,” “connect via FTP to this host and retrieve this file,” or “send email to this address.”

The great variety inherent in URLs is both a blessing and a curse. On one hand, you can stretch the URL syntax to address almost any type of network resource. However, this very flexibility means attempts to parse arbitrary URLs with regular expressions rapidly run into a quagmire of special cases.

The LWP suite of modules provides the URI class to manage URLs. This chapter describes how to create objects that represent URLs, extract information from those objects, and convert between absolute and relative URLs. This last task is particularly useful for link checkers and spiders, which take partial URLs from HTML links and turn those into absolute URLs to request.

Parsing URLs

Rather than attempt to pull apart URLs with regular expressions, which is difficult to do in a way that works with all the many types of URLs, you should use the URI class. When you create an object representing a URL, it has attributes for each part of a URL (scheme, username, hostname, port, etc.). Make method calls to get and set these attributes.

Example 4-1 creates a URI object representing a complex URL, then calls methods to discover the various components of the URL.

Example 4-1. Decomposing ...