Skip to Main Content
Perl in a Nutshell, 2nd Edition
book

Perl in a Nutshell, 2nd Edition

by Nathan Patwardhan, Ellen Siever, Stephen Spainhour
June 2002
Beginner content levelBeginner
759 pages
80h 42m
English
O'Reilly Media, Inc.
Content preview from Perl in a Nutshell, 2nd Edition

LWP::RobotUA Sections

The Robot User Agent (LWP::RobotUA) is a subclass of LWP::UserAgent and is used to create robot client applications. A robot application requests resources in an automated fashion. Robots perform such activities as searching, mirroring, and surveying. Some robots collect statistics, while others wander the Web and summarize their findings for a search engine.

The LWP::RobotUA module defines methods to help program robot applications and observes the Robot Exclusion Standards, which web server administrators can define on their web site to keep robots away from certain (or all) areas of the site.

The constructor for an LWP::RobotUA object looks like this:

$rob = LWP::RobotUA->new(agent_name, email, [$rules]);

The first parameter, agent_name, is the user agent identifier used for the value of the User-Agent header in the request. The second parameter is the email address of the person using the robot, and the optional third parameter is a reference to a WWW::RobotRules object, which is used to store the robot rules for a server. If you omit the third parameter, the LWP::RobotUA module requests the robots.txt file from every server it contacts and generates its own WWW::RobotRules object.

Since LWP::RobotUA is a subclass of LWP::UserAgent, the LWP::UserAgent methods are used to perform the basic client activities. The following methods are defined by LWP::RobotUA for robot-related functionality.

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

Perl by Example, Fourth Edition

Perl by Example, Fourth Edition

Ellie Quigley
Perl Cookbook, 2nd Edition

Perl Cookbook, 2nd Edition

Tom Christiansen, Nathan Torkington
Perl in a Nutshell

Perl in a Nutshell

Nathan Patwardhan, Ellen Siever, Stephen Spainhour
Learning Perl, 7th Edition

Learning Perl, 7th Edition

Randal L. Schwartz, brian d foy, Tom Phoenix

Publisher Resources

ISBN: 0596002416Errata Page