O'Reilly logo

Amazon Hacks by Paul Bausch

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Scrape Customer Advice

Screen scraping can give you access to community features not yet implemented through the API—like customer buying advice.

Customer buying advice isn’t available through Amazon’s Web Services, so if you’d like to include this information on a remote site, you’ll have to get it from Amazon’s site through scraping. The first step to this hack is knowing where to find all of the customer advice on one page. The following URL links directly to the advice page for a given ASIN:

http://amazon.com/o/tg/detail/-/insert ASIN/?vi=advice

The Code

This Perl script, get_advice.pl , splits the advice page into two variables based on the headings “in addition to” and “instead of.” It then loops through those sections, using regular expressions to match the products’ information. The script then formats and prints the information.

#!/usr/bin/perl
# get_advice.pl
# A script to scrape Amazon to retrieve customer buying advice
# Usage: perl get_advice.pl <asin>

#Take the asin from the command-line
my $asin =shift @ARGV or die "Usage:perl get_advice.pl <asin>\n";

#Assemble the URL
my $url = "http://amazon.com/o/tg/detail/-/" . $asin . 
          "/?vi=advice";

#Set up unescape-HTML rules
my %unescape = ('&quot;'=>'"', '&amp;'=>'&', '&nbsp;'=>' ');
my $unescape_re = join '|' => keys %unescape;

use strict;
use LWP::Simple;

#Request the URL
my $content = get($url);
die "Could not retrieve $url" unless $content;

my($inAddition) = (join '', $content) =~ m!in addition to(.*?)<tr>[RETURN] <td colspan=3><br></td></tr>!mis; ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required