Amazon Hacks

Scrape Customer Advice

Screen scraping can give you access to community features not yet implemented through the API—like customer buying advice.

Customer buying advice isn’t available through Amazon’s Web Services, so if you’d like to include this information on a remote site, you’ll have to get it from Amazon’s site through scraping. The first step to this hack is knowing where to find all of the customer advice on one page. The following URL links directly to the advice page for a given ASIN:

http://amazon.com/o/tg/detail/-/insert ASIN/?vi=advice

The Code

This Perl script, get_advice.pl , splits the advice page into two variables based on the headings “in addition to” and “instead of.” It then loops through those sections, using regular expressions to match the products’ information. The script then formats and prints the information.

#!/usr/bin/perl
# get_advice.pl
# A script to scrape Amazon to retrieve customer buying advice
# Usage: perl get_advice.pl <asin>

#Take the asin from the command-line
my $asin =shift @ARGV or die "Usage:perl get_advice.pl <asin>\n";

#Assemble the URL
my $url = "http://amazon.com/o/tg/detail/-/" . $asin . 
          "/?vi=advice";

#Set up unescape-HTML rules
my %unescape = ('&quot;'=>'"', '&amp;'=>'&', '&nbsp;'=>' ');
my $unescape_re = join '|' => keys %unescape;

use strict;
use LWP::Simple;

#Request the URL
my $content = get($url);
die "Could not retrieve $url" unless $content;

my($inAddition) = (join '', $content) =~ m!in addition to(.*?)<tr>[RETURN] <td colspan=3><br></td></tr>!mis; ...