Google Hacks

Scraping Google Results

Scraping the results of a Google search into a comma-delimited file.

Because you can use the Google API to get results and put them in any format you like, why in the world would you bother to do a manual search result on Google, save the results, and then scrape them with a Perl program? You might not want, or indeed be able, to do anything as fancy as the Google API allows; you might just want to grab some results, drop them into a spreadsheet, and go.

Just like we did in Peeling Phone Numbers [Hack #49], you can save Google web search results to a file, and then process them into a comma-delimited text file with a short Perl script.

Tip

Be sure to set your preferences [Hack #1] to 100 results per page to get the most out of this hack.

The Code

#!/usr/bin/perl
# google2csv.pl # Google Web Search Results exported to CSV suitable
# for import into Excel 
# Usage: perl google2csv.pl < results.html > results.csv

print qq{"title","url","size","domain suffix"\n};

my($results) = (join '', <>) =~
  m!<div>(.*?)</div>!mis;

while ( $results =~ 
m!<p><a href="?(.+?)"?>(.+?)</a>.+?\s+-\s+(\d+k)?!mgis
) { 
  my($url,$title, $size) = ($1||'',$2||'',$3||''); 
  my($suffix) = $url =~ m!\.(\w+)/!; 
  $title =~ s!"!""!g;         # double escape " marks 
  $title =~ s!<.+?>!!g; # drop all HTML tags
  print qq{"$title","$url","$size","$suffix"\n};
}

Running the Script

Run the script from the command line, specifying the result’s HTML filename and name of the CSV file you wish to create or to which you ...