As youâve seen so far in this book, amazing things can be done with Google data access without ever using the Google API. This section of the book deals with Google applications that scrape Googleâs HTML to access its data rather than use the sanctioned Google API.
What is scraping and how is it different from using the Google API? Scraping is the act of using a program to pull information from an HTML page.
The Google API, on the other hand, allows you to query Googleâs search data directly, instead of pulling information from saved HTML as the scrapers in this section do.
Why have Google scrapers in this book anyway? Canât you do everything with the Google API? Alas, you canât. The Google API is a great way to search Googleâs main body of web pages, but it doesnât go much further than that. Itâs even limited in what it can pull from Googleâs main web search. For example, the Google API canât do a phonebook search. And it canât access the data from Google News, Google Catalogs, or most of Googleâs other specialty search properties.
Thatâs too bad, because cool things can be done with the data from those searches. Need to track news stories for a certain topic? Itâs a lot easier to access several compiled searches from a spreadsheet than to manually scan HTML pages. Plus, once the information is loaded into a spreadsheet, you can resort and manipulate the data just about any way you please.
Though the programs provided in this section will provide you with hours of fun Google scraping, there are a few things youâll need to keep in mind.
Scrapers break These scrapers are built based on the format of the Google results at this writing. If the format of results changes significantly, scrapers can, and will, break.
Donât automate your scraping It might be tempting to go one step further and create programs that automate retrieving and scraping of Google pages.
Donât do it. Retrieving Google pages by any automated methods other than Googleâs API is against Googleâs Terms of Service (TOS). So what, you might think. After all, they canât find you, right? They might not be able to find you specifically, but they can ban access from an entire block of IP addresses based on your IP address, affecting you and others around you. Would they do this? They would and they have. See http://news.com.com/2100-1023-883558.html for information about a Comcast block that took place in early 2002.
Search results have limits Remember that even though youâre scraping saved result pages, youâre still subject to the limitations of Googleâs searchâyou canât get more than 1,000 results for a web search, for example. That said, make sure that youâve set up your web search to get the most out of each scraped page. Make sure youâre getting 100 results per page.
Once youâve got some results, scraped them, and saved them to a comma-delimited file, what use are they?
Most of the time, you think of comma-delimited files as spreadsheet files, but theyâre more versatile than that. Comma-delimited files can be loaded into databases and different data-handling programs. You could even write a Perl program that did something interesting with comma-delimited files (or get a geek friend to write it for you).
But the best thing about comma-delimited files is that they age well. Theyâre just text files. You can use them with several different programs, you donât have to worry about compatibility issues, and theyâre small.
Get Google Hacks now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.