Scattersearch with Yahoo! and Google
Sometimes, illuminating results can be found when scraping from one site and feeding the results into the API of another. With scattersearching, you can narrow down the most popular related results, as suggested by Yahoo! and Google.
We’ve combined a scrape of a Yahoo! web page with a Google search [Hack #41] , blending scraped data with data generated via a web service API to good effect. In this hack, we’re doing something similar, except this time we’re taking the results of a Yahoo! search and blending it with a Google search.
Yahoo! has a “Related searches” feature, where you enter a search term and get a list of related terms under the search box, if any are available. This hack scrapes those related terms and performs a Google search for the related terms in the title. It then returns the count for those searches, along with a direct link to the results. Aside from showing how scraped and API-generated data can live together in harmony, this hack is good to use when you’re exploring concepts; for example, you might know that something called Pokemon exists, but you might not know anything about it. You’ll get Yahoo!’s related searches and an idea of how many results each of those searches generates in Google. From there, you can choose the search terms that generate the most results or look the most promising based on your limited knowledge, or you can simply pick a road that appears less traveled.
The Code
Save the following code to a file ...
Get Google Hacks, 2nd Edition now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
