Python Web Scraping - Second Edition

Edge cases

The AJAX search script is quite simple, but it can be simplified further by utilizing possible edge cases. So far, we have queried each letter, which means 26 separate queries, and there are duplicate results between these queries. It would be ideal if a single search query could be used to match all results. We will try experimenting with different characters to see if this is possible. This is what happens if the search term is left empty:

>>> url = 'http://example.webscraping.com/ajax/search.json?page=0&page_size=10&search_term=' >>> requests.get(url).json()['num_pages'] 0 

Unfortunately, this did not work-there are no results. Next we will check if '*' will match all results:

>>> requests.get(url + '*').json()['num_pages'] ...