Example 2 – looping with XPath and scraping data from multiple pages

In example 1, we tried the simple XPath-based technique for a URL with a limited number of results on a single page. In this case, we will be targeting a food and drink category, that is, http://books.toscrape.com/catalogue/category/books/food-and-drink_33/index.html, which has its content across pages. An XPath-based looping operation will be used in this example, which supports a more effective collection of data.

As we will be dealing with multiple pages, it's good practice to check for a few individual page URLs that can be found in the browser while moving through the listed pages. Most of the time, it might contain some patterns that can solve the puzzle easily, as ...

Get Hands-On Web Scraping with Python now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.