Learning Python Networking - Second Edition

XPath expressions

To use Scrapy, it is necessary to define rules that Scrapy will use for extracting information. These rules can be XPath expressions. Scrapy has an interpreter that allows you to test XPath expressions on a website, which facilitates the debugging and development of web spiders. For example, if we want to extract the text corresponding to the title of the page, we can do with the '//title/text()' XPath expression:

>>> fetch('http://www.scrapy.org') >>> response.xpath('//title/text()').extract() >>> ['Scrapy | A Fast and Powerful Scraping and Web Crawling Framework']

In the following screenshot, you can see the result of the execution of the fetch command in the Scrapy shell and extract the title of the page with the XPath ...