Web Crawling with Scrapy – Mapping the Application

In Chapter 2, Interacting with Web Applications, we learned how to interact with a web application programmatically using Python and the requests library. In this chapter, we will cover the following topics:

  • Web application mapping
  • Creating our own crawler/spider with Scrapy
  • Making our crawler recursive
  • Scraping interesting stuff

Get Learning Python Web Penetration Testing now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.