Chapter 3. Crawlers and Spiders

In this chapter, we will cover:

  • Downloading a page for offline analysis with Wget
  • Downloading a page for offline analysis with HTTrack
  • Using ZAP's spider
  • Using Burp Suite to crawl a website
  • Repeating requests with Burp's repeater
  • Using WebScarab
  • Identifying relevant files and directories from crawling results

Introduction

A penetration test can be performed using different approaches, such as Black, Grey, and White box. A Black box test is performed when the testing team doesn't have any previous information about the application to test other than the URL of the server. A White box test is performed when the team has all the information about the target, its infrastructure, software versions, test users, development information, ...

Get Kali Linux Web Penetration Testing Cookbook now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.