Python Penetration Testing Cookbook

How to do it...

Here is how we can use Scrapy to crawl websites which require logging in:

1. To use the FormRequest object, we can update the parse_page method as follows:

def parse(self, response): 
    return scrapy.FormRequest.from_response( 
        response, 
        formdata={'username': 'username', 'password': 'password'}, 
        callback=self.parse_after_login 
     ) 

Here, the response object is the HTTP response of the page where we have to fill in the login form. The FormRequest method includes the credentials that we need to log in and the callback method that is used to parse the page after login.

2. To paginate after logging in while preserving the logged-in session, we can use the method we used in the previous recipe.