Crawler code –

The following code snippet represents the constructor of the crawler class. It initializes all the relevant instance variables. logger is one of the custom classes written to log debug messages, so that if any error occurs during the execution of the crawler, which will have been spawned as a subprocess and will be running in the background, it can be debugged:

Let's now take a look at the start() method of the crawler, from where the crawling actually begins:

It can be seen in section (1), which will be true ...

Get Hands-On Penetration Testing with Python now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.