Hands-On Software Engineering with Golang

Extracting outgoing links from retrieved webpages

The task of the link extractor is to scan the body of each retrieved HTML document and extract the unique set of links contained within it. Each uniform resource locator (URL) in a web page can be classified into one of the following categories:

• URL with a network path reference ^[1]: This type of link is quite easy to identify as it does not include a URL scheme (for example, <img src="//banner.png"/>). When the web browser (or crawler, in our case) needs to access the link, it will substitute the protocol used to access the web page that contained it. Consequently, if the parent page was accessed via HTTPS, then the browser will also request the banner image over HTTPS.
• Absolute links: These ...