January 2020
Intermediate to advanced
640 pages
16h 56m
English
Content preview from Hands-On Software Engineering with Golang
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
Start your free trial
The link extractor
The last crawler component that we will be examining is the link extractor. It scans retrieved HTML documents and attempts to identify and extract all links present inside.
Link extraction is unfortunately not a trivial task. While it's true that the majority of links can be extracted via a bunch of regular expressions, there are a few edge-cases that require additional logic from our end, as in the following examples:
- Relative links need to be converted into absolute links.
- If the document <head> section includes the <base href="xxx"> tag, we need to parse it and use its content to rewrite relative links.
- We might encounter links that do not specify a protocol. These special links begin with // and are commonly used when ...