Hands-On Software Engineering with Golang

Extracting the title and text from retrieved web pages

The next stage of the pipeline is responsible for extracting an index-friendly, text-only version of the web page contents and its title. The easiest way to achieve this is by stripping off any HTML tag in the page body and replacing consecutive whitespace characters with a single space.

A fairly straightforward approach would be to come up with a bunch of regular expressions for matching and then removing HTML tags. Unfortunately, the fact that HTML syntax is quite forgiving (that is, you can open a tag and never close it) makes HTML documents notoriously hard to properly clean up just with the help of regular expressions. Truth be told, to cover all possible edge cases, we need to use ...