Fetching the contents of graph links 

The link fetcher serves as the first stage of the crawler pipeline. It operates on Payload values emitted by the input source and attempts to retrieve the contents of each link by sending out HTTP GET requests. The retrieved link web page contents are stored within the payload's RawContent field and made available to the following stages of the pipeline.

Let's now take a look at the definition of the linkFetcher type and its associated methods:

Get Hands-On Software Engineering with Golang now with the O’Reilly learning platform.

O’Reilly members experience live online training, plus books, videos, and digital content from nearly 200 publishers.