Hands-On Software Engineering with Golang

Fetching the contents of graph links

The link fetcher serves as the first stage of the crawler pipeline. It operates on Payload values emitted by the input source and attempts to retrieve the contents of each link by sending out HTTP GET requests. The retrieved link web page contents are stored within the payload's RawContent field and made available to the following stages of the pipeline.

Let's now take a look at the definition of the linkFetcher type and its associated methods:

type linkFetcher struct {
    urlGetter URLGetter
    netDetector PrivateNetworkDetector
}

func newLinkFetcher(urlGetter URLGetter, netDetector PrivateNetworkDetector) *linkFetcher {
    return &linkFetcher{
        urlGetter: urlGetter,
        netDetector: netDetector,
    }
}

func (lf *linkFetcher) ...