Hands-On Software Engineering with Golang

Indexing the contents of retrieved web pages

The last component in our pipeline is the text indexer. As the name implies, the text indexer is responsible for keeping the search index up to date by reindexing the content of each crawled web page.

In a similar fashion to the graph updater stage, we apply the single-responsibility principle and define the Indexer interface that gets passed to the text indexer component via its constructor:

// Indexer is implemented by objects that can index the contents of webpages retrieved by the crawler pipeline.
type Indexer interface {
    Index(doc *index.Document) error
}

type textIndexer struct {
    indexer Indexer
}

func newTextIndexer(indexer Indexer) *textIndexer {
    return &textIndexer{ indexer: indexer, ...