Hands-On Software Engineering with Golang

Assembling and running the pipeline

Congratulations for making it this far! We have finally implemented all individual components that are required for constructing a pipeline for our crawler service. All that's left is to add a little bit of glue code to assemble the individual crawler stages into a pipeline and provide a simple API for running a full crawler pass. All this glue logic is encapsulated inside the Crawler type whose definition and constructor details are listed as follows:

type Crawler struct {
    p *pipeline.Pipeline
}

// NewCrawler returns a new crawler instance.
func NewCrawler(cfg Config) *Crawler {
    return &Crawler{
        p: assembleCrawlerPipeline(cfg),
    }
}

The Config type holds all required configuration options for creating ...