Hands-On Software Engineering with Golang

Implementing a source and a sink for the crawler

A prerequisite for executing the crawler pipeline is to provide an input source that conforms to the pipeline.Source interface and an output sink that implements pipeline.Sink. We have discussed both these interfaces in the previous sections, but I am copying their definitions as follows for reference:

type Source interface {
    Next(context.Context) bool
    Payload() Payload
    Error() error
}

type Sink interface {
    Consume(context.Context, Payload) error
}

In Chapter 6, Building a Persistence Layer, we put together the interface of the link graph component and came up with two alternative, concrete implementations. One of the methods of the graph.Graph interface that is of particular interest at this ...