Go Web Scraping Quick Start Guide

Avoiding loops

If you are building a web scraper that follows links, you might need to be aware of which pages you've already visited. It's quite possible that a page you are visiting contains a link to a page you have already visited, sending you into an infinite loop. Therefore, it is very important to build a tracking system into your scraper that records its history.

The simplest data structure for storing a unique collection of items would be a set. The Go standard library does not have a set data structure, but it can be emulated by using a map[string]interface{}{}.

An interface{} in Go is a generic object, similar to java.lang.Object.

In Go, you can define a map as follows:

visitedMap := map[string]interface{}{}

In this case, we would ...