Go Web Scraping Quick Start Guide

Queue

Before a web scraper can start collecting information, it needs to know where to go. It also needs to know where it has been. A proper queuing system will accomplish both of these goals. Queues can be set up in many different ways. In many of the previous examples, we used a []string or a map[string]string to hold the target URLs the scraper should pursue. This works for smaller scale web scrapers where the work is being pushed to the workers.

In larger applications, a work-stealing queue would be preferred. In a work-stealing queue, the worker threads would take the first available job out of the queue as fast as they can accomplish the task. This way, if you need your system to increase throughput, you can simply add more worker threads. ...