Go Web Scraping Quick Start Guide

Atomic counters

In the previous example, we surrounded any increment or decrement of activeThreads with a Lock() / Unlock(). This can become quite verbose if it needs to be done multiple times, as in our case. The sync package offers a subpackage called atomic that provides methods for updating objects without the need for locks. This is done by creating a new variable each time a change is made, while preventing multiple writes from occurring at the same time. The following example shows some of the changes necessary to use activeThreads as an atomic counter:

package mainimport (  "sync"  "sync/atomic")// ...var activeThreads int32 = 0// ...func scrapeSite(site string, condition *sync.Cond) { condition.L.Lock() if activeThreads >= maxActiveThreads ...