Go Web Scraping Quick Start Guide

Logs

One system that is often overlooked during the design of a scraping system is the logging system. It is important, first and foremost, to have clear log statements without logging too many unnecessary items. These statements should be informing the operator of the current status of scraping and any errors, or successes, the scraper encounters. This helps you get a picture of the overall health of your web scraper.

The simplest logging that can be done is printing messages to the terminal with println() or fmt.Println() type statements. This works well enough for a single node, but, as your scraper grows into a distributed architecture, it causes problems. In order to check how things are running in your system an operator would need ...