Building a web spider using goroutines and channels
Let's take the largely useless capitalization application and do something practical with it. Here, our goal is to build a rudimentary spider. In doing so, we'll accomplish the following tasks:
- Read five URLs
- Read those URLs and save the contents to a string
- Write that string to a file when all URLs have been scanned and read
These kinds of applications are written every day, and they're the ones that benefit the most from concurrency and non-blocking code.
It probably goes without saying, but this is not a particularly elegant web scraper. For starters, it only knows a few start points—the five URLs that we supply it. Also, it's neither recursive nor is it thread-safe in terms of data integrity. ...