Skip to Main Content
The Go Programming Language
book

The Go Programming Language

by Alan A. A. Donovan, Brian W. Kernighan
October 2015
Beginner to intermediate content levelBeginner to intermediate
400 pages
14h 44m
English
Addison-Wesley Professional
Content preview from The Go Programming Language

8.6 Example: Concurrent Web Crawler

In Section 5.6, we made a simple web crawler that explored the link graph of the web in breadth-first order. In this section, we’ll make it concurrent so that independent calls to crawl can exploit the I/O parallelism available in the web. The crawl function remains exactly as it was in gopl.io/ch5/findlinks3:

gopl.io/ch8/crawl1
func crawl(url string) []string {
    fmt.Println(url)
    list, err := links.Extract(url)
    if err != nil {
        log.Print(err)
    }
    return list
}

The main function resembles breadthFirst (§5.6). As before, a worklist records the queue of items that need processing, each item being a list of URLs to crawl, but this time, instead of representing the queue using a slice, ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

The Rust Programming Language, 2nd Edition

The Rust Programming Language, 2nd Edition

Steve Klabnik, Carol Nichols
Programming Rust, 2nd Edition

Programming Rust, 2nd Edition

Jim Blandy, Jason Orendorff, Leonora F. S. Tindall

Publisher Resources

ISBN: 9780134190570Purchase book