Hands-On Design Patterns and Best Practices with Julia

Introducing the web crawler use case

Suppose that we have to build a web crawler that can be used to index content from various websites. The process of doing that involves setting up a list of target sites and then kicking off the crawler. Let's create a module with the structure, as follows:

module WebCrawlerusing Dates# public interfaceexport Targetexport add_site!, crawl_sites!, current_sites, reset_crawler!# == insert global variables and functions here ==end # module

Our programming interface is quite simple. Let's see how to do this:

1. Target is a data type that represents the website being crawled. Then, we can use the add_site! function to add new target sites to the list. 
2. When ready, we just call the crawl_sites! function to visit ...