Now that we have executed the codes in parallel using PP to dispatch the local processes, it is time to verify that the code is executed in a distributed way. For this, we will use the following three different machines:
The idea is to dispatch the executions to the three machines listed using PP. For this, we will make use of a case study of the Web crawler. In the code of
web_crawler_pp_cluster.py, for each URL informed in the
input_list, we will dispatch a local or remote process for execution, and at the end of each execution, a
callback function will group the URLs and their first three links found. ...