Hands-On Design Patterns and Best Practices with Julia

Overview of a high-performance solution

The beauty of SharedArrays is that the data is maintained as a single copy, and multiple processes can have both read and write access. It is a perfect solution to our problem.

In this solution, we will do the following:

1. The master program creates a shared array.
2. Using a distributed for loop, the master program commands worker processes to read each individual file into a specific segment of the array.
3. Again, using a distributed for loop, the master program commands worker process to perform statistical analysis.

As we have 16 vCPUs, we can utilize all of them.

In practice, we should probably utilize fewer vCPUs so that we can leave some room for the operating system itself. Your mileage may vary ...