Processing a large number of files in parallel
If you have worked through the two previous recipes, you should already have in place a framework for detecting the presence of a variable number of files in a designated location. You will also have developed a method for processing each of those files as they appear in a prepared list (that is, a table). But consider a situation where there are hundreds or even thousands of files to be processed. Managing all of those files serially would likely prove to be a bottleneck.
In this recipe, we will enhance the file-processing framework by introducing a way to execute the most critical components in parallel.
We will start from where we left off with the previous recipe and to do that, we ...