Processing a large number of files in parallel
If you have worked through the two previous recipes, you should already have in place a framework for detecting the presence of a variable number of files in a designated location. You will also have developed a method for processing each of those files as they appear in a prepared list (that is, a table). But consider a situation where there are hundreds or even thousands of files to be processed. Managing all of those files serially would likely prove to be a bottleneck.
In this recipe, we will enhance the file-processing framework by introducing a way to execute the most critical components in parallel.
Getting ready
We will start from where we left off with the previous recipe and to do that, we ...
Get Oracle Data Integrator 11g Cookbook now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
