Processing a large number of files in parallel

If you have worked through the two previous recipes, you should already have in place a framework for detecting the presence of a variable number of files in a designated location. You will also have developed a method for processing each of those files as they appear in a prepared list (that is, a table). But consider a situation where there are hundreds or even thousands of files to be processed. Managing all of those files serially would likely prove to be a bottleneck.

In this recipe, we will enhance the file-processing framework by introducing a way to execute the most critical components in parallel.

Getting ready

We will start from where we left off with the previous recipe and to do that, we ...

Get Oracle Data Integrator 11g Cookbook now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.