Design – Build – Run: Applied Practices and Principles for Production-Ready Software Development

16.3. Multiple Threads and Multiple Jobs

The preceding section touched on the possibility of the batch processor executing its processing in a separate thread to support stop and possibly other commands issued by the manager. This section takes the concept of multi-threading further and examines the benefits of multi-threaded processing in the batch components, specifically in the processing.

There are many situations in batch where multi-threading can improve the overall end-to-end transaction processing time by performing multiple operations in parallel. However, there is generally a limit where multi-threaded processing on a single machine becomes counter-productive and it is far better to perform processing across a number of machines. The sections that follow discuss:

• Multi-threaded jobs

• Multiple jobs

• Batch job parameters

16.3.1. Multi-Threaded Jobs

Getting the most out of multi-threaded processing is really about examining the individual process and ascertaining where separate worker threads or processing threads would generally improve the end-to-end performance. The first and most important thing to take into account when multi-threading is to ensure that each worker thread doesn't impact any of the others. Each thread needs to be working on distinct and separate items as well as using resources effectively.

Assume that you have a batch job that is archiving and/or removing log files and other transient files from a number of different servers around the estate. The manager's ...