Design – Build – Run: Applied Practices and Principles for Production-Ready Software Development

10.1. Batch Processing

Although batch is traditionally associated with large transaction-processing systems and conjures up visions of overnight processing, batch is very much an integral part of modern systems. Batch can also be referred to as scheduled processing, but essentially it is a process or function that executes in the background. It is important to differentiate batch processes from other processes, such as Web services, that also execute in the background and don't have a user interface. Typically a Web service provides functionality based on a single transaction (known as event-based processing), whereas batch typically operates on larger sets of transactions (known as scheduled tasks).

Batch jobs perform a variety of different tasks in modern computer systems. Some perform a single function, whereas others perform more traditional functions, such as processing large numbers of records, performing intense calculations, and compiling statistical information. There is no real distinction between what each batch job performs; it is merely a process or function that is not included as part of the Online Transaction Processing (OLTP) System.

Figure 10-1 shows a hypothetical high-level "daily lifecycle" which will help when looking at some of the jobs listed shortly. Batch jobs can be divided into two main categories:

• Functional jobs — Functional batch jobs perform business-related actions. For example, a functional batch job might cycle through a set of orders, bundle ...