Enterprise Service Bus

Chapter 9. Batch Transfer Latency

This chapter will examine the most common form of integration being done today: the practice of performing bulk data transfer and batch updating using Extract, Transform, and Load (ETL) techniques. We will discuss the business impact of the latency and reliability issues associated with this method of integration, and examine prescriptive steps for migrating toward real-time integration using an ESB in the context of a case study using pattern sketches.

Chapter 2 presented statistical data showing that enterprises aren’t as integrated as you might think. Another stark reality is that most organizations don’t have a good snapshot of their critical data at any given point in time. Amazon.com is well known for being completely automated, with instant access to all aspects of information in their supply chain, but this is the extreme exception to the norm.

Due to the latency of batch updates and rekeying information from “sneakernet” integration, and the margins of error associated with those common practices, there is always some latency between business events and the processing and recording of the associated data. In a supply chain, this can result in not knowing what’s in inventory at any given time. The business impact from this method of integration can cause large amounts of business capital to be unnecessarily tied up.

ETL integration generally consists of a complex maze of applications, scripts, manual processes, and FTP file transfers. ...