Preparing for DB2 Near-Realtime Business Intelligence

44 Preparing for DB2 Near-Realtime Business Intelligence

3.6 Putting it together

In the previous sections, we have looked at each architectural function as an

individual entity. An advantage of segmenting the overall process into these

functions is that we can then introduce some type of data buffering mechanism

between the functions to help compensate for natural differences in arrival rates

and processing throughput between the functions.

While, in theory, each of these functions can be independent of the others, in

reality the Capture and Deliver functions are very closely related - while the

Transform and Apply functions are closely related. However, the Capture/Deliver

functions can operate in a very different manner than the Transform/Apply

functions. For example, the Capture/Deliver functions may use batch or

near-realtime techniques, while the Transform/Apply functions use batch or

continuous techniques, but independently.

As you can see in Table 3-1 on page 44, we can organize these approaches into

four categories. The first category (Batch-Batch) actually represents the status of

the ETL processes where we are performing batch processing in an offline

database batch window. We can also see that there is not a requirement to

immediately implement an environment with all processes being near-realtime,

because we have other choices. One of these choices will hopefully satisfy your

requirement. In the following sections, we look at examples of each of these

choice scenarios.

Table 3-1 CDTA approaches

Batch-Continuous

In this scenario you will still be capturing and delivering the data in batches,

much as you may do currently. These batches may be delivered as files, as

selects to remote databases, or even as batches of messages. However, instead

of waiting to process them in a batch window when the data warehouse is offline,

we can process the data and update the data warehouse as the batches arrive,

or at some interval, while the data warehouse is still online and actively being

queried by users.

CAPTURE/DELIVER TRANSFORM/APPLY DATABASE

Batch Batch Offline

Batch Continuous Online

Near-Realtime Batch Online

Near-Realtime Near-Realtime Online