44 Preparing for DB2 Near-Realtime Business Intelligence
3.6 Putting it together
In the previous sections, we have looked at each architectural function as an
individual entity. An advantage of segmenting the overall process into these
functions is that we can then introduce some type of data buffering mechanism
between the functions to help compensate for natural differences in arrival rates
and processing throughput between the functions.
While, in theory, each of these functions can be independent of the others, in
reality the Capture and Deliver functions are very closely related - while the
Transform and Apply functions are closely related. However, the Capture/Deliver
functions can operate in a very different manner than the Transform/Apply
functions. For example, the Capture/Deliver functions may use batch or
near-realtime techniques, while the Transform/Apply functions use batch or
continuous techniques, but independently.
As you can see in Table 3-1 on page 44, we can organize these approaches into
four categories. The first category (Batch-Batch) actually represents the status of
the ETL processes where we are performing batch processing in an offline
database batch window. We can also see that there is not a requirement to
immediately implement an environment with all processes being near-realtime,
because we have other choices. One of these choices will hopefully satisfy your
requirement. In the following sections, we look at examples of each of these
choice scenarios.
Table 3-1 CDTA approaches
Batch-Continuous
In this scenario you will still be capturing and delivering the data in batches,
much as you may do currently. These batches may be delivered as files, as
selects to remote databases, or even as batches of messages. However, instead
of waiting to process them in a batch window when the data warehouse is offline,
we can process the data and update the data warehouse as the batches arrive,
or at some interval, while the data warehouse is still online and actively being
queried by users.
CAPTURE/DELIVER TRANSFORM/APPLY DATABASE
Batch Batch Offline
Batch Continuous Online
Near-Realtime Batch Online
Near-Realtime Near-Realtime Online

Get Preparing for DB2 Near-Realtime Business Intelligence now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.