34 Preparing for DB2 Near-Realtime Business Intelligence
application integration (EAI) systems already. You may be able then to use those
EAI systems to capture the data and route it to the data warehouse. What is
important is that you are able to capture the required data as it changes, and
immediately provide it to some delivery mechanism.
3.5.2 Deliver
Next is the Deliver function. As mentioned in 3.4, “Going realtime, nearly” on
page 30, you can probably use some type of asynchronous delivery mechanism
between the operational systems and the data warehouse environment. Two
common techniques to accomplish this asynchronous communication use
relational tables and message queues. It is likely that both techniques may be
deployed to meet various business and technological requirements. There is
actually a third mechanism which can be considered a delivery mechanism, but it
only moves the data on-demand. That mechanism is federated technology. It can
extend the access to data to include data that can’t, or shouldn’t, be kept in the
data warehouse.
Relational tables
Let’s first explore how relational tables can be used. Refer to Figure 3-5 on
page 35 as you read this discussion.
The fact that the use of relational tables is so pervasive and easy can help make
the evolution to realtime a little easier. It is technology that we understand and
that is supported by many great data management tools. However, the way in
which we use it will perhaps be different.
In 3.5.1, “Capture” on page 31, we discussed how the capture function can put
the data changes into a relational table. Three of the four scenarios we discussed
could put the changed data into a relational table, the modified application (Circle
3), a Web Service (Circle 4), and relational database capture programs (Circle
1). Once the operational application has committed the data to the table it can
forget about the data, and it becomes independent of whatever application will
then read that data. This is because the data is protected via normal relational
database logging and backup capabilities.
Chapter 3. Architectural considerations 35
Figure 3-5 Delivering changed data via relational tables
The deliver function becomes a very simple matter of providing relational access
to these tables. The tables could be local to the data warehouse environment or,
more likely, remote to the data warehouse environment. They might even be in
different databases or different database management systems. We can use
federation technology to hide the fact that these might even be relational
database products from different vendors. This will simplify the access by the
transform function by allowing it to access one dialect of the SQL language and
functionality.
Message queues
If you don’t want to use relational or if you don’t want to physically write the data
to disk, you can utilize asynchronous messaging middleware. You may already
have this technology in your company in the form of Enterprise Application
Integration (EAI) software, which is usually implemented with some type of
messaging hub. You may even be able to capture data that is already flowing
through the message hub to feed the data warehouse.
Applications
messages
changes
Databases
DBMS
Web Services
messages
1
3
4
Deliver
Transform
Applications
messages
changes
Databases
DBMS
Web Services
messages
1
3
4
Deliver
Transform

Get Preparing for DB2 Near-Realtime Business Intelligence now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.