Preparing for DB2 Near-Realtime Business Intelligence

34 Preparing for DB2 Near-Realtime Business Intelligence

application integration (EAI) systems already. You may be able then to use those

EAI systems to capture the data and route it to the data warehouse. What is

important is that you are able to capture the required data as it changes, and

immediately provide it to some delivery mechanism.

3.5.2 Deliver

Next is the Deliver function. As mentioned in 3.4, “Going realtime, nearly” on

page 30, you can probably use some type of asynchronous delivery mechanism

between the operational systems and the data warehouse environment. Two

common techniques to accomplish this asynchronous communication use

relational tables and message queues. It is likely that both techniques may be

deployed to meet various business and technological requirements. There is

actually a third mechanism which can be considered a delivery mechanism, but it

only moves the data on-demand. That mechanism is federated technology. It can

extend the access to data to include data that can’t, or shouldn’t, be kept in the

data warehouse.

Relational tables

Let’s first explore how relational tables can be used. Refer to Figure 3-5 on

page 35 as you read this discussion.

The fact that the use of relational tables is so pervasive and easy can help make

the evolution to realtime a little easier. It is technology that we understand and

that is supported by many great data management tools. However, the way in

which we use it will perhaps be different.

In 3.5.1, “Capture” on page 31, we discussed how the capture function can put

the data changes into a relational table. Three of the four scenarios we discussed

could put the changed data into a relational table, the modified application (Circle

3), a Web Service (Circle 4), and relational database capture programs (Circle

1). Once the operational application has committed the data to the table it can

forget about the data, and it becomes independent of whatever application will

then read that data. This is because the data is protected via normal relational

database logging and backup capabilities.

Chapter 3. Architectural considerations 35

Figure 3-5 Delivering changed data via relational tables

The deliver function becomes a very simple matter of providing relational access

to these tables. The tables could be local to the data warehouse environment or,

more likely, remote to the data warehouse environment. They might even be in

different databases or different database management systems. We can use

federation technology to hide the fact that these might even be relational

database products from different vendors. This will simplify the access by the

transform function by allowing it to access one dialect of the SQL language and

functionality.

Message queues

If you don’t want to use relational or if you don’t want to physically write the data

to disk, you can utilize asynchronous messaging middleware. You may already

have this technology in your company in the form of Enterprise Application

Integration (EAI) software, which is usually implemented with some type of

messaging hub. You may even be able to capture data that is already flowing

through the message hub to feed the data warehouse.

Applications

messages

changes

Databases

DBMS

Web Services

messages

1

3

4

Deliver

Transform

Applications

messages

changes

Databases

DBMS

Web Services

messages

1

3

4

Deliver

Transform