Building Real-Time Data Pipelines

Chapter 7. Getting to Operational Systems

Operational systems are at times mistakenly conflated with online transaction processing (OLTP) systems. They are in fact not the same. While operational systems process day-to-day transactions similarly to OLTP systems, they can also perform batch processing similarly to online analytical processing (OLAP) systems. An operational system is the system that processes daily transactions, but its use does not end there. The appropriate operational system for your enterprise can also enable real-time analysis, reporting, and decision making.

Getting to that ideal operational system requires choosing the appropriate technological components. Modern technology available today makes the choice simpler. It is important to consider several guiding principles.

Have Fewer Systems Doing More 

There are two schools of thought here around this subject—“best of breed” and “consolidation.” You will typically hear various vendors speak differently about both these approaches.

With the “best of breed” approach, you can add or remove components to or from your system, and ensure that only the “best” software for each of your needs are in your architecture. This example works well in theory, and promises that you will always have the best software for all your use cases without getting locked into one vendor. The reality is that in many cases the “best of breed” software solution for one usage scenario does not integrate well with the “best of breed” solution ...