Building Real-Time Data Pipelines

Chapter 10. Conclusion

In-memory optimized databases are filling the gap where legacy relational database management systems and NoSQL databases have failed to deliver. By implementing a hybrid data processing model, organizations can obtain instant access to incoming data while gaining faster and more targeted insights. With the ability to process and analyze data as it is being generated, data-driven businesses can detect operational trends as they happen rather than reacting after the fact.

Recommended Next Steps

Now is the time to begin exploring in-memory options. Organizations with a focus on quickly deriving business value from emerging and growing data sources should identify data processing and storage solutions with in-memory storage, compiled query execution, enterprise-ready fault tolerance, and ACID compliance.

To get a competitive advantage from real-time data pipelines, we recommend the following:

• Identify real-time use cases within your organization, prioritizing by selecting processes that will either have the biggest revenue impact or that are easiest to implement.
• Investigate in-memory database solutions available in the market, giving preference to distributed systems that offer a memory optimized architecture.
• Explore leveraging open source frameworks such as Apache Kafka and Apache Spark to streamline data pipelines and enrich data for analysis.
• Select a vendor and run a proof of concept that puts your use case(s) to the test.
• Go to production at a manageable ...