In a large global health services company, streaming data for processing and sharing comes with its own challenges. Data science and analytics platforms need data fast, from relevant sources, to act on this data quickly and share the insights with consumers with the same speed and urgency.
Join Mohammad Quraishi (Cigna) to learn why streaming data architectures are a necessity—Kafka and Hadoop are key. Mohammad outlines architectures centered around the Hadoop Platform and Kafka that were implemented to support a variety of integration and analytics requirements.
- Enabling streaming to and from relational sources and files using custom frameworks that automate and speed up workflows
- Combining the polyglot techniques with Kafka API to support various streaming solutions
- Combining data driven techniques to support consumers through a simple streaming architecture and microservices
- How HBase, Kudu, and Kafka Streams are used to reduce latency between these microservices and frontend application APIs
- Enabling the consumption and sharing of data sources and results using streams
- Enabling Spark Structured Streaming, Flink, and Spark ML on these streams
- Enabling data sync between on-premises data lakes and the cloud
- Supporting cloud native architectures that enable machine learning in the cloud
This session was recorded at the 2019 O'Reilly Strata Data Conference in San Francisco.
- Title: Enabling insights and analytics with data streaming architectures and pipelines using Kafka and Hadoop
- Release date: October 2019
- Publisher(s): O'Reilly Media, Inc.
- ISBN: 0636920339854
You might also like
O'Reilly Strata Data and AI Superstream
You’ll get access to O’Reilly data and AI experts. Deep dives into some of the hottest …
O'Reilly Software Architecture Conference 2019 - New York, New York
What do you really need to consider when adopting a microservices architecture? How do you unite …
O'Reilly Software Architecture Conference 2019 - San Jose, California
The O'Reilly Software Architecture Conference San Jose 2019 (SACON) gathered the world's leading software architects and …
OSCON 2019 - Portland, Oregon
OSCON Portland 2019 brought together a vibrant and diverse collection of talented speakers (open source leaders …