Enabling insights and analytics with data streaming architectures and pipelines using Kafka and Hadoop

In a large global health services company, streaming data for processing and sharing comes with its own challenges. Data science and analytics platforms need data fast, from relevant sources, to act on this data quickly and share the insights with consumers with the same speed and urgency.

Join Mohammad Quraishi (Cigna) to learn why streaming data architectures are a necessity—Kafka and Hadoop are key. Mohammad outlines architectures centered around the Hadoop Platform and Kafka that were implemented to support a variety of integration and analytics requirements.

Topics include:

• Enabling streaming to and from relational sources and files using custom frameworks that automate and speed up workflows
• Combining the polyglot techniques with Kafka API to support various streaming solutions
• Combining data driven techniques to support consumers through a simple streaming architecture and microservices
• How HBase, Kudu, and Kafka Streams are used to reduce latency between these microservices and frontend application APIs
• Enabling the consumption and sharing of data sources and results using streams
• Enabling Spark Structured Streaming, Flink, and Spark ML on these streams
• Enabling data sync between on-premises data lakes and the cloud
• Supporting cloud native architectures that enable machine learning in the cloud

This session was recorded at the 2019 O'Reilly Strata Data Conference in San Francisco.