Today’s approach to processing streaming data is based on legacy big-data centric architectures, the cloud, and the assumption that organizations have access to data scientists to make sense of it all—leaving organizations increasingly overwhelmed.
Simon Crosby (SWIM.AI) shares a new architecture for edge intelligence that turns this thinking on its head. Edge intelligence (encompassing analytics, learning and prediction, and edge computing) can frequently be accomplished on the fly on streaming data, cheaply, at the edge, without data scientists. Simon demonstrates how you can save up to $5,000 a month in cloud processing and storage costs while delivering accurate predictions that can transform outcomes, using well-established architectural pillars, such as the distributed actor model, to process voluminous real-time data at the edge, along with the rich commons of open source analytics and learning tools like Flink and Spark, on nothing more than a $200 device such as an NVIDIA Jetson.
The key insight is to use streaming data to build a digital twin model on the fly at the edge, avoiding a ton of complexity and infrastructure costs. Instead, a user defines the entities in their environment (e.g., traffic intersections, compressors, or assembly robots) that deliver data. Using the stateful distributed actor model, you can dynamically build a digital twin (actor) model of the real-world from the data, linking twins based on their relationships. Each digital twin reduces, labels, and analyzes its data and self-trains a machine learning model to predict future performance, at the edge, discarding the original data. This method needs only a tiny fraction of the resources of a big data solution and delivers results in real time. As a result, it bypasses the dev, ops, and data science challenges of edge intelligence, effectively turning devices into data scientists—or at least, building data science twins for entities in the real world.
This session was recorded at the 2019 O'Reilly Artificial Intelligence Conference in New York.
Table of contents
- Title: Turn devices into data scientists—at the edge
- Release date: October 2019
- Publisher(s): O'Reilly Media, Inc.
- ISBN: 0636920339434
You might also like
Software Architecture Fundamentals, Second Edition
Being a successful software architect is more than just possessing technical knowledge. It’s about thinking like …
Strata Data Conference - New York, NY 2018
The chief data officer for Goldman Sachs, a cofounder of the blockchain computing platform Ethereum, Google …
Designing Data-Intensive Applications
Data is at the center of many challenges in system design today. Difficult issues need to …
Strata Data Conference 2019 - San Francisco, California
Thousands of the data scientists, analysts, engineers, developers, and executives converged at the Strata Data Conference …