O'Reilly logo

Spark for Python Developers by Amit Nandi

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Building a reliable and scalable streaming app

Ingesting data is the process of acquiring data from various sources and storing it for processing immediately or at a later stage. Data consuming systems are dispersed and can be physically and architecturally far from the sources. Data ingestion is often implemented manually with scripts and rudimentary automation. It actually calls for higher level frameworks like Flume and Kafka.

The challenges of data ingestion arise from the fact that the sources are physically spread out and are transient which makes the integration brittle. Data production is continuous for weather, traffic, social media, network activity, shop floor sensors, security, and surveillance. Ever increasing data volumes and rates ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required