Engineering Resilient Systems on AWS

Chapter 8. Real-Time Market Data Analytics

Edward Calahan invented the ticker tape in 1867, using telegraph lines to transmit updates about stock market activity. The name refers to the rhythmic sound of the machine printing a continuous stream of information. Within a few years, other inventors built upon this protocol, evolving it into the scrolling stock price updates we see on news channels today.

Streaming information has fundamentally transformed the stock industry, enabling market participants to make time-sensitive decisions from virtually anywhere. Businesses have matured their data consumption practices, transforming raw data into actionable insights that fuel their processes with ever-increasing volumes at microsecond latencies.

While the underlying technology has shifted from telegraph to electronic circuits, the core process remains and has been replicated across industries. Today, data publishers organize these streams into distinct topics, optimizing the delivery of relevant information to their consumers. For example, individuals interested solely in stock updates can subscribe to a dedicated topic, avoiding irrelevant noise.

The sequential update pattern from a durable list has broad applicability for microservice synchronization. Imagine two services managing recent stock pricing information and client stock portfolios. Modern architectures avoid sharing databases between microservices for scalability and performance reasons, creating the need for discrete systems ...