Video description
This year, Twitter open sourced two powerful real-time analytics tools -- DistributedLog, a high-performance log service, and Heron, a distributed stream computation system.
A few weeks after Heron was open sourced, Karthik Ramasamy, engineering manager and technical lead for real-time analytics at Twitter, delivered a talk at Strata + Hadoop World in London to unveil the system and discuss:
- An overview of Heron as a micro stream engine and its architectural components
- How Twitter has been running Heron in production
- The operational experience and challenges of running Heron at scale, including a discussion of stragglers
- Heron's minimal resource usage and performance numbers
Leading up to Twitter's open sourcing of DistributedLog, software engineer and tech lead of the DistributedLog project Sijie Guo spoke at Strata + Hadoop World in San Jose to introduce the service. Key components of his talk include:
- Why Twitter built DistributedLog
- Technical decisions and challenges behind building DistributedLog
- How Twitter uses DistributedLog to support different workloads
- How Twitter runs the same software stack in multiple data centers to achieve global consistency
Publisher resources
Product information
- Title: Twitter's Real-Time Data Stack
- Author(s):
- Release date: July 2016
- Publisher(s): O'Reilly Media, Inc.
- ISBN: 9781491969694
You might also like
video
Creating an extensible 100+ PB real-time big data platform by unifying storage and serving
Uber relies heavily on making data-driven decisions in every product area and needs to store and …
video
ML at Twitter: A deep dive into Twitter's timeline
Machine learning has allowed Twitter to drive engagement, promote healthier conversations, and deliver catered advertisements. Cibele …
book
Kotlin Blueprints
Get to know the building blocks of Kotlin and best practices when using quality world-class applications …
video
Panel: Real-time Discussions on Real-time Data processing
How do streaming experts define “streaming”? What are their favorite use cases of real-time streaming, and …