Chapter 8. An example batch layer: Architecture and algorithms
This chapter covers
- Building a batch layer from end to end
- Practical examples of precomputation
- Iterative graph algorithms
- HyperLogLog for efficient set-cardinality operations
You’ve now learned all the pieces of the batch layer: formulating a schema for your data, storing a master dataset, and running computations at scale with a minimum of complexity. In this chapter you’ll tie these pieces together into a coherent batch layer. No new theory is introduced in this chapter—our goal is to reinforce the concepts of the previous chapters by going through a batch layer design from start to finish. There is great value in understanding how the theory maps to a non-trivial example. ...
Get Big Data now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
