Streaming Systems

Chapter 6. Streams and Tables

You have reached the part of the book where we talk about streams and tables. If you recall, back in Chapter 1, we briefly discussed two important but orthogonal dimensions of data: cardinality and constitution. Until now, we’ve focused strictly on the cardinality aspects (bounded versus unbounded) and otherwise ignored the constitution aspects (stream versus table). This has allowed us to learn about the challenges brought to the table by the introduction of unbounded datasets, without worrying too much about the lower-level details that really drive the way things work. We’re now going to expand our horizons and see what the added dimension of constitution brings to the mix.

Though it’s a bit of a stretch, one way to think about this shift in approach is to compare the relationship of classical mechanics to quantum mechanics. You know how in physics class they teach you a bunch of classical mechanics stuff like Newtonian theory and so on, and then after you think you’ve more or less mastered that, they come along and tell you it was all bunk, and classical physics gives you only part of the picture, and there’s actually this other thing called quantum mechanics that really explains how things work at a lower level, but it didn’t make sense to complicate matters up front by trying to teach you both at once, and...oh wait...we also haven’t fully reconciled everything between the two yet, so just squint at it and trust us that it all makes sense somehow? ...