Chapter 7. Big Data stack
This chapter covers
- Adding reliability to the data store from chapter 6
- Managing a distributed persistent data store in CoreOS
- Simulating failures in the data system
In this chapter, you’ll build a Big Data aggregation platform that seeds a database with random search queries against Twitter.
You’ll build a small corpus of data, make Twitter rate-limit you (while still being a good API citizen), and see how to take
care of your mission-critical (although random) data. Your application will function like this:
- Six stateless workers will generate a random word and search for it on the Twitter API.
- The results will be stored in Couchbase.
- Workers will continue to search every 100 ms in parallel until they’re ...