May 2017
Beginner to intermediate
596 pages
15h 2m
English
Content preview from Data Lake for Enterprises
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
Start your free trial
Three tier design - Flume architecture
Most of the Flume deployments follow a well documented three tier design. The three main tiers of this design are:
- Agent Tier: This is the tier where the Flume agents are located along with sources which contains data which have to be moved.
- Collector Tier: This is the tier where the data from the agent tier is collected using multiple collectors and then these are forwarded to the next layer.
- Storage Tier: This is the tier where data from collector tier flows finally and stored. This will have file systems like HDFS where the data is stored.
The following figure (Figure 11) shows the three tier design of a Flume Architecture in action:
Figure 11: The three-tier design of Flume architecture