Virtually every enterprise depends on big data analysis, but distributed computing environments such as Hadoop and Spark are complicated, to say the least. Multiple users, business units, and workload types often compete for valuable computing resources. Monitoring tools are not well equipped to handle this level of complexity, and typically provide only very high-level and historical information. The lack of fine-grained visibility for making real-time adjustments to running workloads means that high-priority jobs can easily be pushed aside by lower-priority jobs.
It’s time to bring Quality of Service (QoS) to distributed processing in multi-tenant Hadoop environments. This O’Reilly report explains how QoS allows operators to assign priorities to jobs, ensuring that higher-priority tasks get the resources needed to meet critical deadlines. Author Andy Oram examines the critical role of performance in the evolution of operating systems, data warehouses, and distributed processing. He also discusses Quasar (part of Mesos) and Pepperdata, two tools that can help improve performance in distributed computing environments.
You’ll discover how tools that help ensure QoS can help distributed environments evolve to accommodate:
- Multiple users contending for resources, such as those on operating systems
- Jobs that grow or shrink in hardware usage, so they don’t strain at resource limits or let resources go to waste
- Jobs of different priorities, including soft real-time requirements that allow them to override lower-priority or adhoc jobs
- Performance guarantees, similar to service-level agreements (SLAs)
Table of contents
- Title: Hadoop and Spark Performance for the Enterprise
- Release date: June 2016
- Publisher(s): O'Reilly Media, Inc.
- ISBN: 9781491963197
You might also like
Architecting Modern Data Platforms
There’s a lot of information about big data technologies, but splicing these technologies into an end-to-end …
Need to move a relational database application to Hadoop? This comprehensive guide introduces you to Apache …
Data Science from Scratch, 2nd Edition
To really learn data science, you should not only master the tools—data science libraries, frameworks, modules, …
Data in all domains is getting bigger. How can you work with it efficiently? Recently updated …