Project Tungsten focuses on substantially improving the efficiency of memory and CPU for Spark applications, to push performance closer to the limits of modern hardware. This effort includes three initiatives:1. Memory Management and Binary Processing: leveraging application semantics to manage memory explicitly and eliminate the overhead of JVM object model and garbage collection2. Cache-aware computation: algorithms and data structures to exploit memory hierarchy3. Code generation: using code generation to exploit modern compilers and CPUsProject Tungsten will be the largest change to Spark’s execution engine since the project's inception. In this talk, we will give an update on its progress and dive into some of the technical challenges we are solving.
Table of contents
- Title: Deep Dive into Project Tungsten: Bring Spark Closer to Bare Metal
- Release date: September 2015
- Publisher(s): O'Reilly Media, Inc.
- ISBN: 9781491954843
You might also like
Need to move a relational database application to Hadoop? This comprehensive guide introduces you to Apache …
Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition
Through a series of recent breakthroughs, deep learning has boosted the entire field of machine learning. …
Architecting Modern Data Platforms
There’s a lot of information about big data technologies, but splicing these technologies into an end-to-end …
Data Science from Scratch, 2nd Edition
To really learn data science, you should not only master the tools—data science libraries, frameworks, modules, …