Skip to Content
Practical Predictive Analytics
book

Practical Predictive Analytics

by Ralph Winters
June 2017
Beginner to intermediate content levelBeginner to intermediate
576 pages
15h 22m
English
Packt Publishing
Content preview from Practical Predictive Analytics

Cleaning up and caching the table in memory

Since Spark excels at processing in-memory data, we will first remove our intermediary data and then cache our out_sd dataframe, so that subsequent queries run much faster. Caching data in memory works best when similar types of queries are repeated. In that way, Spark is able to know how to juggle memory so that most of what you need resides in memory.

However, this is not foolproof. Good Spark query and table design will help with optimization, but out-of-the-box caching usually gives some benefit. Often, the first queries will not benefit from memory caching, but subsequent queries will run much faster.

Since we will no longer use the intermediary dataframes we created, we will remove them with ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

Data Superstream: Analytics Engineering

Data Superstream: Analytics Engineering

Alistair Croll, Anna Filippova, Emilie Schario, Lewis Davies, Jacob Frackson, Benn Stancil, Nick Acosta, Elizabeth Caley
R: Predictive Analysis

R: Predictive Analysis

Tony Fischetti, Eric Mayor, Rui Miguel Forte
Python: Advanced Predictive Analytics

Python: Advanced Predictive Analytics

Ashish Kumar, Joseph Babcock

Publisher Resources

ISBN: 9781785886188Supplemental Content