Skip to Main Content
Hands-On Deep Learning with Apache Spark
book

Hands-On Deep Learning with Apache Spark

by Guglielmo Iozzia
January 2019
Intermediate to advanced content levelIntermediate to advanced
322 pages
7h 29m
English
Packt Publishing
Content preview from Hands-On Deep Learning with Apache Spark

Performance considerations

This section presents some recommendations to get the most from DL4J when training on Spark. Let's start with some considerations about memory configuration. It is important to understand how DL4J manages memory first. This framework is built upon the ND4J scientific library (written in C++). ND4J utilizes off-heap memory management—this means that the memory allocated for INDArrays isn't on the JVM heap, as happens for Java objects, but it is allocated outside the JVM. This kind of memory management allows for the efficient use of high-performance native code for numerical operations and it is also necessary for efficient operations with CUDA (https://developer.nvidia.com/cuda-zone) when running on GPUs.

This way, ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

Next-Generation Machine Learning with Spark: Covers XGBoost, LightGBM, Spark NLP, Distributed Deep Learning with Keras, and More

Next-Generation Machine Learning with Spark: Covers XGBoost, LightGBM, Spark NLP, Distributed Deep Learning with Keras, and More

Butch Quinto
Apache Spark Deep Learning Cookbook

Apache Spark Deep Learning Cookbook

Ahmed Sherif, Amrith Ravindra, Michal Malohlava, Adnan Masood

Publisher Resources

ISBN: 9781788994613Supplemental Content