Scala and Spark for Big Data Analytics

Data serialization

Serialization is an important tuning for performance improvement and optimization in any distributed computing environment. Spark is not an exception, but Spark jobs are often data and computing extensive. Therefore, if your data objects are not in a good format, then you first need to convert them into serialized data objects. This demands a large number of bytes of your memory. Eventually, the whole process will slow down the entire processing and computation drastically.

As a result, you often experience a slow response from the computing nodes. This means that we sometimes fail to make 100% utilization of the computing resources. It is true that Spark tries to keep a balance between convenience and performance. This ...