97 Things Every Data Engineer Should Know

Chapter 82. Three Important Distributed Programming Concepts

Adi Polak

Many data engineers create pipelines for extract, transform, load (ETL) or extract, load, transform (ELT) operations. During a transform (T) task, you might be working with data that fits in one machine’s memory. However, often the data will require you to use frameworks/solutions that leverage distributed parallel computation to achieve the desired goal. To support that, many researchers have developed models of distributed programming and computation embodied in known frameworks such as Apache Spark, Apache Cassandra, Apache Kafka, TensorFlow, and more. Let’s look at the three most used distributed programming models for data analytics and distributed machine learning.

MapReduce Algorithm

MapReduce is a distributed computation algorithm developed by Google in 2004. As developers, we specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. This approach is an extension of the split-apply-combine strategy for data analysis.

In practice, every task is split into multiple map and reduce functions. Data is distributed over multiple nodes/machines, and each chunk of data is processed on a node. A logic function is applied to the data on that node, and later the reduce ...