The Future of Machine Intelligence

Chapter 10. Reza Zadeh: On the Evolution of Machine Learning

Reza Zadeh is a consulting professor at the Institute for Computational and Mathematical Engineering at Stanford University and a technical advisor to Databricks. His work focuses on machine learning theory and applications, distributed computing, and discrete applied mathematics.

Key Takeaways

• Neural networks have made a comeback and are playing a growing role in new approaches to machine learning.

• The greatest successes are being achieved via a supervised approach leveraging established algorithms.

• Spark is an especially well-suited environment for distributed machine learning.

Tell us a bit about your work at Stanford.

At Stanford, I designed and teach distributed algorithms and optimization (CME 323) as well as a course called discrete mathematics and algorithms (CME 305). In the discrete mathematics course, I teach algorithms from a completely theoretical perspective, meaning that it is not tied to any programming language or framework, and we fill up whiteboards with many theorems and their proofs.

On the more practical side, in the distributed algorithms class, we work with the Spark cluster programming environment. I spend at least half my time on Spark. So all the theory that I teach in regard to distributed algorithms and machine learning gets implemented and made concrete by Spark, and then put in the hands of thousands of industry and academic folks who use commodity clusters.

I started running MapReduce ...