Frank Kane's Taming Big Data with Apache Spark and Python

Managing dependencies

Another thing I want to talk about is managing dependencies. Something to remember is, even if something runs perfectly on your desktop, it doesn't mean it will run on your cluster. This is because you might not have the same Java environment, the same Scala environment, or the same Python environment. Another point to make is that the same data won't be there. If your scripts are referring to absolute file paths or file paths on your local desktop system, you're going to want to make sure you fix those before you run them on the cluster. What you could do is refer to a path that's on some distributed file system that each node has access to, or use things such as broadcast variables to actually share data outside of ...