Distributed Computing with Python

Common problems – software environments

Another common challenge is making sure that the software installed on all the various machines we are ever going to use is consistent and consistently upgraded.

Unfortunately, it is frustratingly common to spend hours debugging a distributed application only to discover that for some unknown and seemingly impossible reason, some computers had an old version of the code and/or its dependencies. Sometimes, we might even find the code to have disappeared completely.

The reasons for these discrepancies can be many: from a mount point that failed, to a bug in our deployment procedures, to a simple human mistake.

A common approach, especially in the HPC world, is to always create a self-contained environment for ...