Native Docker Clustering with Swarm

Deploying Spark, again

We choose a host where we want to run the Spark standalone manager, be aws-105, and tag it as such:

docker node update --label-add type=sparkmaster aws-105

Other nodes will host our Spark workers.

We start the Spark master on aws-105:

$ docker service create \
--container-label spark-master \
--network spark \
--constraint 'node.labels.type == sparkmaster' \
--publish 8080:8080 \
--publish 7077:7077 \
--publish 6066:6066 \
--name spark-master \
--replicas 1 \
--env SPARK_MASTER_IP=0.0.0.0 \
--mount type=volume,target=/data,source=spark,volume-driver=flocker 
    \
fsoppelsa/spark-master

First, the image. I discovered that there are some annoying things included into the Google images (such as unsetting some environment variables, ...