Hands-On Deep Learning with Go

Creating a data pipeline

Now we need to specify a data pipeline in standard JSON. Here, we are mapping a repository to a directory and executing our network in either training or inference mode.

Let's look at our cifar_cnn.json file:

{ "pipeline": {    "name": "cifarcnn"  },  "transform": {    "image": "cifarcnn:latest",    "cmd": [  "go run main.go"    ]  },  "enable_stats": true,  "parallelism_spec": {    "constant": "1"  },  "input": {    "pfs": {      "repo": "data",      "glob": "/"    }  }}

The options we've chosen here are straightforward, and you can see the references to the Docker image, commands, and switches, as well as repo and the mount point we're specifying. One thing to note is the parallelism_spec option. Setting this above the default of 1 allows us to scale ...