Processing stages are created declaratively via a pipeline specification in Pachyderm. If you have worked with Kubernetes before, this type of interaction should be familiar. Basically, we declare to Pachyderm what processing we want to take place, and Pachyderm handles all the details to make sure that this processing happens as declared.
Let's first create the model stage of our pipeline using a JSON pipeline specification that is stored in train.json. This JSON specification is as follows:
{ "pipeline": { "name": "model" }, "transform": { "image": "dwhitena/goregtrain:single", "cmd": [ "/goregtrain", "-inDir=/pfs/training", "-outDir=/pfs/out" ] }, "parallelism_spec": { "constant": "1" }, "input": ...
