Common datasets

[carlfischerjba]

Support for cross-model data on Azure

Platform 2 model workers can now specify volumes with optional custom subpaths. This allows workers for different models to share files and thus avoids some duplication of data.

Example:

    volumes:
      # Existing behaviour: `AZURE_MODELS_ROOT/supplier/model/version/` mounted at `/home/worker/model`.
      - name: oasis-models-azure-file
        type: model-data
        mountPath: /home/worker/model
        source: azureFiles
      # New option: `AZURE_MODELS_ROOT/common/datasets/` mounted at `/datasets`.
      - name: oasis-models-azure-file
        type: model-data
        mountPath: /datasets
        source: azureFiles
        subPath: common/datasets

#870

[carlfischerjba]

This might not quite work.

Error: INSTALLATION FAILED: 2 errors occurred:
	* Deployment.apps "worker-jba-xx-1.0.0" is invalid: spec.template.spec.volumes[2].name: Duplicate value: "oasis-models-azure-file"
	* Job.batch "model-registration-210xwb" is invalid: spec.template.spec.containers[0].volumeMounts[3].mountPath: Invalid value: "/mnt/model_data_0": must be unique

[carlfischerjba]

Closing. It's more complicated than this.

Currently the volumes from the workers values are used for two things: creating the volumes and creating the volumeMounts. It's fine to have several volumeMounts with the same name, i.e. referencing the same volume, perhaps using different subPaths. But the volumes themselves need to be unique.