volumeClaimTemplates in WorkflowTemplate works only with workflowTemplateRef and not templateRef

[apiwoni]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

volumeClaimTemplates defined in WorkflowTemplate does not work when workflow template referred to via templateRef.
It errors with the following message volume 'workdir' not found in workflow spec

However, it works when WorkflowTemplate is referred to via workflowTemplateRef.
Documentation does not mention anything about this limitation.

Use case:
Multiple pods running in parallel in the same instance of workflow can only use ReadWriteOnce access mode but this does not work when pods are scheduled on different nodes. Since there's no requirement for pods to share the same volume we would like to create volume per pod and the number of pods varies so number of volumes cannot be determined upfront.

Version(s)

v3.5.12

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

#Converted https://github.com/argoproj/argo-workflows/blob/main/examples/volumes-pvc.yaml to WorkflowTemplate as follows:

# This example demonstrates the ability for a workflow to create a
# temporary, ephemeral volume used by the workflow, and delete it
# when the workflow completes. It uses the same volumeClaimTemplates
# syntax as statefulsets.
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: volumes-pvc-wftmpl
spec:
  entrypoint: volumes-pvc-example
  volumeClaimTemplates:
  - metadata:
      name: workdir
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 64Mi

  templates:
  - name: volumes-pvc-example
    steps:
    - - name: generate
        template: whalesay
    - - name: print
        template: print-message

  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [sh, -c]
      args: ["echo generating message in volume; cowsay hello world | tee /mnt/vol/hello_world.txt"]
      volumeMounts:
      - name: workdir
        mountPath: /mnt/vol

  - name: print-message
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["echo getting message from volume; find /mnt/vol; cat /mnt/vol/hello_world.txt"]
      volumeMounts:
      - name: workdir
        mountPath: /mnt/vol
--- 
# This works
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: volumes-pvc-
spec:
  entrypoint: volumes-pvc-example
  workflowTemplateRef:
    name: volumes-pvc-wftmpl
---
# This does not work
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: volumes-pvc2-
spec:
  entrypoint: main
  templates:
  - name: main
    dag:
      tasks:
      - name: failed-volume
        templateRef:
          name: volumes-pvc-wftmpl
          template: volumes-pvc-example

Logs from the workflow controller

N/A

Logs from in your workflow's wait container

N/A

[shuangkun]

I think volumeClaimTemplates should be explicitly written in the workflow, because the template reference does not include volumeClaimTemplates, while the workflow reference includes this volumeClaimTemplates.

[Joibel]

This is not a bug, it is by design, see #7444 (comment)

However the documentation doesn't make that clear, so a PR would be welcome to fix that.

I would also suggest that this should be fixed, so converting this to a feature request would be good. I don't think it's easy to fix in a good way though, it would require quite a lot of work.

[apiwoni]

It would be great to convert this to feature request since there's no technical reason why we should not be able to configure pods to have their own PVC which does not share storage.
As a workaround for this limitation, I have used workflow-of-workflows pattern where new instance of workflow is created in a loop withItems or withParam since I can utilize workflowTemplateRef in this pattern.

[Joibel]

As a feature request things like PDBs should also become "per template" as well.