Inconsistent variable substitution in Inputs.Parameters block

[encigem]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

I'm having some issue with variable substitutions for 'inputs.paramters' blocks in Workflow files.
For example, i've provided a YAML file which reproduces my issue. In 'step1' the name: "{{workflow.parameters.cm-name-full}}" works within the configMapKeyRef , but with the name: "{{workflow.parameters.cm-name}}-cm" in step2 it does not.

What Happened?: The step with partial variable string input failed, but the step with input consisting only of a string with a variable succeeded.

What I expected?: I expected that both of the steps should've been able to substitute the given variable in the 'inputs.parameters' block

Version(s)

v3.5.10, latest(b26ed4a)

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: v1
kind: ConfigMap
data:
  testVar: testVal
metadata:
  labels:
    workflows.argoproj.io/configmap-type: Parameter
  name: test-cm
  namespace: argo

---

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: substitute-test-
  namespace: argo
spec:
  entrypoint: steps
  arguments:
    parameters:
      - name: image
        value: "alpine:3.17"
      - name: cm-key
        value: "testVar"
      - name: cm-name
        value: "test"
      - name: cm-name-full
        value: "test-cm"
  templates:
  - name: steps
    steps:
    - - name: step1
        template: step1
    - - name: step2
        template: step2

  - name: step1
    inputs:
      parameters:
        - name: cm-var
          valueFrom:
            configMapKeyRef:
              name: "{{workflow.parameters.cm-name-full}}" # only variable defined; succeeds
              key: "{{workflow.parameters.cm-key}}"
    container:
      image: "{{workflow.parameters.image}}"
      command: [sh, -c]
      args: ["echo 'testVar value is: {{inputs.parameters.cm-var}}'"]

  - name: step2
    inputs:
      parameters:
        - name: cm-var
          valueFrom:
            configMapKeyRef:
              name: "{{workflow.parameters.cm-name}}-cm"  # combine variable with string; fails
              key: "{{workflow.parameters.cm-key}}"
    container:
      image: "{{workflow.parameters.image}}"
      command: [sh, -c]
      args: ["echo 'testVar value is: {{inputs.parameters.cm-var}}'"]

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

time="2024-11-08T16:25:42.231Z" level=info msg="Processing workflow" Phase= ResourceVersion=725462700 namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.237Z" level=info msg="Task-result reconciliation" namespace=test-il numObjs=0 workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.237Z" level=info msg="Updated phase  -> Running" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.237Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.237Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.237Z" level=info msg="Steps node substitute-test-fpg66 initialized Running" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.238Z" level=info msg="StepGroup node substitute-test-fpg66-2823940808 initialized Running" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.238Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.238Z" level=info msg="Pod node substitute-test-fpg66-296026515 initialized Pending" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.264Z" level=info msg="Created pod: substitute-test-fpg66[0].step1 (substitute-test-fpg66-step1-296026515)" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.265Z" level=info msg="Workflow step group node substitute-test-fpg66-2823940808 not yet completed" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.265Z" level=info msg="TaskSet Reconciliation" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.265Z" level=info msg=reconcileAgentPod namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:42.286Z" level=info msg="Workflow update successful" namespace=test-il phase=Running resourceVersion=725462704 workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.266Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=725462704 namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.267Z" level=info msg="Task-result reconciliation" namespace=test-il numObjs=1 workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.267Z" level=info msg="task-result changed" namespace=test-il nodeID=substitute-test-fpg66-296026515 workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node changed" namespace=test-il new.message= new.phase=Succeeded new.progress=0/1 nodeID=substitute-test-fpg66-296026515 old.message= old.phase=Pending old.progress=0/1 workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="Step group node substitute-test-fpg66-2823940808 successful" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66-2823940808 phase Running -> Succeeded" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66-2823940808 finished: 2024-11-08 16:25:52.268414392 +0000 UTC" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="StepGroup node substitute-test-fpg66-3964671805 initialized Running" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="SG Outbound nodes of substitute-test-fpg66-296026515 are [substitute-test-fpg66-296026515]" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="Skipped node substitute-test-fpg66-3148240723 initialized Error (message: unable to retrieve inputs.parameters.cm-var from ConfigMap: ConfigMap '{{workflow.parameters.cm-name}}-cm' does not exist. Please make sure it has the label workflows.argoproj.io/configmap-type: Parameter to be detectable by the controller)" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=error msg="Mark error node" error="step group deemed errored due to child substitute-test-fpg66[1].step2 error: unable to retrieve inputs.parameters.cm-var from ConfigMap: ConfigMap '{{workflow.parameters.cm-name}}-cm' does not exist. Please make sure it has the label workflows.argoproj.io/configmap-type: Parameter to be detectable by the controller" namespace=test-il nodeName="substitute-test-fpg66[1]" workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66-3964671805 phase Running -> Error" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66-3964671805 message: step group deemed errored due to child substitute-test-fpg66[1].step2 error: unable to retrieve inputs.parameters.cm-var from ConfigMap: ConfigMap '{{workflow.parameters.cm-name}}-cm' does not exist. Please make sure it has the label workflows.argoproj.io/configmap-type: Parameter to be detectable by the controller" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66-3964671805 finished: 2024-11-08 16:25:52.268667688 +0000 UTC" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="step group substitute-test-fpg66-3964671805 was unsuccessful: step group deemed errored due to child substitute-test-fpg66[1].step2 error: unable to retrieve inputs.parameters.cm-var from ConfigMap: ConfigMap '{{workflow.parameters.cm-name}}-cm' does not exist. Please make sure it has the label workflows.argoproj.io/configmap-type: Parameter to be detectable by the controller" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="Outbound nodes of substitute-test-fpg66-3148240723 is [substitute-test-fpg66-3148240723]" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="Outbound nodes of substitute-test-fpg66 is [substitute-test-fpg66-3148240723]" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66 phase Running -> Failed" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66 message: step group deemed errored due to child substitute-test-fpg66[1].step2 error: unable to retrieve inputs.parameters.cm-var from ConfigMap: ConfigMap '{{workflow.parameters.cm-name}}-cm' does not exist. Please make sure it has the label workflows.argoproj.io/configmap-type: Parameter to be detectable by the controller" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="node substitute-test-fpg66 finished: 2024-11-08 16:25:52.268736598 +0000 UTC" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="TaskSet Reconciliation" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg=reconcileAgentPod namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="Updated phase Running -> Failed" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="Updated message  -> step group deemed errored due to child substitute-test-fpg66[1].step2 error: unable to retrieve inputs.parameters.cm-var from ConfigMap: ConfigMap '{{workflow.parameters.cm-name}}-cm' does not exist. Please make sure it has the label workflows.argoproj.io/configmap-type: Parameter to be detectable by the controller" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.268Z" level=info msg="Marking workflow completed" namespace=test-il workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.300Z" level=info msg="Workflow update successful" namespace=test-il phase=Failed resourceVersion=725462793 workflow=substitute-test-fpg66
time="2024-11-08T16:25:52.302Z" level=info msg="Queueing Failed workflow test-il/substitute-test-fpg66 for delete in 168h0m0s due to TTL"
time="2024-11-08T16:26:02.382Z" level=info msg="Queueing Failed workflow test-il/substitute-test-fpg66 for delete in 167h59m50s due to TTL"
time="2024-11-08T16:46:02.378Z" level=info msg="Queueing Failed workflow test-il/substitute-test-fpg66 for delete in 167h39m50s due to TTL"
time="2024-11-08T17:06:02.389Z" level=info msg="Queueing Failed workflow test-il/substitute-test-fpg66 for delete in 167h19m50s due to TTL"
time="2024-11-08T17:25:52.321Z" level=info msg="cleaning up pod" action=deletePod key=test-il/substitute-test-fpg66-step1-296026515/deletePod
time="2024-11-08T17:26:02.383Z" level=info msg="Queueing Failed workflow test-il/substitute-test-fpg66 for delete in 166h59m50s due to TTL"

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

time="2024-11-11T13:15:40.526Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2024-11-11T13:15:40.526Z" level=info msg="No output parameters"
time="2024-11-11T13:15:40.526Z" level=info msg="No output artifacts"
time="2024-11-11T13:15:40.527Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: artifacts/2024/11/11/substitute-test-7bbdj/substitute-test-7bbdj-step1-1047580867/main.log"
time="2024-11-11T13:15:40.527Z" level=info msg="Creating minio client using static credentials" endpoint="seliis3obj.seli.gic.company.se:10444"
time="2024-11-11T13:15:40.527Z" level=info msg="Saving file to s3" bucket=testdev-moore030 endpoint="seliis3obj.seli.gic.company.se:10444" key=artifacts/2024/11/11/substitute-test-7bbdj/substitute-test-7bbdj-step1-1047580867/main.log path=/tmp/argo/outputs/logs/main.log
time="2024-11-11T13:15:40.601Z" level=info msg="Save artifact" artifactName=main-logs duration=74.237363ms error="<nil>" key=artifacts/2024/11/11/substitute-test-7bbdj/substitute-test-7bbdj-step1-1047580867/main.log
time="2024-11-11T13:15:40.601Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-11-11T13:15:40.601Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2024-11-11T13:15:40.632Z" level=info msg="Alloc=7710 TotalAlloc=17424 Sys=22357 NumGC=5 Goroutines=10"

[tooptoop4]

this should be marked as feature not bug