Most workflow variables are missing (not being interpolated)

[jobinkavalam]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

Workflow variables - https://argo-workflows.readthedocs.io/en/latest/variables/#dag-templates - that I expect to be available are not present.

Specifically, for the Dag Templates only the tasks.name variable is available; others such as tasks.<TASKNAME>.exitCode is not.

dag-exit-code-kn7bh-combine-2072748230: debug:{name:combine}
dag-exit-code-kn7bh-combine-2072748230: exit code is {{=tasks.test.exitCode}}

Expected: exit code is 0 but got exit code is {{=tasks.test.exitCode}}
Expected: debug: to log a map with lot more variables such as those listed in the documentation. But only got one key name

Version(s)

v3.5.11

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

{
   "apiVersion": "argoproj.io/v1alpha1",
   "kind": "Workflow",
   "metadata": {
      "annotations": {
         "workflows.argoproj.io/description": "This workflow has two tasks. the first one exits with code 0. The second tries to query that exit code."
      },
      "generateName": "dag-exit-code-",
      "labels": {
         "workflows.argoproj.io/archive-strategy": "false"
      }
   },
   "spec": {
      "entrypoint": "main",
      "templates": [
         {
            "dag": {
               "tasks": [
                  {
                     "name": "test",
                     "template": "test"
                  },
                  {
                     "name": "combine",
                     "depends": "test",
                     "template": "combine"
                  }
               ]
            },
            "name": "main"
         },
         {
            "container": {
               "args": [
                  "sh",
                  "-c",
                  "exit 0"
               ],
               "image": "busybox"
            },
            "name": "test"
         },
         {
            "name": "combine",
            "script": {
               "command": [
                  "sh"
               ],
               "image": "busybox",
               "source": "echo debug:{{=toJson(tasks)}}; echo exit code is {{=tasks.test.exitCode}}"
            }
         }
      ]
   }
}

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}


time="2024-10-29T07:44:43.329Z" level=info msg="Processing workflow" Phase= ResourceVersion=149852 namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:43.332Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:43.332Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:43.332Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:43.332Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:43.332Z" level=info msg="DAG node dag-exit-code-kn7bh initialized Running" namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:43.332Z" level=warning msg="was unable to obtain the node for dag-exit-code-kn7bh-2072748230, taskName combine"
time="2024-10-29T07:44:43.332Z" level=warning msg="was unable to obtain the node for dag-exit-code-kn7bh-570652817, taskName test"
time="2024-10-29T07:44:43.332Z" level=warning msg="was unable to obtain the node for dag-exit-code-kn7bh-570652817, taskName test"
time="2024-10-29T07:44:43.332Z" level=info msg="All of node dag-exit-code-kn7bh.test dependencies [] completed" namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:43.332Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:43.332Z" level=info msg="Pod node dag-exit-code-kn7bh-570652817 initialized Pending" namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:43.338Z" level=info msg="Created pod: dag-exit-code-kn7bh.test (dag-exit-code-kn7bh-test-570652817)" namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:43.338Z" level=warning msg="was unable to obtain the node for dag-exit-code-kn7bh-2072748230, taskName combine"
time="2024-10-29T07:44:43.338Z" level=warning msg="was unable to obtain the node for dag-exit-code-kn7bh-2072748230, taskName combine"
time="2024-10-29T07:44:43.338Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:43.338Z" level=info msg=reconcileAgentPod namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:43.345Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=149856 workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:53.331Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=149856 namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:53.331Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:53.331Z" level=info msg="task-result changed" namespace=argo nodeID=dag-exit-code-kn7bh-570652817 workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:53.331Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=dag-exit-code-kn7bh-570652817 old.message= old.phase=Pending old.progress=0/1 workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:53.332Z" level=warning msg="was unable to obtain the node for dag-exit-code-kn7bh-2072748230, taskName combine"
time="2024-10-29T07:44:53.332Z" level=warning msg="was unable to obtain the node for dag-exit-code-kn7bh-2072748230, taskName combine"
time="2024-10-29T07:44:53.332Z" level=warning msg="was unable to obtain the node for dag-exit-code-kn7bh-2072748230, taskName combine"
time="2024-10-29T07:44:53.332Z" level=info msg="All of node dag-exit-code-kn7bh.combine dependencies [test] completed" namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:53.332Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:53.332Z" level=info msg="Pod node dag-exit-code-kn7bh-2072748230 initialized Pending" namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:53.339Z" level=info msg="Created pod: dag-exit-code-kn7bh.combine (dag-exit-code-kn7bh-combine-2072748230)" namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:53.339Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:53.339Z" level=info msg=reconcileAgentPod namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:53.350Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=149889 workflow=dag-exit-code-kn7bh
time="2024-10-29T07:44:53.357Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/dag-exit-code-kn7bh-test-570652817/labelPodCompleted
time="2024-10-29T07:45:03.340Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=149889 namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:45:03.340Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=2 workflow=dag-exit-code-kn7bh
time="2024-10-29T07:45:03.340Z" level=info msg="task-result changed" namespace=argo nodeID=dag-exit-code-kn7bh-2072748230 workflow=dag-exit-code-kn7bh
time="2024-10-29T07:45:03.340Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=dag-exit-code-kn7bh-2072748230 old.message= old.phase=Pending old.progress=0/1 workflow=dag-exit-code-kn7bh
time="2024-10-29T07:45:03.341Z" level=info msg="Outbound nodes of dag-exit-code-kn7bh set to [dag-exit-code-kn7bh-2072748230]" namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:45:03.341Z" level=info msg="node dag-exit-code-kn7bh phase Running -> Succeeded" namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:45:03.341Z" level=info msg="node dag-exit-code-kn7bh finished: 2024-10-29 07:45:03.341058217 +0000 UTC" namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:45:03.341Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:45:03.341Z" level=info msg=reconcileAgentPod namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:45:03.341Z" level=info msg="Updated phase Running -> Succeeded" namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:45:03.341Z" level=info msg="Marking workflow completed" namespace=argo workflow=dag-exit-code-kn7bh
time="2024-10-29T07:45:03.346Z" level=info msg="Workflow update successful" namespace=argo phase=Succeeded resourceVersion=149924 workflow=dag-exit-code-kn7bh
time="2024-10-29T07:45:03.361Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/dag-exit-code-kn7bh-combine-2072748230/labelPodCompleted

### Logs from in your workflow's wait container

```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded


time="2024-10-29T07:44:58.676Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2024-10-29T07:44:58.676Z" level=info msg="No output parameters"
time="2024-10-29T07:44:58.676Z" level=info msg="No output artifacts"
time="2024-10-29T07:44:58.676Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: dag-exit-code-kn7bh/dag-exit-code-kn7bh-combine-2072748230/main.log"
time="2024-10-29T07:44:58.676Z" level=info msg="Creating minio client using static credentials" endpoint="minio:9000"
time="2024-10-29T07:44:58.676Z" level=info msg="Saving file to s3" bucket=my-bucket endpoint="minio:9000" key=dag-exit-code-kn7bh/dag-exit-code-kn7bh-combine-2072748230/main.log path=/tmp/argo/outputs/logs/main.log
time="2024-10-29T07:44:58.681Z" level=info msg="Save artifact" artifactName=main-logs duration=5.071833ms error="<nil>" key=dag-exit-code-kn7bh/dag-exit-code-kn7bh-combine-2072748230/main.log
time="2024-10-29T07:44:58.681Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-10-29T07:44:58.681Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2024-10-29T07:44:58.686Z" level=info msg="Alloc=7933 TotalAlloc=14037 Sys=24677 NumGC=4 Goroutines=10"
time="2024-10-29T07:44:48.527Z" level=info msg="No output artifacts"
time="2024-10-29T07:44:48.527Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: dag-exit-code-kn7bh/dag-exit-code-kn7bh-test-570652817/main.log"
time="2024-10-29T07:44:48.527Z" level=info msg="Creating minio client using static credentials" endpoint="minio:9000"
time="2024-10-29T07:44:48.527Z" level=info msg="Saving file to s3" bucket=my-bucket endpoint="minio:9000" key=dag-exit-code-kn7bh/dag-exit-code-kn7bh-test-570652817/main.log path=/tmp/argo/outputs/logs/main.log
time="2024-10-29T07:44:48.532Z" level=info msg="Save artifact" artifactName=main-logs duration=5.485291ms error="<nil>" key=dag-exit-code-kn7bh/dag-exit-code-kn7bh-test-570652817/main.log
time="2024-10-29T07:44:48.532Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2024-10-29T07:44:48.532Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2024-10-29T07:44:48.537Z" level=info msg="Alloc=7853 TotalAlloc=14082 Sys=24933 NumGC=4 Goroutines=10"
time="2024-10-29T07:44:48.539Z" level=info msg="Deadline monitor stopped"
time="2024-10-29T07:44:48.539Z" level=info msg="stopping progress monitor (context done)" error="context canceled"

[tooptoop4]

https://github.com/argoproj/argo-workflows/blob/v3.6.0-rc4/workflow/controller/dag.go#L323-L351 seems to be relevant section. maybe @smile-luobin knows as worked on #6684