Updates to Agent Image Tag not picked up by pod template

[shezaan-hashgraph]

Describe the bug

I had the following agent configuration in my values.yaml file upon initial jenkins installation

agent:
  image:
    repository: gcr.io/xxxxx/jenkins-agent
    tag: 2024-08-19
  imagePullPolicy: IfNotPresent
  # Max number of spawned agents
  containerCap: 0
  # Set to something otherwise it defaults to `sleep`
  command: "jenkins-agent"
  nodeSelector:
    workloads-jenkins-agents: "true"
  podName: jenkins-agent
  resources:
    requests:
      cpu: "2500m"
      memory: "2500Mi"
    limits:
      cpu: "5000m"
      memory: "5000Mi"

A few weeks later, I tried updating the image tag to tag: 2024-09-19. The agents that came up thereafter still used the old tag that was used upon initial Jenkins setup.

Version of Helm and Kubernetes

- Helm: version.BuildInfo{Version:"v3.16.1", GitCommit:"5a5449dc42be07001fd5771d56429132984ab3ab", GitTreeState:"dirty", GoVersion:"go1.23.1"}

- Kubernetes: 

Client Version: v1.31.1
Kustomize Version: v5.4.2
Server Version: v1.29.7-gke.1104000

### Chart version

5.6.0

### What happened?

- Updated the agent image tag
- Applied the changes via the command `kubectl kustomize --enable-helm | kubectl apply -f -`
- The main jenkins pod successfully restarted
- The configmap entry for `jcasc-default-config.yaml` shows the newly applied agent image tag.
- The pod template did not get updated with the new tag and subsequent jenkins-agents continue to use the old image tag used during the initial Jenkins setup.

### What you expected to happen?

The pod template should be automatically updated and the jenkins-agent images should use the new image tag.

### How to reproduce it

```markdown
- Update the agent image tag
- Apply the changes via the command `kubectl kustomize --enable-helm | kubectl apply -f -`
- Make sure main jenkins pod successfully restarts
- Make sure the configmap entry for `jcasc-default-config.yaml` shows the newly applied agent image tag.
- The pod template should get updated with the new tag and subsequent jenkins-agents should now use the new jenkins-agent image tag.

Anything else we need to know?

No response

[hervelemeur]

Is it a private repository of yours? We don't publish images with this kind of tags AFAIK.

[shezaan-hashgraph]

Is it a private repository of yours? We don't publish images with this kind of tags AFAIK.

Yes it is but it doesn't explain why the Jenkins initial setup would work. Also when I change the tag via the Jenkins UI it works just fine.

[shezaan-hashgraph]

Any update on this?

[ElijahLynn]

Note to myself: need to change in the UI here: Manage Jenkins > Clouds > Kubernetes > Pod Templates > jenkins-agent (edit) > Container Template > Docker image