optimize armada api requests during container step

[mladjan-gadzic]

feedback from mvp testing:

Looking at the plugin code, I think there is a new connection instantiated to receive events for every armadaContainer step, as this seems to be implemented in the KubernetesNodeContext here: https://github.com/armadaproject/jenkins-plugin/blob/main/src/main/java/io/armadaproject/jenkins/plugin/pipeline/KubernetesNodeContext.java#L74 . Is my assessment correct? If so unfortunately this is less than ideal, because this will make armada emit all the events produced by all the jobs in the day's jobset for ever single armadaContainer call, this will cause a very significant load on the armada api...could you look into centralizing the event watching? The plugin should have only one connection to armada to receive events for the jobset.

[mladjan-gadzic]

idea is to cetralize armada connection add caching mechanism to its client. similar to k8s plugin https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/KubernetesClientProvider.java#L54

[mladjan-gadzic]

quick brainstorm:

- add armada client with observer pattern per cloud instance
- once cloud is created start observing immediatelly
    - check how to handle errors if connection cannot be established
- add two variables for jobsetid
    - current jobsetid
    - old jobsetid
- create two structures
    - one holds current jobsetid events
    - other holds old jobseid events
    - once jobsetid changes current structure empties and that data goes to old (emptying it beforehand)
- filter from structure (current or old) for events based on current jobseit