AAW dev: general nodepools track resource usage

[Jose-Matsuda]

EPIC
Follow up to #1997

We want to observe if there are any workloads being booted off / acting weirdly.

[jacek-dudek]

Here is a table of resource usage metrics comparing the discrepancies between actual usage and resource requests before request adjustments were made and after adjustments were made.
filtered-resource-utilization-on-aaw-dev-general-nodes.ods
There is a problem here, in that some of the workloads seem to have reverted back to the resource requests that were in effect before the adjustments I made. I checked the source code changes that I made in previous issue to update the requests. Those are still in place. Perhaps I made changes in the wrong place? I'm posting a topic on elab for it. For the ones where the changes persisted the actual usage tracks the updated requests better than previously.

[jacek-dudek]

I got some of these outstanding workloads to update their resource requests by manually syncing in argocd web app with help from Mathis. I noticed another issue in that there is an additional container defined running istio-proxy that bumps up the requests, so they're not matching actual usage as well as they could be. I missed those containers in making the original resource request updates, so I will be making another commit to fix that issue.