Use scrapeconfig instead of servicemonitor

[csibbitt]

This allows us to create a prometheus config where targets are specified by DNS name, rather than IP addresses.

Solves a problem discovered here: #599 (comment)

Risks:

• This may not work when using observability_strategy: use_community on default deployments of OCP <= 4.14 (at least)
  • the prometheus operator from openshift-monitoring looks new enough, but the scrapeconfig CRD was missing in my test environments
  • observability_strategy: use_redhat w/ obo-prometheus-operator from COO 0.2.0 works fine
• I haven't done additional testing, for example to ensure metrics labeling compatibility with all the other parts of the system, or impact on the (deprecated) HA mode

[csibbitt]

This was missing previously, and I'm not 100% sure the effects. My guess is it would affected garbage collection in the operator-sdk, preventing these servicemonitors from being cleaned up automatically when the servicetelemetry object is deleted.

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/f801d6c3273d471990d68e8f495dbc93

❌ stf-crc-ocp_412-local_build FAILURE in 38m 04s
❌ stf-crc-ocp_414-local_build FAILURE in 37m 39s
❌ stf-crc-ocp_412-local_build-index_deploy FAILURE in 47m 40s
❌ stf-crc-ocp_414-local_build-index_deploy FAILURE in 47m 35s

[csibbitt]

I believe the smoketest failures are related to label changes. I'll take a closer look at this locally and see how bad it is (I see a lot of labels in that part of the code)

[csibbitt]

I adjusted the labels to exactly match what we were producing before and the smoketests passed locally.

[vkmc]

Looks good Chris, thanks! Considering we are removing the service monitors, what happen in brownfield deployments? Do the existing STF continue to run without change and the new scrape config component is deployed with new STF deployments? Thanks!

[csibbitt]

Looks good Chris, thanks! Considering we are removing the service monitors, what happen in brownfield deployments? Do the existing STF continue to run without change and the new scrape config component is deployed with new STF deployments? Thanks!

No. The code as-is deploys the new ScrapeConfig objects, and then deletes the old ServiceMonitor objects[1]
If users have manually configured additional servicemonitors via the "servicemonitor_manifest" option[2], then they will be preserved.

Upgrade testing will be required, but I don't forsee any difficulties. The labels of the new metrics exactly match the old ones.

[1] https://github.com/infrawatch/service-telemetry-operator/pull/607/files#diff-6b852f8c4b44841b7ddd16194f6304203cf7c6b134b7a008a0fc4b3c043b4d1cR87
[2] https://github.com/infrawatch/service-telemetry-operator?tab=readme-ov-file#overriding-default-manifests

[vyzigold]

Otherwise LGTM

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/1f17428efe534a5b87a77ec40cc61ec4

❌ stf-crc-ocp_412-local_build FAILURE in 35m 42s
✔️ stf-crc-ocp_414-local_build SUCCESS in 32m 58s
✔️ stf-crc-ocp_412-local_build-index_deploy SUCCESS in 42m 54s
✔️ stf-crc-ocp_414-local_build-index_deploy SUCCESS in 35m 49s

change commited

[paramite]

Didnt' test this, but it seems lgtm to me.

[vkmc]

LGTM

[vkmc]

recheck