Adds enhancement for FIPS with multiple RHEL installer versions

In order for an OpenShift cluster to be considered FIPS compliant the installer
must be run on a system with FIPS compliant openssl libraries. This means using
a dynamically linked `openshift-install` binary against the openssl libraries
present on our container image. Today this is not a problem because all
`openshift-install` binaries in use have been expecting to link to RHEL 8 based
openssl libraries, but now OpenShift 4.16 will ship and installer that requires
RHEL 9 libraries.

This will require assisted-service to maintain a way to run the
`openshift-install` binary in a compatible environment for multiple openssl
versions.

Resolves https://issues.redhat.com/browse/MGMT-17771

docs/enhancements/multi-rhel-fips.md

@@ -0,0 +1,137 @@
+---
+title: fips-with-multiple-rhel-versions
+authors:
+  - "@carbonin"
+creation-date: 2024-05-07
+last-updated: 2024-05-08
+---
+
+# Support FIPS for installers built for different RHEL releases
+
+## Summary
+
+In order for an OpenShift cluster to be considered FIPS compliant the installer
+must be run on a system with FIPS compliant openssl libraries. This means using
+a dynamically linked `openshift-install` binary against the openssl libraries
+present on our container image. Today this is not a problem because all
+`openshift-install` binaries in use have been expecting to link to RHEL 8 based
+openssl libraries, but now OpenShift 4.16 will ship and installer that requires
+RHEL 9 libraries.
+
+This will require assisted-service to maintain a way to run the
+`openshift-install` binary in a compatible environment for multiple openssl
+versions.
+
+## Motivation
+
+FIPS compliance is important for our customers and assisted-service should be
+able to install FIPS compliant clusters.
+
+### Goals
+
+- Allow for a single installation of assisted-service to install FIPS-compliant
+  clusters using `openshift-install` binaries built against RHEL 8 or RHEL 9
+
+- Allow for FIPS compliant clusters to be installed from the SaaS offering or
+  the on-prem offering
+
+### Non-Goals
+
+- Changing cluster install interfaces to accommodate new FIPS requirements
+  should be avoided
+
+- Dynamically determining a given release's RHEL version. The RHEL version will
+  be a part of configuration tied to the minor release
+
+## Proposal
+
+Two additional containers will run alongside assisted-service in the same pod.
+These "installer-runner" containers will, when triggered by assisted-service,
+run a specified installer binary to generate the manifests required for a
+particular install. These manifests will then be uploaded to whatever storage
+is in use for this deployment (local for on-prem, or s3 for SaaS) and
+assisted-service will take over as usual from there.
+
+Each of these containers should maintain its own cache of installers as
+assisted-service does now and should be built with the required packages to run
+the installer in FIPS compliance mode.
+
+### User Stories
+
+#### Story 1
+
+As a Cluster Creator, I want to install FIPS compliant clusters for any supported OpenShift version
+
+### Implementation Details/Notes/Constraints
+
+#### New Images
+
+Two new container images will need to be built, and published for every release
+footprint we support. These images will be created based on existing
+assisted-service code, but could be split into their own independent projects
+later.
+
+### Risks and Mitigations
+
+Shipping a new image is a non-trivial process. This may take more time than we
+have to set up. We could likely get away with using the existing
+assisted-service image with a different entrypoint for one of the runner images,
+but that still requires us to publish a new one for the architecture
+assisted-service will not be using.
+
+## Design Details [optional]
+
+The new runner containers will expose a HTTP server listening only for local
+connections. assisted-service will POST to this server when it needs manifests
+generated. The runner container will respond with any error that occurred while
+generating the manifests.
+
+### Open Questions
+
+1. What does the API look like for the runner containers? What data should be
+   passed in an API call and what should be configured in the container
+   environment?
+
+### UI Impact
+
+No need to go into great detail about the UI changes that will need to be made.
+However, this is an excellent time to mention 1) if UI changes are required if
+this enhancement were accepted and 2) at a high-level what those UI changes
+would be.
+
+### Test Plan
+
+FIPS and regular installs should be tested for all supported OpenShift versions.
+Since this should be mostly transparent to the user, regression testing in
+addition to testing 4.16 with and without FIPS should be sufficient.
+
+## Drawbacks
+
+This is a complicated change in architecture something simpler might be more
+desirable.
+
+Creating two additional containers in a pod makes the assisted service more
+expensive to scale.
+
+Creating, maintaining, and releasing additional images is a non-trivial amount
+of additional work.
+
+## Alternatives
+
+Hive is investigating using the container image from the release payload to run
+the installer. This seems possible, but doesn't make much sense if we want to
+run a persistent service rather than a job per cluster. Running a job per
+cluster wouldn't scale particularly well and would also be an even larger
+architectural change.
+
+Clusters that have installers that already match the assisted service container's
+architecture could be handled by the assisted-service container as we do today.
+This would require one less image and container per pod, but having the same
+process for every cluster install would be easier to understand and maintain.
+
+[Go's RPC](https://pkg.go.dev/net/[email protected]) could be used instead of a direct
+HTTP server (RPC can be hosted over HTTP, but that's not what is being addressed
+here). RPC would make this a simpler change as the code for generating the
+manifests is already contained in a single package, but RPC would be a strange
+choice if we were to move the handling into a truly separate service in the
+future.