From dfbaa7adf2112a46d00e5a142a0b2007ef04c68f Mon Sep 17 00:00:00 2001 From: Emma Foley Date: Tue, 5 Mar 2024 20:19:54 +0000 Subject: [PATCH] Propose release 1.5.4 (#534) * Fix xrefs for director Operator (#481) Fix the xrefs for the director Operator. The xrefs were referring to the filename instead of the id+assembly value. * Initial pass for external ES (#483) * Initial pass for external ES * Updates for external ES * Notice about deprecated behaviour * Mention how the migration works (automatically) * Adjust comments about observabilityStrategy: none ** Events SGs will now deploy if events are enabled ** Adjusted relevant outputs * Apply suggestions from code review Co-authored-by: Leif Madsen * Minor adjustments from review --------- Co-authored-by: Leif Madsen * Trivial leftover suggestions (#485) * Trivial leftover suggestions * Link ES section to KB article (#486) * Link ES section to KB article * Update doc-Service-Telemetry-Framework/modules/con_primary-parameters-of-the-servicetelemetry-object.adoc --------- Co-authored-by: Leif Madsen * Initial changes to installation for STF 1.5.3 (#484) * Initial changes to installation for STF 1.5.3 Make the initial changes to the installation documentation for STF 1.5.3, which uses observabilityStrategy: use_redhat by default along with preferring to install Observability Operator. Uses the community operators catalogsource for now until OBO is officially available from redhat-operators CatalogSource. Updates the Makefile as well to include Red Hat OpenStack Platform 17.1. Signed-off-by: Leif Madsen * Update install guide for pre-installed Operators Update the installation guide layout for pre-installed Operators that cannot be managed with OLM (due to them being cluster-scoped Operators vs namespace-scoped Operators). Resolves: STF-1485 Signed-off-by: Leif Madsen * Update doc-Service-Telemetry-Framework/modules/proc_deploying-stf-to-the-openshift-environment.adoc Co-authored-by: mickogeary * Update doc-Service-Telemetry-Framework/modules/proc_deploying-stf-to-the-openshift-environment.adoc Co-authored-by: mickogeary * Update doc-Service-Telemetry-Framework/modules/proc_deploying-stf-to-the-openshift-environment.adoc Co-authored-by: mickogeary * Update doc-Service-Telemetry-Framework/modules/proc_deploying-stf-to-the-openshift-environment.adoc Co-authored-by: mickogeary * Adjust wording for cert-manager installation module * Update doc-Service-Telemetry-Framework/modules/proc_deploying-stf-to-the-openshift-environment.adoc Co-authored-by: mickogeary * Reword section that repeats itself --------- Signed-off-by: Leif Madsen Co-authored-by: mickogeary * use_redhat and migration link (#462) * use_redhat and migration link * Update doc-Service-Telemetry-Framework/modules/con_observability-strategy.adoc Co-authored-by: Leif Madsen * Update doc-Service-Telemetry-Framework/modules/con_observability-strategy.adoc Co-authored-by: Leif Madsen * Update doc-Service-Telemetry-Framework/modules/con_observability-strategy.adoc Co-authored-by: Leif Madsen * Apply suggestions from code review Co-authored-by: Leif Madsen * Minor typo fix * Visual tweak * Update doc-Service-Telemetry-Framework/modules/con_observability-strategy.adoc Co-authored-by: Leif Madsen * Update doc-Service-Telemetry-Framework/modules/con_observability-strategy.adoc Co-authored-by: Leif Madsen --------- Co-authored-by: Leif Madsen * Override qdr::router_id defaults in stf-connectors (#487) Update the documentation to provide an override to the FQDN in the qdr::router_id configuration to avoid hostnames longer than 61 chars. Closes rhbz#2208020 * Don't enable event collection by default on OSP (#488) * Don't enable event collection by default on OSP Closes STF-1498 * Remove events configurations and use defaults The defaults for events pipeline and Ceilometer QDR publish events is disabled and does not need to be called out specifically. * No longer import the events dashboard (#490) With a refocus on telemetry by default and without event usage, remove the event dashboards as an event data store is optional and no longer included by default. Related STF-1498 * Installation of cluster monitoring is no longer necessary (#491) Installation of cluster monitoring in CRC (and elsewhere) is no longer necessary for installation of STF. Resolved by #465 * Adjust the default polling interval for collectd (#489) Adjust the collectd polling interval to be 30 seconds instead of 5 seconds. Related STF-1512 Co-authored-by: Victoria Martinez de la Cruz * Remove logs configuration from sample CR (#493) Related STF-1504 * mg_master_RHOSPDOC-1380_chunk-installation-procedure (#492) * mg_master_RHOSPDOC-1380_chunk-installation-procedure * mg_master_RHOSPDOC-1380_chunk-installation-procedure 2nd commoit with further modularisation and chunkage * Commit 3: renaming proc_deploying-stf-to-the-openshift-environment.adoc to con_deploying-stf-to-the-openshift-environment.adoc * Reduce the number of Ceilometer pollsters (#497) Reduce the number of Ceilometer pollsters to only those used by the sample STF dashboards. Closes: rhbz#2239390 * Deprecate the use of high availability mode in STF (#494) * Deprecate the use of high availability mode in STF Resolves STF-1507 * Update doc-Service-Telemetry-Framework/modules/con_high-availability.adoc Co-authored-by: mickogeary --------- Co-authored-by: mickogeary * Fix up the table syntax in Observability Strategy (#495) The existing table was in markdown format which isn't compatible with asciidoc syntax. * Do not manage the event pipeline by default (#498) We do not want events to be sent to QDR by default, as the STF 1.5.3 default configuration will deploy telemetry only Related STF-1498 * Minor clean up and user experience updates (#496) Some minor clean up items and convert some commands to be a bit more user friendly and generic Resolves STF-1533 * Creating an alert does not use curl (#500) The Creating a standard alert route in Alertmanager section no longer uses curl to verify the configuration was loaded, since it uses the prometheus pod and the wget command instance. Removes an extra procedure step that is no longer applicable. * Eliminate duplicate line (#501) * Adding details for QDR password auth (#502) * Adding details for QDR password auth * Move note about disabling auth to main section * Update doc-Service-Telemetry-Framework/modules/proc_retrieving-the-qdr-password.adoc Co-authored-by: Leif Madsen * Update doc-Service-Telemetry-Framework/modules/proc_retrieving-the-qdr-password.adoc Co-authored-by: mickogeary --------- Co-authored-by: Leif Madsen Co-authored-by: mickogeary * Support OCP versions 4.12 through 4.14 (#503) * Support OCP versions 4.12 through 4.14 Update the stf-attributes to cover OCP 4.12 through 4.14 as our default, as OCP 4.10 is EOL. Update the Makefile for building to only cover RHOSP 17.1 and 16.2. * Need html-latest for upstream publish script * Summary: Replace incorrect stf-connectors.yaml filename with enable-stf.yaml (#504) Tracker: https://bugzilla.redhat.com/show_bug.cgi?id=2239871 Branch: master-BZ-2239871 * Clean up the STF install (#505) * Clean up the STF install for OCP 4.12 and later Clean up the STF installation documentation along with a command that will wait for the STO CSV to be ready and then automatically show the dependencies. Also hide contents that are no longer applicable when the supported base version of OCP is greater than 4.10, since OCP 4.10 is now EOL. * Adjust the ifeval to be < 4.12 * Provide the preferred STF object for deployment (#507) Provide the preferred ServiceTelemetry object for deployments rather than asking the administrator to build a configuration. The provided object will result in a metrics-focused deployment without extra configuration options, which will be a foundation for disconnected installations in the future. * Fix various RHOSP links and versions (#508) Fix various links to RHOSP documentation as the paths are different between RHOSP 16.2 and 17.1. Guides were updated but there is no auto-redirect, so we'll need to verify every link that uses defaultURL parameter. This covers the initial ones while working through documentation. Update some older version links and add a new parameter for 17.1 paths specifically. * Update and adjust dashboard procedures (#509) Update and adjust the dashboard installation procedures based on testing. * Add deprecation note for Grafana authentication (#510) Deprecate the basic auth login parameters for Grafana login. Preference is to use the Log in for OpenShift button going forward. Fix syntax issues in asciidoc. * Update deprecated Grafana login warning (#511) Update the Grafana login deprecation warning with wording from the documentation team. * Add updated architecture diagrams (#499) * Add updated architecture diagrams * Use updated architecture diagrams * Update architecture overview to focus on metrics Update the architecture overview to make it clear STF is focused on delivery of metrics from RHOSP. Provide information about use_redhat observability strategy, and note that prior versions of STF would manage Elasticsearch instances. Note that new installations use the observability strategy of use_redhat, and that the guide will focus on that deployment model. Provide a placeholder for a new xref that would guide the user towards the deprecated architecture using events, where our updated metrics-and-events architecture diagram would live. * Link to observability strategy * Remove community components from core overview * Use ObservabilityOperator parameter to refer to OBO/COO * Update doc-Service-Telemetry-Framework/modules/con_stf-architecture.adoc Co-authored-by: mickogeary * Update doc-Service-Telemetry-Framework/modules/con_stf-architecture.adoc Co-authored-by: mickogeary * Update doc-Service-Telemetry-Framework/modules/con_stf-architecture.adoc Co-authored-by: mickogeary * Update doc-Service-Telemetry-Framework/modules/con_stf-architecture.adoc Co-authored-by: mickogeary * Update doc-Service-Telemetry-Framework/modules/con_stf-architecture.adoc * Update doc-Service-Telemetry-Framework/modules/con_stf-architecture.adoc Co-authored-by: mickogeary * Update doc-Service-Telemetry-Framework/modules/con_stf-architecture.adoc Co-authored-by: mickogeary --------- Co-authored-by: mickogeary * Update install guide for dependent operators (#513) * Update install guide for dependent operators Update the installation guide for dependent operators. Adds installation instructions for Cluster Observability Operator and cert-manager for Red Hat OpenShift using the latest channels available for those Operators. The result is that deployment of observabilityStrategy: use_redhat is now possible as the default installation method. Related: STF-1636 * Update doc-Service-Telemetry-Framework/modules/proc_deploying-certificate-manager-for-openshift-operator.adoc Co-authored-by: Chris Sibbitt * Add prerequisites to STF deployment * Adjust wording based on editorial feedback --------- Co-authored-by: Chris Sibbitt * Clean up the prerequisites lists (#514) The prerequisite lists were slightly wrong and have been adjusted for correctness. Minor update of output in the same area to match latest version of STF. * Add removal instructions for COO (#516) * Add removal instructions for COO Add removal instructions for Cluster Observability Operator, pointing at the existing product documentation. Closes: STF-1643 * Update based on editor feedback * Refer to cert-manager removal documentation (#515) * Refer to cert-manager removal documentation Update the STF removal guide to refer to the cert-manager uninstallation procedure which is maintained by that team. Closes: STF-1642 * Adjust cert-manager removal after editor review * Pre-STF 1.5.3 Documentation Walkthrough and Cleanup (#517) * Documentation walk-through and clean up * Update architecture documentation, creating a new section describing the architecture changes in STF 1.5.3 * Update style for knowledge base article references based on editorial feedback * Add links to COO and cert-manager * Update cert-manager install to use oc wait * Multi-Cloud: Add warning about unique domains * HA: Move warning to top * Params: Add warning about HA deprecation * Obs Strat: Add link to migration KBA * Multi-Cloud: Remove reference to Ansible-based deployments * Dashboard: Fix links to collectd plugins Add wrappers to the collectd plugins in the Dashboard guide because paths changed between 16.2 and 17.1. * Update wording for CloudDomain overview Update the wording in the CloudDomain overview since router connections are controlled with router_id parameters now. * Modularize STF architecture changes (#518) * Update diagrams for Cluster Observability Operator (#519) * mg_master_517_minor-style-edits (#521) * mg_master_517_minor-style-edits * Update doc-Service-Telemetry-Framework/modules/con_primary-parameters-of-the-servicetelemetry-object.adoc --------- Co-authored-by: Leif Madsen * Reference 17.1 in docinfo.xml (#522) * PrometheusRules must reference monitoring.rhobs (#523) The PrometheusRules and editing must all reference the new monitoring.rhobs CRD vs the old monitoring.coreos.com CRD which was provided by the community Prometheus Operator (and potentially conflicted with user-workload monitoring, and openshift-monitoring). All references to PrometheusRules now refer to the monitoring.rhobs CRD and any CLI commands are expanded for the full CRD path. * Basic Auth in Grafana no longer supported (#525) * Adjust prometheus query to use token (#520) * Adjust prometheus query to use token * Add section for prometheus token handling * Correction for RBAC changes * Add link to OCP token secret docs * Specifics about UI perms * Update doc-Service-Telemetry-Framework/modules/proc_connecting-an-external-dashboard-system.adoc Co-authored-by: Leif Madsen * Update doc-Service-Telemetry-Framework/modules/proc_connecting-an-external-dashboard-system.adoc Co-authored-by: Leif Madsen * Update doc-Service-Telemetry-Framework/modules/proc_connecting-an-external-dashboard-system.adoc Co-authored-by: Leif Madsen * Update doc-Service-Telemetry-Framework/modules/proc_connecting-an-external-dashboard-system.adoc Co-authored-by: Leif Madsen * Update doc-Service-Telemetry-Framework/modules/proc_accessing-uis-for-stf-components.adoc Co-authored-by: mickogeary * Update doc-Service-Telemetry-Framework/modules/proc_accessing-uis-for-stf-components.adoc Co-authored-by: mickogeary --------- Co-authored-by: Leif Madsen Co-authored-by: mickogeary * Update installation to target Grafana Operator v5 (#526) Update the dashboarding installation procedures to target Grafana Operator v5 by default. Resolves: JIRA#STF-1680 * Add enable dashboard procedure (#527) * Add enable dashboard procedure Update the import dashboards procedure to be enable dashboards procedure now that STF has the ability to manage the dashboards which were formerly imported via URL. Also includes some minor procedure updates in related areas that were referenced in dashboard documentation. Resolves: JIRA#STF-1624 * Update doc-Service-Telemetry-Framework/modules/proc_importing-dashboards.adoc Co-authored-by: mickogeary * Update doc-Service-Telemetry-Framework/modules/proc_importing-dashboards.adoc Co-authored-by: mickogeary * Update doc-Service-Telemetry-Framework/modules/proc_importing-dashboards.adoc Co-authored-by: mickogeary * Update doc-Service-Telemetry-Framework/modules/proc_importing-dashboards.adoc Co-authored-by: mickogeary * Update doc-Service-Telemetry-Framework/modules/proc_importing-dashboards.adoc Co-authored-by: mickogeary * Update doc-Service-Telemetry-Framework/modules/proc_importing-dashboards.adoc Co-authored-by: mickogeary --------- Co-authored-by: mickogeary * Update OCP version support status (#529) Update the version support status to specifically say that STF is supported on OCP EUS releases. While the STF bundles are generated for a range of releases, this is to support the ability of customers to upgrade OCP clusters between EUS releases without needing to remove STF first. Only minor testing is done against standard lifecycle releases of OCP (odd-numbered minor releases). * Update required resource permission reference (#528) Update the required resource permission reference to use the Grafana Operator v5 group. * Drop unused module found in other issue (#533) * mg-master_RHOSPDOC-1200_STF-disconnected (#531) * mg-master_RHOSPDOC-1200_STF-disconnected * added more info about mirror types and verificiation * 3rd commit * another commit from feedback. Added xref and removed openshiftshort as well as a few other changes * another push to fix broken xref * Update doc-Service-Telemetry-Framework/modules/proc_deploying-stf-on-openshift-disconnected-environments.adoc * fix mentions of OCP * edits based on SME feedback * more edits based on SME feedback * Minor syntax clean up * Update doc-Service-Telemetry-Framework/assemblies/assembly_preparing-your-ocp-environment-for-stf.adoc --------- Co-authored-by: Leif Madsen --------- Signed-off-by: Leif Madsen Co-authored-by: Leif Madsen Co-authored-by: Chris Sibbitt Co-authored-by: mickogeary Co-authored-by: Victoria Martinez de la Cruz Co-authored-by: Roger Heslop --- common/global/stf-attributes.adoc | 2 - .../assembly_advanced-features.adoc | 11 +-- ...installing-the-core-components-of-stf.adoc | 7 +- .../assembly_introduction-to-stf.adoc | 3 +- ...reparing-your-ocp-environment-for-stf.adoc | 1 - .../modules/con_dashboards.adoc | 1 - .../modules/con_manifest-overrides.adoc | 2 +- ...tions-for-service-telemetry-framework.adoc | 2 +- .../modules/con_stf-architecture.adoc | 4 +- .../modules/con_support-for-stf.adoc | 2 +- ...proc_accessing-uis-for-stf-components.adoc | 13 ++- ...oc_configuring-observability-strategy.adoc | 2 +- ...nnecting-an-external-dashboard-system.adoc | 77 ++++++++++++++++ ..._creating-an-alert-rule-in-prometheus.adoc | 2 +- ...n-openshift-disconnected-environments.adoc | 86 ++++++++++++++++++ .../modules/proc_deploying-the-overcloud.adoc | 2 +- .../modules/proc_importing-dashboards.adoc | 87 +++++++++---------- .../proc_overriding-a-managed-manifest.adoc | 76 ---------------- ...and-setting-grafana-login-credentials.adoc | 41 --------- ...ting-up-grafana-to-host-the-dashboard.adoc | 42 ++++----- 20 files changed, 248 insertions(+), 215 deletions(-) create mode 100644 doc-Service-Telemetry-Framework/modules/proc_connecting-an-external-dashboard-system.adoc create mode 100644 doc-Service-Telemetry-Framework/modules/proc_deploying-stf-on-openshift-disconnected-environments.adoc delete mode 100644 doc-Service-Telemetry-Framework/modules/proc_overriding-a-managed-manifest.adoc delete mode 100644 doc-Service-Telemetry-Framework/modules/proc_retrieving-and-setting-grafana-login-credentials.adoc diff --git a/common/global/stf-attributes.adoc b/common/global/stf-attributes.adoc index 8627eb74..4060da23 100644 --- a/common/global/stf-attributes.adoc +++ b/common/global/stf-attributes.adoc @@ -39,7 +39,6 @@ endif::[] ifeval::["{build}" == "upstream"] :ObservabilityOperator: Observability{nbsp}Operator :OpenShift: OpenShift -:OpenShiftShort: OKD :OpenStack: OpenStack :OpenStackShort: OSP :OpenStackVersion: Wallaby @@ -58,7 +57,6 @@ endif::[] ifeval::["{build}" == "downstream"] :ObservabilityOperator: Cluster{nbsp}Observability{nbsp}Operator :OpenShift: Red{nbsp}Hat{nbsp}OpenShift{nbsp}Container{nbsp}Platform -:OpenShiftShort: OCP :OpenStack: Red{nbsp}Hat{nbsp}OpenStack{nbsp}Platform :OpenStackShort: RHOSP :OpenStackVersion: 17.1 diff --git a/doc-Service-Telemetry-Framework/assemblies/assembly_advanced-features.adoc b/doc-Service-Telemetry-Framework/assemblies/assembly_advanced-features.adoc index 28916ec4..c7a64342 100644 --- a/doc-Service-Telemetry-Framework/assemblies/assembly_advanced-features.adoc +++ b/doc-Service-Telemetry-Framework/assemblies/assembly_advanced-features.adoc @@ -18,17 +18,11 @@ ifdef::include_when_16[] * xref:container-health-and-api-status_assembly-advanced-features[Monitoring container health and API status] endif::include_when_16[] - //Dashboards include::../modules/con_dashboards.adoc[leveloffset=+1] include::../modules/proc_setting-up-grafana-to-host-the-dashboard.adoc[leveloffset=+2] -ifdef::include_when_16[] -// TODO: either rewrite or drop this procedure. We now provide the preferred downstream RHEL Grafana workload image in the deployment procedure. -//include::../modules/proc_overriding-the-default-grafana-container-image.adoc[leveloffset=+2] include::../modules/proc_importing-dashboards.adoc[leveloffset=+2] -endif::include_when_16[] -include::../modules/proc_retrieving-and-setting-grafana-login-credentials.adoc[leveloffset=+2] - +include::../modules/proc_connecting-an-external-dashboard-system.adoc[leveloffset=+2] //Editing the metrics retention time period include::../modules/con_metrics-retention-time-period.adoc[leveloffset=+1] @@ -69,13 +63,10 @@ include::../modules/con_resource-usage-of-openstack.adoc[leveloffset=+1] include::../modules/proc_disabling-resource-usage-monitoring-of-openstack-services.adoc[leveloffset=+2] //Monitoring container health - include::../modules/con_container-health-and-api-status.adoc[leveloffset=+1] include::../modules/proc_disabling-container-health-and-api-status-monitoring.adoc[leveloffset=+2] endif::include_when_16[] - - //reset the context ifdef::parent-context[:context: {parent-context}] ifndef::parent-context[:!context:] diff --git a/doc-Service-Telemetry-Framework/assemblies/assembly_installing-the-core-components-of-stf.adoc b/doc-Service-Telemetry-Framework/assemblies/assembly_installing-the-core-components-of-stf.adoc index 7b084d58..bbbbd7f9 100644 --- a/doc-Service-Telemetry-Framework/assemblies/assembly_installing-the-core-components-of-stf.adoc +++ b/doc-Service-Telemetry-Framework/assemblies/assembly_installing-the-core-components-of-stf.adoc @@ -21,10 +21,10 @@ ifeval::["{SupportedOpenShiftVersion}" == "{NextSupportedOpenShiftVersion}"] * {OpenShift} version {SupportedOpenShiftVersion} is running. endif::[] ifeval::["{SupportedOpenShiftVersion}" != "{NextSupportedOpenShiftVersion}"] -* An {OpenShift} version inclusive of {SupportedOpenShiftVersion} through {NextSupportedOpenShiftVersion} is running. +* An {OpenShift} Extended Update Support (EUS) release version {SupportedOpenShiftVersion} or {NextSupportedOpenShiftVersion} is running. endif::[] * You have prepared your {OpenShift} environment and ensured that there is persistent storage and enough resources to run the {ProjectShort} components on top of the {OpenShift} environment. For more information about {ProjectShort} performance, see the Red Hat Knowledge Base article https://access.redhat.com/articles/4907241[Service Telemetry Framework Performance and Scaling]. -* Your environment is fully connected. {ProjectShort} does not work in a {OpenShift}-disconnected environments or network proxy environments. +* You have deployed {ProjectShort} in a fully connected or {OpenShift}-disconnected environments. {ProjectShort} is unavailable in network proxy environments. ifeval::["{build}" == "downstream"] [IMPORTANT] @@ -32,7 +32,7 @@ ifeval::["{SupportedOpenShiftVersion}" == "{NextSupportedOpenShiftVersion}"] {ProjectShort} is compatible with {OpenShift} version {SupportedOpenShiftVersion} endif::[] ifeval::["{SupportedOpenShiftVersion}" != "{NextSupportedOpenShiftVersion}"] -{ProjectShort} is compatible with {OpenShift} version {SupportedOpenShiftVersion} through {NextSupportedOpenShiftVersion}. +{ProjectShort} is compatible with {OpenShift} versions {SupportedOpenShiftVersion} and {NextSupportedOpenShiftVersion}. endif::[] endif::[] @@ -42,6 +42,7 @@ endif::[] * For more information about Operator catalogs, see https://docs.openshift.com/container-platform/{NextSupportedOpenShiftVersion}/operators/understanding/olm-rh-catalogs.html[_Red Hat-provided Operator catalogs_]. * For more information about the cert-manager Operator for Red Hat, see https://docs.openshift.com/container-platform/{NextSupportedOpenShiftVersion}/security/cert_manager_operator/index.html[_cert-manager Operator for Red Hat OpenShift overview_]. * For more information about {ObservabilityOperator}, see https://docs.openshift.com/container-platform/{NextSupportedOpenShiftVersion}/monitoring/cluster_observability_operator/cluster-observability-operator-overview.html[_Cluster Observability Operator Overview_]. +* For more information about OpenShift life cycle policy and Extended Update Support (EUS), see https://access.redhat.com/support/policy/updates/openshift[_Red Hat OpenShift Container Platform Life Cycle Policy_]. include::../modules/con_deploying-stf-to-the-openshift-environment.adoc[leveloffset=+1] diff --git a/doc-Service-Telemetry-Framework/assemblies/assembly_introduction-to-stf.adoc b/doc-Service-Telemetry-Framework/assemblies/assembly_introduction-to-stf.adoc index 2178e603..8221cb70 100644 --- a/doc-Service-Telemetry-Framework/assemblies/assembly_introduction-to-stf.adoc +++ b/doc-Service-Telemetry-Framework/assemblies/assembly_introduction-to-stf.adoc @@ -31,7 +31,7 @@ ifeval::["{SupportedOpenShiftVersion}" == "{NextSupportedOpenShiftVersion}"] {ProjectShort} is compatible with {OpenShift} version {SupportedOpenShiftVersion} endif::[] ifeval::["{SupportedOpenShiftVersion}" != "{NextSupportedOpenShiftVersion}"] -{ProjectShort} is compatible with {OpenShift} version {SupportedOpenShiftVersion} through {NextSupportedOpenShiftVersion}. +{ProjectShort} is compatible with {OpenShift} Extended Update Support (EUS) release versions {SupportedOpenShiftVersion} and {NextSupportedOpenShiftVersion}. endif::[] endif::[] @@ -40,6 +40,7 @@ endif::[] * https://access.redhat.com/documentation/en-us/openshift_container_platform/{NextSupportedOpenShiftVersion}/[{OpenShift} product documentation] * https://access.redhat.com/articles/4907241[Service Telemetry Framework Performance and Scaling] * https://docs.openshift.com/container-platform/{NextSupportedOpenShiftVersion}/welcome/index.html#cluster-installer-activities[OpenShift Container Platform {NextSupportedOpenShiftVersion} Documentation] +* https://access.redhat.com/support/policy/updates/openshift[Red Hat OpenShift Container Platform Life Cycle Policy] diff --git a/doc-Service-Telemetry-Framework/assemblies/assembly_preparing-your-ocp-environment-for-stf.adoc b/doc-Service-Telemetry-Framework/assemblies/assembly_preparing-your-ocp-environment-for-stf.adoc index ea2ef706..5b4a3f7e 100644 --- a/doc-Service-Telemetry-Framework/assemblies/assembly_preparing-your-ocp-environment-for-stf.adoc +++ b/doc-Service-Telemetry-Framework/assemblies/assembly_preparing-your-ocp-environment-for-stf.adoc @@ -10,7 +10,6 @@ To prepare your {OpenShift} environment for {Project} ({ProjectShort}), you must * Ensure that you have persistent storage available in your {OpenShift} cluster for a production-grade deployment. For more information, see <>. * Ensure that enough resources are available to run the Operators and the application containers. For more information, see <>. -* Ensure that you have a fully connected network environment. For more information, see xref:con-network-considerations-for-service-telemetry-framework_assembly-preparing-your-ocp-environment-for-stf[]. include::../modules/con_observability-strategy.adoc[leveloffset=+1] include::../modules/con_persistent-volumes.adoc[leveloffset=+1] diff --git a/doc-Service-Telemetry-Framework/modules/con_dashboards.adoc b/doc-Service-Telemetry-Framework/modules/con_dashboards.adoc index 6e9cf0e9..07745dd9 100644 --- a/doc-Service-Telemetry-Framework/modules/con_dashboards.adoc +++ b/doc-Service-Telemetry-Framework/modules/con_dashboards.adoc @@ -7,7 +7,6 @@ Use the third-party application, Grafana, to visualize system-level metrics that For more information about configuring data collectors, see xref:configuring-red-hat-openstack-platform-overcloud-for-stf_assembly-completing-the-stf-configuration[]. ifdef::include_when_16[] -//TODO: can re-work this once we have OSP13 dashboard(s) to show. Can't use container health checks or monitoring in OSP13. You can use dashboards to monitor a cloud: Infrastructure dashboard:: diff --git a/doc-Service-Telemetry-Framework/modules/con_manifest-overrides.adoc b/doc-Service-Telemetry-Framework/modules/con_manifest-overrides.adoc index 1b1159ba..1478ecb2 100644 --- a/doc-Service-Telemetry-Framework/modules/con_manifest-overrides.adoc +++ b/doc-Service-Telemetry-Framework/modules/con_manifest-overrides.adoc @@ -26,7 +26,7 @@ = Customizing the deployment [role="_abstract"] -The Service Telemetry Operator watches for a `ServiceTelemetry` manifest to load into {OpenShift} ({OpenShiftShort}). The Operator then creates other objects in memory, which results in the dependent Operators creating the workloads they are responsible for managing. +The Service Telemetry Operator watches for a `ServiceTelemetry` manifest to load into {OpenShift}. The Operator then creates other objects in memory, which results in the dependent Operators creating the workloads they are responsible for managing. [WARNING] ==== diff --git a/doc-Service-Telemetry-Framework/modules/con_network-considerations-for-service-telemetry-framework.adoc b/doc-Service-Telemetry-Framework/modules/con_network-considerations-for-service-telemetry-framework.adoc index e22f0275..58706cc5 100644 --- a/doc-Service-Telemetry-Framework/modules/con_network-considerations-for-service-telemetry-framework.adoc +++ b/doc-Service-Telemetry-Framework/modules/con_network-considerations-for-service-telemetry-framework.adoc @@ -3,4 +3,4 @@ [id="con-network-considerations-for-service-telemetry-framework_{context}"] = Network considerations for Service Telemetry Framework -You can only deploy {Project} ({ProjectShort}) in a fully connected network environment. You cannot deploy {ProjectShort} in {OpenShift}-disconnected environments or network proxy environments. +You can deploy {Project} ({ProjectShort}) in fully connected network environments or in {OpenShift}-disconnected environments. You cannot deploy {ProjectShort} in network proxy environments. \ No newline at end of file diff --git a/doc-Service-Telemetry-Framework/modules/con_stf-architecture.adoc b/doc-Service-Telemetry-Framework/modules/con_stf-architecture.adoc index c81d282b..b14e8087 100644 --- a/doc-Service-Telemetry-Framework/modules/con_stf-architecture.adoc +++ b/doc-Service-Telemetry-Framework/modules/con_stf-architecture.adoc @@ -87,10 +87,12 @@ ifeval::["{SupportedOpenShiftVersion}" == "{NextSupportedOpenShiftVersion}"] * {OpenShift} {SupportedOpenShiftVersion} endif::[] ifeval::["{SupportedOpenShiftVersion}" != "{NextSupportedOpenShiftVersion}"] -* {OpenShift} {SupportedOpenShiftVersion} through {NextSupportedOpenShiftVersion} +* {OpenShift} Extended Update Support (EUS) releases {SupportedOpenShiftVersion} and {NextSupportedOpenShiftVersion} endif::[] * Infrastructure platform +For more information about the {OpenShift} EUS releases, see link:https://access.redhat.com/support/policy/updates/openshift[Red Hat OpenShift Container Platform Life Cycle Policy]. + [[osp-stf-server-side-monitoring]] .Server-side STF monitoring infrastructure image::363_OpenStack_STF_updates_0923_deployment_prereq.png[Server-side STF monitoring infrastructure] diff --git a/doc-Service-Telemetry-Framework/modules/con_support-for-stf.adoc b/doc-Service-Telemetry-Framework/modules/con_support-for-stf.adoc index f9bc7c19..dff80c81 100644 --- a/doc-Service-Telemetry-Framework/modules/con_support-for-stf.adoc +++ b/doc-Service-Telemetry-Framework/modules/con_support-for-stf.adoc @@ -4,6 +4,6 @@ [role="_abstract"] Red Hat supports the core Operators and workloads, including {MessageBus}, {ObservabilityOperator} (Prometheus, Alertmanager), Service Telemetry Operator, and Smart Gateway Operator. Red Hat does not support the community Operators or workload components, inclusive of Elasticsearch, Grafana, and their Operators. -You can only deploy {ProjectShort} in a fully connected network environment. You cannot deploy {ProjectShort} in {OpenShift}-disconnected environments or network proxy environments. +You can deploy {Project} ({ProjectShort}) in fully connected network environments or in {OpenShift}-disconnected environments. You cannot deploy {ProjectShort} in network proxy environments. For more information about {ProjectShort} life cycle and support status, see the https://access.redhat.com/node/6225361[{Project} Supported Version Matrix]. diff --git a/doc-Service-Telemetry-Framework/modules/proc_accessing-uis-for-stf-components.adoc b/doc-Service-Telemetry-Framework/modules/proc_accessing-uis-for-stf-components.adoc index f75060ea..874f7c04 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_accessing-uis-for-stf-components.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_accessing-uis-for-stf-components.adoc @@ -4,7 +4,18 @@ [role="_abstract"] In {OpenShift}, applications are exposed to the external network through a route. For more information about routes, see https://docs.openshift.com/container-platform/{NextSupportedOpenShiftVersion}/networking/configuring_ingress_cluster_traffic/overview-traffic.html[Configuring ingress cluster traffic]. -In {Project} ({ProjectShort}), HTTPS routes are exposed for each service that has a web-based interface. These routes are protected by {OpenShift} RBAC and any user that has a `ClusterRoleBinding` that enables them to view {OpenShift} Namespaces can log in. For more information about RBAC, see https://docs.openshift.com/container-platform/{NextSupportedOpenShiftVersion}/authentication/using-rbac.html[Using RBAC to define and apply permissions]. +In {Project} ({ProjectShort}), HTTPS routes are exposed for each service that has a web-based interface and protected by {OpenShift} role-based access control (RBAC). + +You need the following permissions to access the corresponding component UI's: + +[source,json,options="nowrap"] +---- +{"namespace":"service-telemetry", "resource":"grafana", "group":"grafana.integreatly.org", "verb":"get"} +{"namespace":"service-telemetry", "resource":"prometheus", "group":"monitoring.rhobs", "verb":"get"} +{"namespace":"service-telemetry", "resource":"alertmanager", "group":"monitoring.rhobs", "verb":"get"} +---- + +For more information about RBAC, see https://docs.openshift.com/container-platform/{NextSupportedOpenShiftVersion}/authentication/using-rbac.html[Using RBAC to define and apply permissions]. .Procedure diff --git a/doc-Service-Telemetry-Framework/modules/proc_configuring-observability-strategy.adoc b/doc-Service-Telemetry-Framework/modules/proc_configuring-observability-strategy.adoc index e406ae5d..fa356b4f 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_configuring-observability-strategy.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_configuring-observability-strategy.adoc @@ -24,7 +24,7 @@ EOF + [source,bash] ---- -$ for o in alertmanager/default prometheus/default elasticsearch/elasticsearch grafana/default; do oc delete $o; done +$ for o in alertmanagers.monitoring.rhobs/default prometheuses.monitoring.rhobs/default elasticsearch/elasticsearch grafana/default-grafana; do oc delete $o; done ---- + . To verify that all workloads are operating correctly, view the pods and the status of each pod: diff --git a/doc-Service-Telemetry-Framework/modules/proc_connecting-an-external-dashboard-system.adoc b/doc-Service-Telemetry-Framework/modules/proc_connecting-an-external-dashboard-system.adoc new file mode 100644 index 00000000..4cc39b7a --- /dev/null +++ b/doc-Service-Telemetry-Framework/modules/proc_connecting-an-external-dashboard-system.adoc @@ -0,0 +1,77 @@ + +[id="connecting-an-external-dashboard-system_{context}"] += Connecting an external dashboard system + +It is possible to configure third-party visualization tools to connect to the {ProjectShort} Prometheus for metrics retrieval. Access is controlled via an OAuth token, and a ServiceAccount is already created that has (only) the required permissions. A new OAuth token can be generated against this account for the external system to use. + +To use the authentication token, the third-party tool must be configured to supply an HTTP Bearer Token Authorization header as described in RFC6750. Consult the documentation of the third-party tool for how to configure this header. For example link:https://grafana.com/docs/grafana/latest/datasources/prometheus/configure-prometheus-data-source/#custom-http-headers[Configure Prometheus - Custom HTTP Headers] in the _Grafana Documentation_. + +.Procedure + +. Log in to {OpenShift}. + +. Change to the `service-telemetry` namespace: ++ +[source,bash] +---- +$ oc project service-telemetry +---- + +. Create a new token secret for the stf-prometheus-reader service account ++ +[source,bash] +---- +$ oc create -f - < +---- ++ +* Replace with the filepath to the mirror registry that you want to use. + +. Locate the generated manifests and apply them to the target {OpenShift} cluster. For more information, see link:https://access.redhat.com/documentation/en-us/openshift_container_platform/{NextSupportedOpenShiftVersion}/html/installing/disconnected-installation-mirroring#oc-mirror-updating-cluster-manifests_installing-mirroring-disconnected[Configuring your cluster to use the resources generated by oc-mirror] in the {OpenShift} _Installing_ guide. ++ +NOTE: The manifests that you generate with `oc-mirror` produce catalogs with the full index name, such as `redhat-operator-index` instead of `redhat-operators` for `CatalogSource`. Ensure that you use the correct index name for the {ProjectShort} subscriptions. For more information, see xref:deploying-stf-to-the-openshift-environment_assembly-installing-the-core-components-of-stf[]. For more information about customizing Operators with oc mirror, see the Red Hat Knowledgebase solution link:https://access.redhat.com/solutions/7016714[How to customize the catalog name and tags of Operators mirrored to the mirror registry using the oc mirror plugin.] + +.Verification + +* Check that the catalog sources are applied. You can return the entries for new catalogs that reference the {ProjectShort} operators and their dependencies: ++ +[source,bash,options="nowrap",role="white-space-pre"] +---- +$ oc get catalogsources +---- + +* You have deployed {ProjectShort} in a disconnected {OpenShift} cluster and therefore cannot access external networks. diff --git a/doc-Service-Telemetry-Framework/modules/proc_deploying-the-overcloud.adoc b/doc-Service-Telemetry-Framework/modules/proc_deploying-the-overcloud.adoc index bbea3297..91bc7fad 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_deploying-the-overcloud.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_deploying-the-overcloud.adoc @@ -17,7 +17,7 @@ $ source ~/stackrc . Add your data collection and {MessageBus} environment files to the stack with your other environment files and deploy the overcloud: + -[source,bash] +[source,bash,options="nowrap"] ---- (undercloud)$ openstack overcloud deploy --templates \ -e [your environment files] \ diff --git a/doc-Service-Telemetry-Framework/modules/proc_importing-dashboards.adoc b/doc-Service-Telemetry-Framework/modules/proc_importing-dashboards.adoc index 86602898..21c4d2d6 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_importing-dashboards.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_importing-dashboards.adoc @@ -1,76 +1,67 @@ [id="importing-dashboards_{context}"] -= Importing dashboards += Enabling dashboards [role="_abstract"] -The Grafana Operator can import and manage dashboards by creating `GrafanaDashboard` objects. You can view example dashboards at https://github.com/infrawatch/dashboards. +The Grafana Operator can import and manage dashboards by creating `GrafanaDashboard` objects. Service Telemetry Operator can enable a set of default dashboards that create the `GrafanaDashboard` objects that load dashboards into the Grafana instance. -.Procedure +Set the value of `graphing.grafana.dashboards.enabled` to `true` to load the following dashboards into Grafana : -. Import the infrastructure dashboard: -+ -[source,bash,options="nowrap"] ----- -$ oc apply -f https://raw.githubusercontent.com/infrawatch/dashboards/master/deploy/stf-1/rhos-dashboard.yaml +* Infrastructure dashboard +* Cloud view dashboard +* Virtual machine view dashboard +* Memcached view dashboard -grafanadashboard.integreatly.org/rhos-dashboard-1 created ----- -. Import the cloud dashboard: -+ -[WARNING] -In the `enable-stf.yaml` file, ensure you set the value of the collectd `virt` plugin parameter `hostname_format` to `name uuid hostname`, otherwise some of the panels on the cloud dashboard display no information. -ifdef::include_before_17[] -For more information about the `virt` plugin, see link:{defaultURL}/operational_measurements/collectd-plugins_assembly[collectd plugins]. -endif::include_before_17[] -ifdef::include_when_17[] -For more information about the `virt` plugin, see link:{defaultURL}/managing_overcloud_observability/collectd-plugins_assembly[collectd plugins]. -endif::include_when_17[] -+ -[source,bash,options="nowrap"] ----- -$ oc apply -f https://raw.githubusercontent.com/infrawatch/dashboards/master/deploy/stf-1/rhos-cloud-dashboard.yaml +You can use the `GrafanaDashboard` object to create and load additional dashboards into Grafana. For more information about managing dashboards with Grafana Operator, see link:https://grafana.github.io/grafana-operator/docs/dashboards/[Dashboards] in the _Grafana Operator project documentation_. -grafanadashboard.integreatly.org/rhos-cloud-dashboard-1 created ----- -. Import the virtual machine dashboard: -+ -[source,bash,options="nowrap"] ----- -$ oc apply -f https://raw.githubusercontent.com/infrawatch/dashboards/master/deploy/stf-1/virtual-machine-view.yaml +.Prerequisites -grafanadashboard.integreatly.org/virtual-machine-view-1 configured ----- -. Import the memcached dashboard: +* You enabled graphing in the `ServiceTelemetry` object. For more information about graphing, see xref:setting-up-grafana-to-host-the-dashboard_assembly-advanced-features[]. + +.Procedure + +. To enable the managed dashboards, create or modify the `ServiceTelemetry` object. Set `graphing.grafana.dashboards.enabled` to `true`: + -[source,bash,options="nowrap"] +[source,bash] ---- -$ oc apply -f https://raw.githubusercontent.com/infrawatch/dashboards/master/deploy/stf-1/memcached-dashboard.yaml +$ oc edit stf default -grafanadashboard.integreatly.org/memcached-dashboard-1 created +apiVersion: infra.watch/v1beta1 +kind: ServiceTelemetry +... +spec: + ... + graphing: + enabled: true + grafana: + dashboards: + enabled: true ---- -. Verify that the dashboards are available: +. Verify that the Grafana dashboards are created. The process of Service Telemetry Operator creating the dashboards might take some time. + -[source,bash] +[source,bash,options="nowrap"] ---- -$ oc get grafanadashboards +$ oc get grafanadashboards.grafana.integreatly.org -NAME AGE -memcached-dashboard-1 7s -rhos-cloud-dashboard-1 23s -rhos-dashboard-1 29s -virtual-machine-view-1 13s +NAME NO MATCHING INSTANCES LAST RESYNC AGE +memcached-dashboard-1 38s 38s +rhos-cloud-dashboard-1 39s 39s +rhos-dashboard-1 39s 39s +virtual-machine-dashboard-1 37s 37s ---- . Retrieve the Grafana route address: + [source,bash,options="nowrap"] ---- -$ oc get route grafana-route -ojsonpath='{.spec.host}' +$ oc get route default-grafana-route -ojsonpath='{.spec.host}' -grafana-route-service-telemetry.apps.infra.watch +default-grafana-route-service-telemetry.apps.infra.watch ---- . In a web browser, navigate to https://__. Replace __ with the value that you retrieved in the previous step. -. To view the dashboard, click *Dashboards* and *Manage*. +. Log in with OpenShift credentials. For more information about logging in, see xref:accessing-uis-for-stf-components_assembly-installing-the-core-components-of-stf[]. + +. To view the dashboard, click *Dashboards* and *Browse*. The managed dashboards are available in the _service-telemetry_ folder. diff --git a/doc-Service-Telemetry-Framework/modules/proc_overriding-a-managed-manifest.adoc b/doc-Service-Telemetry-Framework/modules/proc_overriding-a-managed-manifest.adoc deleted file mode 100644 index 0a7c8ec6..00000000 --- a/doc-Service-Telemetry-Framework/modules/proc_overriding-a-managed-manifest.adoc +++ /dev/null @@ -1,76 +0,0 @@ -// Module included in the following assemblies: -// -// - -// This module can be included from assemblies using the following include statement: -// include::/proc_overriding-a-managed-manifest.adoc[leveloffset=+1] - - -[id="overriding-a-managed-manifest_{context}"] -= Overriding a managed manifest - -[role="_abstract"] -Edit the `ServiceTelemetry` object and provide a parameter and manifest. For a list of available manifest override parameters, see xref:manifest-overrides_assembly-advanced-features[]. The default `ServiceTelemetry` object is `default`. Use `oc get servicetelemetry` to list the available {ProjectShort} deployments. - -[TIP] -The `oc edit` command loads the default system editor. To override the default editor, pass or set the environment variable `EDITOR` to the preferred editor. For example, `EDITOR=nano oc edit servicetelemetry default`. - -.Procedure - -. Log in to {OpenShift}. -. Change to the `service-telemetry` namespace: -+ -[source,bash] ----- -$ oc project service-telemetry ----- - -. Load the `ServiceTelemetry` object into an editor: -+ -[source,bash] ----- -$ oc edit servicetelemetry default ----- - -. To modify the `ServiceTelemetry` object, provide a manifest override parameter and the contents of the manifest to write to {OpenShiftShort} instead of the defaults provided by {ProjectShort}. -+ -[NOTE] -==== -The trailing pipe (`|`) after entering the manifest override parameter indicates that the value provided is multi-line. -==== -+ -[source,yaml,options="nowrap"] ----- -$ oc edit stf default - -apiVersion: infra.watch/v1beta1 -kind: ServiceTelemetry -metadata: - ... -spec: - alertmanagerConfigManifest: | - apiVersion: v1 - kind: Secret - metadata: - name: 'alertmanager-default' - namespace: 'service-telemetry' - type: Opaque - stringData: - alertmanager.yaml: |- - global: - resolve_timeout: 10m - route: - group_by: ['job'] - group_wait: 30s - group_interval: 5m - repeat_interval: 12h - receiver: 'null' - receivers: - - name: 'null' -status: - ... ----- -+ -The `alertmanagerConfigManifest` manifest override parameter is defined in the `spec` of the `ServiceTelemetry` object. - -. Save and close. diff --git a/doc-Service-Telemetry-Framework/modules/proc_retrieving-and-setting-grafana-login-credentials.adoc b/doc-Service-Telemetry-Framework/modules/proc_retrieving-and-setting-grafana-login-credentials.adoc deleted file mode 100644 index b158ee74..00000000 --- a/doc-Service-Telemetry-Framework/modules/proc_retrieving-and-setting-grafana-login-credentials.adoc +++ /dev/null @@ -1,41 +0,0 @@ -[id="proc-retrieving-and-setting-grafana-credentials_{context}"] -= Retrieving and setting Grafana login credentials - -[role="_abstract"] -When Grafana is enabled, you can login using OpenShift authentication, or the default username and password set by the Grafana Operator. - -[WARNING] -==== -Ensure that you log in to access the dashboards with the _Log in with OpenShift_ button, because the default username and password set by the Grafana Operator is deprecated. -==== - -You can override the credentials in the `ServiceTelemetry` object to have {Project} ({ProjectShort}) set the username and password for Grafana instead. - -.Procedure - -. Log in to {OpenShift}. -. Change to the `service-telemetry` namespace: -+ -[source,bash] ----- -$ oc project service-telemetry ----- -. Retrieve the existing username and password from the {ProjectShort} object: -+ -[source,bash] ----- -$ oc get stf default -o jsonpath="{.spec.graphing.grafana['adminUser','adminPassword']}" ----- - -. To modify the default values of the Grafana administrator username and password through the ServiceTelemetry object, use the `graphing.grafana.adminUser` and `graphing.grafana.adminPassword` parameters. -+ -[source,bash] ----- -$ oc edit stf default ----- -. Wait for the grafana pod to restart with the new credentials in place -+ -[source,bash] ----- -$ oc get po -l app=grafana -w ----- diff --git a/doc-Service-Telemetry-Framework/modules/proc_setting-up-grafana-to-host-the-dashboard.adoc b/doc-Service-Telemetry-Framework/modules/proc_setting-up-grafana-to-host-the-dashboard.adoc index d3de9443..5cdde97f 100644 --- a/doc-Service-Telemetry-Framework/modules/proc_setting-up-grafana-to-host-the-dashboard.adoc +++ b/doc-Service-Telemetry-Framework/modules/proc_setting-up-grafana-to-host-the-dashboard.adoc @@ -4,17 +4,9 @@ [role="_abstract"] Grafana is not included in the default {Project} ({ProjectShort}) deployment, so you must deploy the Grafana Operator from community-operators CatalogSource. If you use the Service Telemetry Operator to deploy Grafana, it results in a Grafana instance and the configuration of the default data sources for the local {ProjectShort} deployment. -ifdef::include_16[The dashboards in {ProjectShort} require features that are available only in Grafana version 8.1.0 and later. By default, the Service Telemetry Operator installs a compatible version. For more information about how to override the Grafana container image, see xref:overriding-the-default-grafana-container-image_assembly-advanced-features[].] - .Procedure -. Log in to {OpenShift}. -. Change to the `service-telemetry` namespace: -+ -[source,bash] ----- -$ oc project service-telemetry ----- +. Log in to your {OpenShift} environment where {ProjectShort} is hosted. . Subscribe to the Grafana Operator by using the community-operators CatalogSource: + @@ -31,10 +23,12 @@ $ oc apply -f - <