From 2f5b866b58818ebd42f531f5311bf1ab9b5f43d1 Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Fri, 30 Aug 2024 13:51:13 +0200 Subject: [PATCH 01/11] docs: Improved metrics troubleshooting --- docs/user/04-metrics.md | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/docs/user/04-metrics.md b/docs/user/04-metrics.md index b8f2fd4df..ad4647f03 100644 --- a/docs/user/04-metrics.md +++ b/docs/user/04-metrics.md @@ -322,6 +322,29 @@ For metrics ingestion to start automatically, simply apply the following annotat | `prometheus.io/path` | `/metrics`, `/custom_metrics` | `/metrics` | Defines the HTTP path where Prometheus can find metrics data. | | `prometheus.io/scheme` | `http`, `https` | If Istio is active, `https` is supported; otherwise, only `http` is available. The default scheme is `http` unless an Istio sidecar is present, denoted by the label `security.istio.io/tlsMode=istio`, in which case `https` becomes the default. | Determines the protocol used for scraping metrics — either HTTPS with mTLS or plain HTTP. | +An example configuration for a `Service` can like like this: +```yaml +apiVersion: v1 +kind: Service +metadata: + annotations: + prometheus.io/port: "8080" + prometheus.io/scrape: "true" + name: sample +spec: + ports: + - name: http-metrics + appProtocol: http + port: 8080 + protocol: TCP + targetPort: 8080 + selector: + app: sample + type: ClusterIP +``` +> [!NOTE] +> When running the pod targetted by a Service with Istio, please ensure that Istio can derive the used [appProtocol](https://kubernetes.io/docs/concepts/services-networking/service/#application-protocol) from the Service port definition. This can be achieved by prefixing the port name with the protocol like in `http-metrics`, or by explicitly defining the `appProtocol` attribute. If Istio cannot derive the protocol, the communication for scraping the metric endpoint cannot be established. + > [!NOTE] > The Metric agent can scrape endpoints even if the workload is a part of the Istio service mesh and accepts mTLS communication. However, there's a constraint: For scraping through HTTPS, Istio must configure the workload using 'STRICT' mTLS mode. Without 'STRICT' mTLS mode, you can set up scraping through HTTP by applying the annotation `prometheus.io/scheme=http`. For related troubleshooting, see [Log entry: Failed to scrape Prometheus endpoint](#log-entry-failed-to-scrape-prometheus-endpoint). @@ -626,13 +649,17 @@ To detect and fix such situations, check the pipeline status and check out [Trou 2023-08-29T09:53:07.123Z warn internal/transaction.go:111 Failed to scrape Prometheus endpoint {"kind": "receiver", "name": "prometheus/app-pods", "data_type": "metrics", "scrape_timestamp": 1693302787120, "target_labels": "{__name__=\"up\", instance=\"10.42.0.18:8080\", job=\"app-pods\"}"} ``` -**Cause**: The workload is not configured to use 'STRICT' mTLS mode. For details, see [Activate Prometheus-based metrics](#4-activate-prometheus-based-metrics). +**Cause 1**: The workload is not configured to use 'STRICT' mTLS mode. For details, see [Activate Prometheus-based metrics](#_4-activate-prometheus-based-metrics). -**Remedy**: You can either set up 'STRICT' mTLS mode or HTTP scraping: +**Remedy 1**: You can either set up 'STRICT' mTLS mode or HTTP scraping: - Configure the workload using “STRICT” mTLS mode (for example, by applying a corresponding PeerAuthentication). - Set up scraping through HTTP by applying the `prometheus.io/scheme=http` annotation. +**Cause 2**: The service enabling the scrape via the prometheus annotations is not revealing the application protocol to use in the port definition. For details, see [Activate Prometheus-based metrics](#_4-activate-prometheus-based-metrics). + +**Remedy 2**: Define the application protocol in the Service port definition by either prefixing the port name with the protocol, like in `http-metrics` or define the `appProtocol`. + ### Gateway Buffer Filling Up **Symptom**: In the MetricPipeline status, the `TelemetryFlowHealthy` condition has status **BufferFillingUp**. From 40c7670ed6642e7912acd20b81c11db4d9d92ef7 Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Fri, 30 Aug 2024 13:53:50 +0200 Subject: [PATCH 02/11] fix --- docs/user/04-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user/04-metrics.md b/docs/user/04-metrics.md index ad4647f03..f75ff7d97 100644 --- a/docs/user/04-metrics.md +++ b/docs/user/04-metrics.md @@ -322,7 +322,7 @@ For metrics ingestion to start automatically, simply apply the following annotat | `prometheus.io/path` | `/metrics`, `/custom_metrics` | `/metrics` | Defines the HTTP path where Prometheus can find metrics data. | | `prometheus.io/scheme` | `http`, `https` | If Istio is active, `https` is supported; otherwise, only `http` is available. The default scheme is `http` unless an Istio sidecar is present, denoted by the label `security.istio.io/tlsMode=istio`, in which case `https` becomes the default. | Determines the protocol used for scraping metrics — either HTTPS with mTLS or plain HTTP. | -An example configuration for a `Service` can like like this: +An example configuration for a `Service` might be: ```yaml apiVersion: v1 kind: Service From 1fb068d23c60ac0e7c58021dbcc7c3fd08c5e7b4 Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Fri, 30 Aug 2024 14:30:25 +0200 Subject: [PATCH 03/11] fix --- docs/user/assets/logs-arch.drawio.svg | 704 +++++++++++++++++++++++++- 1 file changed, 700 insertions(+), 4 deletions(-) diff --git a/docs/user/assets/logs-arch.drawio.svg b/docs/user/assets/logs-arch.drawio.svg index 0c1e902eb..f2e9719ae 100644 --- a/docs/user/assets/logs-arch.drawio.svg +++ b/docs/user/assets/logs-arch.drawio.svg @@ -1,4 +1,700 @@ - - - -
Kyma Cluster
Kyma Cluster
Kyma User
Kyma User
6
6
Kyma-System Namespace
Kyma-System Namespace
User Namespace
User Namespace
Storage and Analysis
Storage and Analysis
External
Environment
External...
Observability System
(like Datadog/Splunk/...)
Observability System...
Observability System
(like Loki/Jaeger/...)
Observability System...
Instrumentation
Instrumentation
Collection
Collection
Telemetry
Manager
Telemetry...
Reference key:
Reference key:
Processes Logs
Processes Logs
Kubernetes
System
Kubernetes...
5
5
Log Agent
(Fluent Bit)
Log Agent...
stdout/stderr
stdout/stderr
2
2
tails
logs
tails...
Log Agent
(Fluent Bit)
Log Agent...
discovers
Pod metadata
discovers...
3
3
4
4
sends logs 
using OTLP
sends logs...
Kubernetes Node
Kubernetes Node
var/log
var/log
1
1
manages
and observes
manages...
Application
Application
 Istio
 Proxy
Istio...
User-Managed Resource
User-Managed Resource
Kyma-Managed Resource
Kyma-Managed Resource
analyzes
logs
analyzes...Text is not SVG - cannot display \ No newline at end of file + + + + + + + + +
+
+
+ + + Kyma Cluster +
+ + +
+
+
+ + + Kyma Cluster + + + + + + + +
+
+
+ + + Kyma User + + +
+
+
+ + + Kyma User + + + + + + + +
+
+
+ + 6 + +
+
+
+ + + 6 + + + + + + + +
+
+
+ + Kyma-System Namespace +
+ +
+
+
+ + + Kyma-System Namespace + + + + + + + + +
+
+
+ + User Namespace +
+ +
+
+
+ + + User Namespace + + + + + + + +
+
+
+ + Storage and Analysis +
+ +
+
+
+ + + Storage and Analysis + + + + + + + +
+
+
+ + + External +
+ Environment +
+ + +
+
+
+ + + External... + + + + + + + +
+
+
+ Observability System +
+ (like Datadog/Splunk/...) +
+
+
+ + + Observability System... + + + + + + + +
+
+
+ Observability System +
+ (like Loki/Jaeger/...) +
+
+
+ + + Observability System... + + + + + + + +
+
+
+ + Instrumentation +
+ +
+
+
+ + + Instrumentation + + + + + + + +
+
+
+ + Collection +
+ +
+
+
+ + + Collection + + + + + + + +
+
+
+ Telemetry +
+ Manager +
+
+
+ + + Telemetry... + + + + + + + +
+
+
+ + + Reference key: + + +
+
+
+ + + Reference key: + + + + + + + +
+
+
+ + Processes Logs + +
+
+
+ + + Processes Logs + + + + + + + +
+
+
+ Kubernetes +
+ System +
+
+
+ + + Kubernetes... + + + + + + + +
+
+
+ + 5 + +
+
+
+ + + 5 + + + + + + + +
+
+
+ Log Agent +
+ (Fluent Bit) +
+
+
+ + + Log Agent... + + + + + + + + +
+
+
+ stdout/stderr +
+
+
+ + + stdout/stderr + + + + + + + +
+
+
+ + 2 + +
+
+
+ + + 2 + + + + + + + + +
+
+
+ + tails + +
+ + logs + +
+
+
+ + + tails... + + + + + + + +
+
+
+ Log Agent +
+ (Fluent Bit) +
+
+
+ + + Log Agent... + + + + + + + + +
+
+
+ + discovers +
+ Pod metadata + +
+
+
+ + + discovers... + + + + + + + +
+
+
+ + 3 + +
+
+
+ + + 3 + + + + + + + +
+
+
+ + 4 + +
+
+
+ + + 4 + + + + + + + + +
+
+
+ sends logs +
+ using Fluents HTTP protocol +
+
+
+ + + sends logs... + + + + + + + + + +
+
+
+ + Kubernetes Node +
+ +
+
+
+ + + Kubernetes Node + + + + + + + +
+
+
+ var/log +
+
+
+ + + var/log + + + + + + + +
+
+
+ + 1 + +
+
+
+ + + 1 + + + + + + + + +
+
+
+ + manages +
+ and observes + +
+
+
+ + + manages... + + + + + + + + +
+
+
+ + Application + +
+
+
+ + + Application + + + + + + + +
+
+
+ + Istio +
+ Proxy + +
+
+
+ + + Istio... + + + + + + + + + + + +
+
+
+ + User-Managed Resource + +
+
+
+ + + User-Managed Resource + + + + + + + +
+
+
+ + Kyma-Managed Resource + +
+
+
+ + + Kyma-Managed Resource + + + + + + + + + + +
+
+
+ analyzes +
+ logs +
+
+
+ + + analyzes... + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file From 6d672d221582b4af458b0c8c987b45c4d7e51b6f Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Mon, 2 Sep 2024 11:38:45 +0200 Subject: [PATCH 04/11] fix --- docs/user/04-metrics.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/user/04-metrics.md b/docs/user/04-metrics.md index f75ff7d97..07b859a43 100644 --- a/docs/user/04-metrics.md +++ b/docs/user/04-metrics.md @@ -322,7 +322,8 @@ For metrics ingestion to start automatically, simply apply the following annotat | `prometheus.io/path` | `/metrics`, `/custom_metrics` | `/metrics` | Defines the HTTP path where Prometheus can find metrics data. | | `prometheus.io/scheme` | `http`, `https` | If Istio is active, `https` is supported; otherwise, only `http` is available. The default scheme is `http` unless an Istio sidecar is present, denoted by the label `security.istio.io/tlsMode=istio`, in which case `https` becomes the default. | Determines the protocol used for scraping metrics — either HTTPS with mTLS or plain HTTP. | -An example configuration for a `Service` might be: + If you're running the Pod targeted by a Service with Istio, Istio must be able to derive the [appProtocol](https://kubernetes.io/docs/concepts/services-networking/service/#application-protocol) from the Service port definition; otherwise the communication for scraping the metric endpoint cannot be established. You must either prefix the port name with the protocol like in `http-metrics`, or explicitly define the `appProtocol` attribute. + For example, see the following `Service` configuration: ```yaml apiVersion: v1 kind: Service @@ -342,8 +343,6 @@ spec: app: sample type: ClusterIP ``` -> [!NOTE] -> When running the pod targetted by a Service with Istio, please ensure that Istio can derive the used [appProtocol](https://kubernetes.io/docs/concepts/services-networking/service/#application-protocol) from the Service port definition. This can be achieved by prefixing the port name with the protocol like in `http-metrics`, or by explicitly defining the `appProtocol` attribute. If Istio cannot derive the protocol, the communication for scraping the metric endpoint cannot be established. > [!NOTE] > The Metric agent can scrape endpoints even if the workload is a part of the Istio service mesh and accepts mTLS communication. However, there's a constraint: For scraping through HTTPS, Istio must configure the workload using 'STRICT' mTLS mode. Without 'STRICT' mTLS mode, you can set up scraping through HTTP by applying the annotation `prometheus.io/scheme=http`. For related troubleshooting, see [Log entry: Failed to scrape Prometheus endpoint](#log-entry-failed-to-scrape-prometheus-endpoint). From 0037f40db2ca83b86dabbe3009fde604b9fb38b5 Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Mon, 2 Sep 2024 11:39:52 +0200 Subject: [PATCH 05/11] Apply suggestions from code review Co-authored-by: Nina Hingerl <76950046+NHingerl@users.noreply.github.com> --- docs/user/04-metrics.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user/04-metrics.md b/docs/user/04-metrics.md index 07b859a43..c3dd2e52c 100644 --- a/docs/user/04-metrics.md +++ b/docs/user/04-metrics.md @@ -655,9 +655,9 @@ To detect and fix such situations, check the pipeline status and check out [Trou - Configure the workload using “STRICT” mTLS mode (for example, by applying a corresponding PeerAuthentication). - Set up scraping through HTTP by applying the `prometheus.io/scheme=http` annotation. -**Cause 2**: The service enabling the scrape via the prometheus annotations is not revealing the application protocol to use in the port definition. For details, see [Activate Prometheus-based metrics](#_4-activate-prometheus-based-metrics). +**Cause 2**: The Service definition enabling the scrape with Prometheus annotations does not reveal the application protocol to use in the port definition. For details, see [Activate Prometheus-based metrics](#_4-activate-prometheus-based-metrics). -**Remedy 2**: Define the application protocol in the Service port definition by either prefixing the port name with the protocol, like in `http-metrics` or define the `appProtocol`. +**Remedy 2**: Define the application protocol in the Service port definition by either prefixing the port name with the protocol, like in `http-metrics` or define the `appProtocol` attribute. ### Gateway Buffer Filling Up From c1d7caf40f3d7b1d22b6b9391be975faf64ac804 Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Mon, 2 Sep 2024 11:42:40 +0200 Subject: [PATCH 06/11] fix --- docs/user/assets/logs-arch.drawio.svg | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/user/assets/logs-arch.drawio.svg b/docs/user/assets/logs-arch.drawio.svg index f2e9719ae..2f0e9307a 100644 --- a/docs/user/assets/logs-arch.drawio.svg +++ b/docs/user/assets/logs-arch.drawio.svg @@ -1,4 +1,4 @@ - + @@ -488,17 +488,19 @@ -
+
sends logs
- using Fluents HTTP protocol + using Fluents protocol +
+ based on HTTP
- + sends logs... From 8b1266f7b75f85e6f9c36e17587113f754100eef Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Mon, 2 Sep 2024 11:43:12 +0200 Subject: [PATCH 07/11] fix --- docs/user/assets/logs-arch.drawio.svg | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user/assets/logs-arch.drawio.svg b/docs/user/assets/logs-arch.drawio.svg index 2f0e9307a..4ff00a496 100644 --- a/docs/user/assets/logs-arch.drawio.svg +++ b/docs/user/assets/logs-arch.drawio.svg @@ -1,4 +1,4 @@ - + @@ -493,7 +493,7 @@
sends logs
- using Fluents protocol + using Fluent Bits protocol
based on HTTP
From 1b4743956319c7b220e9ee9eda9429b1c456c41d Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Mon, 2 Sep 2024 16:30:59 +0200 Subject: [PATCH 08/11] fix --- docs/user/assets/logs-arch.drawio.svg | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/docs/user/assets/logs-arch.drawio.svg b/docs/user/assets/logs-arch.drawio.svg index 4ff00a496..aebe89ca3 100644 --- a/docs/user/assets/logs-arch.drawio.svg +++ b/docs/user/assets/logs-arch.drawio.svg @@ -1,4 +1,4 @@ - + @@ -488,19 +488,17 @@ -
+
sends logs
- using Fluent Bits protocol -
- based on HTTP + using HTTP
- + sends logs... From b0bea1ad84712028d4a245294045650d57c7590a Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Wed, 4 Sep 2024 14:38:14 +0200 Subject: [PATCH 09/11] Apply suggestions from code review Co-authored-by: Nina Hingerl <76950046+NHingerl@users.noreply.github.com> --- docs/user/04-metrics.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user/04-metrics.md b/docs/user/04-metrics.md index c3dd2e52c..b6bf2df4f 100644 --- a/docs/user/04-metrics.md +++ b/docs/user/04-metrics.md @@ -648,14 +648,14 @@ To detect and fix such situations, check the pipeline status and check out [Trou 2023-08-29T09:53:07.123Z warn internal/transaction.go:111 Failed to scrape Prometheus endpoint {"kind": "receiver", "name": "prometheus/app-pods", "data_type": "metrics", "scrape_timestamp": 1693302787120, "target_labels": "{__name__=\"up\", instance=\"10.42.0.18:8080\", job=\"app-pods\"}"} ``` -**Cause 1**: The workload is not configured to use 'STRICT' mTLS mode. For details, see [Activate Prometheus-based metrics](#_4-activate-prometheus-based-metrics). +**Cause 1**: The workload is not configured to use 'STRICT' mTLS mode. For details, see [Activate Prometheus-based metrics](#4-activate-prometheus-based-metrics). **Remedy 1**: You can either set up 'STRICT' mTLS mode or HTTP scraping: - Configure the workload using “STRICT” mTLS mode (for example, by applying a corresponding PeerAuthentication). - Set up scraping through HTTP by applying the `prometheus.io/scheme=http` annotation. -**Cause 2**: The Service definition enabling the scrape with Prometheus annotations does not reveal the application protocol to use in the port definition. For details, see [Activate Prometheus-based metrics](#_4-activate-prometheus-based-metrics). +**Cause 2**: The Service definition enabling the scrape with Prometheus annotations does not reveal the application protocol to use in the port definition. For details, see [Activate Prometheus-based metrics](#4-activate-prometheus-based-metrics). **Remedy 2**: Define the application protocol in the Service port definition by either prefixing the port name with the protocol, like in `http-metrics` or define the `appProtocol` attribute. From 6b60dd96e2d0302f102e08f6db1fcb228c619308 Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Wed, 4 Sep 2024 16:14:05 +0200 Subject: [PATCH 10/11] fix --- docs/user/04-metrics.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/user/04-metrics.md b/docs/user/04-metrics.md index b6bf2df4f..f43ad380a 100644 --- a/docs/user/04-metrics.md +++ b/docs/user/04-metrics.md @@ -289,7 +289,7 @@ Telemetry Manager continuously watches the Secret referenced with the **secretKe > [!TIP] > If you use a Secret owned by the [SAP BTP Service Operator](https://github.com/SAP/sap-btp-service-operator), you can configure an automated rotation using a `credentialsRotationPolicy` with a specific `rotationFrequency` and don’t have to intervene manually. -### 4. Activate Prometheus-Based Metrics +### 4. Activate Prometheus-Based Metric > [!NOTE] > For the following approach, you must have instrumented your application using a library like the [Prometheus client library](https://prometheus.io/docs/instrumenting/clientlibs/), with a port in your workload exposed serving as a Prometheus metrics endpoint. @@ -648,14 +648,14 @@ To detect and fix such situations, check the pipeline status and check out [Trou 2023-08-29T09:53:07.123Z warn internal/transaction.go:111 Failed to scrape Prometheus endpoint {"kind": "receiver", "name": "prometheus/app-pods", "data_type": "metrics", "scrape_timestamp": 1693302787120, "target_labels": "{__name__=\"up\", instance=\"10.42.0.18:8080\", job=\"app-pods\"}"} ``` -**Cause 1**: The workload is not configured to use 'STRICT' mTLS mode. For details, see [Activate Prometheus-based metrics](#4-activate-prometheus-based-metrics). +**Cause 1**: The workload is not configured to use 'STRICT' mTLS mode. For details, see [Activate Prometheus-based metrics](#_4-activate-prometheus-based-metrics). **Remedy 1**: You can either set up 'STRICT' mTLS mode or HTTP scraping: - Configure the workload using “STRICT” mTLS mode (for example, by applying a corresponding PeerAuthentication). - Set up scraping through HTTP by applying the `prometheus.io/scheme=http` annotation. -**Cause 2**: The Service definition enabling the scrape with Prometheus annotations does not reveal the application protocol to use in the port definition. For details, see [Activate Prometheus-based metrics](#4-activate-prometheus-based-metrics). +**Cause 2**: The Service definition enabling the scrape with Prometheus annotations does not reveal the application protocol to use in the port definition. For details, see [Activate Prometheus-based metrics](#_4-activate-prometheus-based-metrics). **Remedy 2**: Define the application protocol in the Service port definition by either prefixing the port name with the protocol, like in `http-metrics` or define the `appProtocol` attribute. From 245b90ffdb1c00b191848aa1a83e0b96d1407995 Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Thu, 5 Sep 2024 13:08:43 +0200 Subject: [PATCH 11/11] Apply suggestions from code review Co-authored-by: Nina Hingerl <76950046+NHingerl@users.noreply.github.com> --- docs/user/04-metrics.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user/04-metrics.md b/docs/user/04-metrics.md index f43ad380a..b0e3124d9 100644 --- a/docs/user/04-metrics.md +++ b/docs/user/04-metrics.md @@ -647,14 +647,14 @@ To detect and fix such situations, check the pipeline status and check out [Trou ```bash 2023-08-29T09:53:07.123Z warn internal/transaction.go:111 Failed to scrape Prometheus endpoint {"kind": "receiver", "name": "prometheus/app-pods", "data_type": "metrics", "scrape_timestamp": 1693302787120, "target_labels": "{__name__=\"up\", instance=\"10.42.0.18:8080\", job=\"app-pods\"}"} ``` - + **Cause 1**: The workload is not configured to use 'STRICT' mTLS mode. For details, see [Activate Prometheus-based metrics](#_4-activate-prometheus-based-metrics). **Remedy 1**: You can either set up 'STRICT' mTLS mode or HTTP scraping: - Configure the workload using “STRICT” mTLS mode (for example, by applying a corresponding PeerAuthentication). - Set up scraping through HTTP by applying the `prometheus.io/scheme=http` annotation. - + **Cause 2**: The Service definition enabling the scrape with Prometheus annotations does not reveal the application protocol to use in the port definition. For details, see [Activate Prometheus-based metrics](#_4-activate-prometheus-based-metrics). **Remedy 2**: Define the application protocol in the Service port definition by either prefixing the port name with the protocol, like in `http-metrics` or define the `appProtocol` attribute.