kube-prometheus-stack - Grafana Errors

[mikev1963]

Describe the bug a clear and concise description of what the bug is.

When I am trying to import a new dashboard I get the following errors:

BAD Gateway

Logs from Grafana:
`logger=context userId=1 orgId=1 uname=admin t=2025-02-01T15:30:22.524729568Z level=error msg="Proxy request failed" err="remote error: tls: handshake failure"
logger=context userId=1 orgId=1 uname=admin t=2025-02-01T15:30:22.524945708Z level=error msg="Request Completed" method=GET path=/api/gnet/dashboards/66 status=502 remote_addr=127.0.0.1 time_ms=50 duration=50.813651ms size=0 referer=http://localhost:3001/dashboard/import handler=/api/gnet/* status_source=downstream

Here is my values.yaml file:
`fullnameOverride: prometheus

defaultRules:
create: true
rules:
alertmanager: true
etcd: true
configReloaders: true
general: true
k8s: true
kubeApiserverAvailability: true
kubeApiserverBurnrate: true
kubeApiserverHistogram: true
kubeApiserverSlos: true
kubelet: true
kubeProxy: true
kubePrometheusGeneral: true
kubePrometheusNodeRecording: true
kubernetesApps: true
kubernetesResources: true
kubernetesStorage: true
kubernetesSystem: true
kubeScheduler: true
kubeStateMetrics: true
network: true
node: true
nodeExporterAlerting: true
nodeExporterRecording: true
prometheus: true
prometheusOperator: true

alertmanager:
fullnameOverride: alertmanager
enabled: true
ingress:
enabled: false

grafana:
enabled: true
fullnameOverride: grafana
forceDeployDatasources: false
forceDeployDashboards: false
defaultDashboardsEnabled: true
defaultDashboardsTimezone: utc
serviceMonitor:
enabled: true

kubeApiServer:
enabled: true

kubelet:
enabled: true
serviceMonitor:
metricRelabelings:
- action: replace
sourceLabels:
- node
targetLabel: instance

kubeControllerManager:
enabled: true
endpoints: # ips of servers
- 192.168.100.210
- 192.168.100.211
- 192.168.100.212

coreDns:
enabled: true

kubeDns:
enabled: false

kubeEtcd:
enabled: true
endpoints: # ips of servers
- 192.168.100.210
- 192.168.100.211
- 192.168.100.212

service:
enabled: true
port: 2381
targetPort: 2381

serviceMonitor:
insecureSkipVerify: true

kubeScheduler:
enabled: true
endpoints: # ips of servers
- 192.168.100.210
- 192.168.100.211
- 192.168.100.212

kubeProxy:
enabled: true
endpoints: # ips of servers
- 192.168.100.210
- 192.168.100.211
- 192.168.100.212

kubeStateMetrics:
enabled: true

kube-state-metrics:
fullnameOverride: kube-state-metrics
selfMonitor:
enabled: true
prometheus:
monitor:
enabled: true
relabelings:
- action: replace
regex: (.*)
replacement: $1
sourceLabels:
- __meta_kubernetes_pod_node_name
targetLabel: kubernetes_node

nodeExporter:
enabled: true
serviceMonitor:
relabelings:
- action: replace
regex: (.*)
replacement: $1
sourceLabels:
- __meta_kubernetes_pod_node_name
targetLabel: kubernetes_node

prometheus-node-exporter:
fullnameOverride: node-exporter
podLabels:
jobLabel: node-exporter
extraArgs:
- --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
- --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
service:
portName: http-metrics
prometheus:
monitor:
enabled: true
relabelings:
- action: replace
regex: (.*)
replacement: $1
sourceLabels:
- __meta_kubernetes_pod_node_name
targetLabel: kubernetes_node
resources:
requests:
memory: 512Mi
cpu: 250m
limits:
memory: 2048Mi

prometheusOperator:
enabled: true
prometheusConfigReloader:
resources:
requests:
cpu: 200m
memory: 50Mi
limits:
memory: 100Mi

prometheus:
enabled: true
prometheusSpec:
replicas: 3
replicaExternalLabelName: "replica"
ruleSelectorNilUsesHelmValues: false
serviceMonitorSelectorNilUsesHelmValues: false
podMonitorSelectorNilUsesHelmValues: false
probeSelectorNilUsesHelmValues: false`

Any help would be appreciated.

I am running a standard K3S cluster

Mike

`

What's your helm version?

version.BuildInfo{Version:"v3.16.4", GitCommit:"7877b45b63f95635153b29a42c0c2f4273ec45ca", GitTreeState:"dirty", GoVersion:"go1.23.4"}

What's your kubectl version?

version.BuildInfo{Version:"v3.16.4", GitCommit:"7877b45b63f95635153b29a42c0c2f4273ec45ca", GitTreeState:"dirty", GoVersion:"go1.23.4"}

Which chart?

kube-prometheus-stack

What's the chart version?

68.4.3

What happened?

BAD Gateway

Logs from Grafana:
`logger=context userId=1 orgId=1 uname=admin t=2025-02-01T15:30:22.524729568Z level=error msg="Proxy request failed" err="remote error: tls: handshake failure"
logger=context userId=1 orgId=1 uname=admin t=2025-02-01T15:30:22.524945708Z level=error msg="Request Completed" method=GET path=/api/gnet/dashboards/66 status=502 remote_addr=127.0.0.1 time_ms=50 duration=50.813651ms size=0 referer=http://localhost:3001/dashboard/import handler=/api/gnet/* status_source=downstream

What you expected to happen?

No response

How to reproduce it?

No response

Enter the changed values of values.yaml?

No response

Enter the command that you execute and failing/misfunctioning.

helm install prometheus-stack prometheus-community/kube-prometheus-stack --namespace monitoring -f values.yaml

Anything else we need to know?

No response