Unable to connect to to Object Bucket exposed using Self Signed Certificate

[Rajpratik71]

How to configure Tempo for Ignoring , Object Bucket having self signed certificate ? means how to disable ssl verification.

Similar to https://github.com/grafana/tempo/blob/a2f70c975850ac4ad31c8d92a11f85bf74a945e4/example/helm/microservices-tempo-values.yaml#L16

I used below cr for creating "TempoSrack" on Openshift and using Object Bucket created using noobaa of ODF.

apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"tempo.grafana.com/v1alpha1","kind":"TempoStack","metadata":{"annotations":{},"name":"gi","namespace":"staging"},"spec":{"resources":{"total":{"limits":{"cpu":"2000m","memory":"2Gi"}}},"storage":{"secret":{"name":"tempo-stack-6cf9749e-f41d-4a17-a3ce-d9445bb2f885","type":"s3"}},"storageSize":"100Gi","template":{"queryFrontend":{"jaegerQuery":{"enabled":true,"ingress":{"route":{"termination":"edge"},"type":"route"}}}}}}
  creationTimestamp: "2024-10-15T07:31:11Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: tempo-operator
    tempo.grafana.com/distribution: openshift
  name: gi
  namespace: staging
  resourceVersion: "17846338"
  uid: d79589a3-a3ff-4e6f-a998-5d0c1e36f0fe
spec:
  hashRing:
    memberlist: {}
  images: {}
  limits:
    global:
      ingestion: {}
      query:
        maxSearchDuration: 0s
  managementState: Managed
  observability:
    grafana:
      instanceSelector: {}
    metrics: {}
    tracing:
      jaeger_agent_endpoint: localhost:6831
  replicationFactor: 1
  resources:
    total:
      limits:
        cpu: "2"
        memory: 2Gi
  retention:
    global:
      traces: 48h0m0s
  search:
    defaultResultLimit: 20
    maxDuration: 0s
  serviceAccount: tempo-gi
  storage:
    secret:
      name: tempo-stack-6cf9749e-f41d-4a17-a3ce-d9445bb2f885
      type: s3
    tls:
      enabled: false
  storageSize: 100Gi
  template:
    compactor:
      replicas: 1
    distributor:
      component:
        replicas: 1
      tls:
        enabled: false
    gateway:
      component:
        replicas: 1
      enabled: false
      ingress:
        route: {}
    ingester:
      replicas: 1
    querier:
      replicas: 1
    queryFrontend:
      component:
        replicas: 1
      jaegerQuery:
        authentication:
          enabled: true
          sar: '{"namespace": "staging", "resource": "pods", "verb": "get"}'
        enabled: true
        ingress:
          route:
            termination: edge
          type: route
        monitorTab:
          enabled: false
          prometheusEndpoint: ""
        servicesQueryDuration: 72h0m0s

Getting similar error on maximum services

pod/tempo-gi-compactor-655b76654b-nl8qq                               0/1     CrashLoopBackOff   4 (88s ago)   2m55s
pod/tempo-gi-distributor-bf8d9c6f8-nvwrp                              1/1     Running            0             2m55s
pod/tempo-gi-ingester-0                                               0/1     CrashLoopBackOff   4 (65s ago)   2m55s
pod/tempo-gi-querier-b8b775b5f-68pt4                                  0/1     CrashLoopBackOff   4 (79s ago)   2m55s
pod/tempo-gi-query-frontend-6bf58b5d68-ksm9q                          2/3     CrashLoopBackOff   4 (73s ago)   2m55s

level=warn ts=2024-10-15T07:31:17.48473324Z caller=main.go:130 msg="-- CONFIGURATION WARNINGS --"
level=warn ts=2024-10-15T07:31:17.484850098Z caller=main.go:136 msg="c.StorageConfig.Trace.Cache is deprecated and will be removed in a future release." explain="Please migrate to the top level cache settings config."
level=info ts=2024-10-15T07:31:17.484877315Z caller=main.go:225 msg="initialising OpenTracing tracer"
level=info ts=2024-10-15T07:31:17.487076797Z caller=main.go:118 msg="Starting Tempo" version="(version=2.5.0, branch=HEAD, revision=46dad341)"
level=info msg="server listening on addresses" http=[::]:3101 grpc=[::]:41539
level=info ts=2024-10-15T07:31:17.488989168Z caller=server.go:240 msg="server listening on addresses" http=[::]:3200 grpc=[::]:9095
level=error ts=2024-10-15T07:31:17.690792421Z caller=main.go:121 msg="error running Tempo" err="failed to init module services: error initialising module: store: failed to create store: unexpected error from ListObjects on tempo-stack-6cf9749e-f41d-4a17-a3ce-d9445bb2f885: Get \"https://s3.staging.svc/tempo-stack-6cf9749e-f41d-4a17-a3ce-d9445bb2f885/?location=\": tls: failed to verify certificate: x509: certificate signed by unknown authority"

[pavolloffay]

cc) @rubenvp8510