Cortex is in pending state without starting workloads

[alexandreLamarre]

steps to reproduce

• Start with a fresh cluster
• Install via HA preset with S3
• UI shows monitoring backend is installed

expected

UI

• expected UI to show some warning that cortex is in fact not installed (driver status = Pending here)
• UI is not using the correct APIs or correct API ordering, needs further investigating

Backend

• expected cortex to deploy workloads

Not classified

• expected monitoring cluster CRD preset to not be "all", expected replicas to be 3

opni-manager logs

[20:46:48] ERROR monitoring failed to reconcile monitoring cluster {"gateway": "opni", "namespace": "opni", "error": "Gateway.core.opni.io \"\" not found"}
github.com/rancher/opni/controllers.(*CoreMonitoringReconciler).Reconcile
	github.com/rancher/opni/controllers/core_monitoring_controller.go:86
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:314
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226
[20:46:48] ERROR Reconciler error {"controller": "monitoringcluster", "controllerGroup": "core.opni.io", "controllerKind": "MonitoringCluster", "MonitoringCluster": {"name":"opni","namespace":"opni"}, "namespace": "opni", "name": "opni", "reconcileID": "6bd50549-e075-4975-b56c-e912bc095992", "error": "Gateway.core.opni.io \"\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226

monitoring cluster CRD

spec:
  cortex:
   // .... (this is correct)
    cortexWorkloads:
      targets:
        all:
          replicas: 1
    enabled: true
  gateway: {}
  grafana:
    config: {}
    dashboardContentCacheDuration: 0s
    enabled: true
    hostname: // ... (this is correct)

gateway pod ENV

// ...
GATEWAY_NAME=opni-gateway
POD_NAMESPACE=opni

[alexandreLamarre]

Deleting the CRD and attempting to re-install via the UI results in the call:

http://localhost:12080/opni-api/CortexOps/configuration/default

�
�(nats: wrong last sequence: 1: key exists�6
(type.googleapis.com/google.rpc.ErrorInfo�

�CONFLICT

[alexandreLamarre]

there is something weird going on with the gateway ref, after I fixed it manually and got cortex to all-in-one to install, when switching to HA preset mode in the UI, it edited the CRD to have:

gateway:
  name : opni-gateway

but did not include the namespace

[alexandreLamarre]

this also did not switch cortex to HA preset:

The status indicates it thinks it should be all in one and has deleted the HA setup

status:
  cortex:
    version: v1.16.0-opni.8
    workloadStatus:
      alertmanager:
        conditions: StatefulSet has been successfully deleted
        ready: true
      all:
        conditions: All replicas are ready
        ready: true
      compactor:
        conditions: StatefulSet has been successfully deleted
        ready: true
      distributor:
        conditions: Deployment has been successfully deleted
        ready: true
      ingester:
        conditions: StatefulSet has been successfully deleted
        ready: true
      purger:
        conditions: Deployment has been successfully deleted
        ready: true
      querier:
        conditions: StatefulSet has been successfully deleted
        ready: true
      query-frontend:
        conditions: Deployment has been successfully deleted
        ready: true
      ruler:
        conditions: Deployment has been successfully deleted
        ready: true
      store-gateway:
        conditions: StatefulSet has been successfully deleted
        ready: true
    workloadsReady: true
  image: >-
    alex7285/opni@sha256:6180a4e04fe1b310b02c437766fbe20bf3702304e5cebf38a797140647d46435
  imagePullPolicy: IfNotPresent
spec:
  cortex:
    cortexConfig:
      limits:
        compactor_blocks_retention_period: {}
      log_level: debug
      storage:
        backend: s3
        filesystem: {}
        s3:
        // ....
    cortexWorkloads:
      targets:
        all:
          replicas: 1
    enabled: true
  gateway:
    name: opni-gateway
    namespace: opni
  grafana:
    config: {}
    dashboardContentCacheDuration: 0s
    enabled: true

[alexandreLamarre]

Trying to then edit the config in the UI, results in the following bug:

[alexandreLamarre]

When the HA configuration was accepted by the UI, it was not applied to the backend:

{
    "enabled": true,
    "revision": {
        "revision": "56399193"
    },
    "cortexWorkloads": {
        "targets": {
            "all": {
                "replicas": 1
            }
        }
    },
    "cortexConfig": {
        "limits": {
            "compactorBlocksRetentionPeriod": "0s"
        },
        "storage": {
            "backend": "s3",
            "s3": {
                "endpoint": "s3.us-east-1.amazonaws.com",
                "region": "us-east-1",
                "secretAccessKey": "***",
                "accessKeyId": "AKIARHLSZXXGKCKBHQVX",
                "sse": {},
                "http": {}
            },
            "filesystem": {}
        },
        "logLevel": "debug"
    },
    "grafana": {
        "enabled": true,
        "hostname": "//...
    }
}

[alexandreLamarre]

This is also tracks the UI failures and expected behavior for the UI