Grafana datasource failed to due readOnly error

[TheoBrigitte]

Observability operator has many error due to datasource being read only.

{"level":"error","ts":"2025-02-06T11:05:32Z","msg":"Reconciler error","controller":"grafanaorganization","controllerGroup":"observability.giantswarm.io","controllerKind":"GrafanaOrganization","GrafanaOrganization":{"name":"shared"},"namespace":"","name":"shared","reconcileID":"9034359f-b1a8-403b-8e07-9ad1fbdf20f6","error":"[PUT /datasources/{id}][403] updateDataSourceByIdForbidden {\"message\":\"Cannot update read-only data source\"}","errorVerbose":"[PUT /datasources/{id}][403] updateDataSourceByIdForbidden {\"message\":\"Cannot update read-only data source\"}\ngithub.com/giantswarm/observability-operator/pkg/grafana.ConfigureDefaultDatasources\n\t/workspace/pkg/grafana/grafana.go:227\ngithub.com/giantswarm/observability-operator/internal/controller.GrafanaOrganizationReconciler.configureDatasources\n\t/workspace/internal/controller/grafanaorganization_controller.go:260\ngithub.com/giantswarm/observability-operator/internal/controller.GrafanaOrganizationReconciler.reconcileCreate\n\t/workspace/internal/controller/grafanaorganization_controller.go:176\ngithub.com/giantswarm/observability-operator/internal/controller.(*GrafanaOrganizationReconciler).Reconcile\n\t/workspace/internal/controller/grafanaorganization_controller.go:85\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:328\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:249\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700\ngithub.com/giantswarm/observability-operator/internal/controller.GrafanaOrganizationReconciler.configureDatasources\n\t/workspace/internal/controller/grafanaorganization_controller.go:263\ngithub.com/giantswarm/observability-operator/internal/controller.GrafanaOrganizationReconciler.reconcileCreate\n\t/workspace/internal/controller/grafanaorganization_controller.go:176\ngithub.com/giantswarm/observability-operator/internal/controller.(*GrafanaOrganizationReconciler).Reconcile\n\t/workspace/internal/controller/grafanaorganization_controller.go:85\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:328\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:249\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700\ngithub.com/giantswarm/observability-operator/internal/controller.GrafanaOrganizationReconciler.reconcileCreate\n\t/workspace/internal/controller/grafanaorganization_controller.go:186\ngithub.com/giantswarm/observability-operator/internal/controller.(*GrafanaOrganizationReconciler).Reconcile\n\t/workspace/internal/controller/grafanaorganization_controller.go:85\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:328\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:249\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:341\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:249"}

[QuentinBisson]

@TheoBrigitte on which installation did it happen?

[QuentinBisson]

The atlas reconciliation error has been gone for more than 12 hours so we should be good thanks to the recent changes, closing