After COS load failure dashboards disappear from Grafana

[natalytvinova]

Bug Description

Hi team!
I have a ceph-mon:cos-agent grafana-agent relation.
I see nice ceph dashboards with data etc. But if COS goes down for some reason (machine is off or it ran out of RAM and everything restarted). Those dashboards with their info disappear. I need to remove and re-add the relation in order to restore the state. I was able to reproduce it on the orangebox.

Workaround:

juju remove-relation ceph-mon:cos-agent grafana-agent-container

juju add-relation ceph-mon:cos-agent grafana-agent-container

To Reproduce

cos--bundle.txt
openstack-bundle.txt
cos-status.txt
openstack-status.txt

Environment

Openstack Yoga, Ceph Quincy, Juju 3.4.2 and 3.4.4 but also saw this on 3.5

Relevant log output

-

Additional context

No response

[lucabello]

We should try to reproduce this by:

1. If on Multipass, restart the VM; or, kubectl delete both the controller pod and the grafana pod.
2. When everything comes back up, check if relation data is there / if the dashboards are in the Grafana container / that they don't appear in Grafana.

[ca-scribner]

This feels like a case where the data is probably there, but the event sequence we receive in this case differs from what was expected. It would be interesting to cause this issue, then force the charm to loop on its existing relation data and see if everything is repopulated. We should also document here the charm event sequence that happens when this failure is triggered so we know what events the charm should be designed to