admin password has been changed: error on get-admin-password

[PietroPasotti]

Bug Description

$ juju run grafana/0 get-admin-password 
Running operation 5 with 1 task
  - task 6 on unit-grafana-0

Waiting for task 6...
admin-password: Admin password has been changed by an administrator
url: http://10.42.42.42/cos-grafana

To Reproduce

deploy cos-lite --edge

Environment

microk8s

Relevant log output

n/a

Additional context

hard to reproduce.

Known workaround:
juju ssh --container grafana grafana/0 /usr/share/grafana/bin/grafana-cli --homepath "/usr/share/grafana" admin reset-admin-password foobarbaz

[sed-i]

Would be nice to fix it together with #261.

[taurus-forever]

Just for the history.
There was a similar fix: #236
Originally reported here: https://chat.charmhub.io/charmhub/pl/93y7pea4ubgp3ew9ruwa74g5jo

[simskij]

We are unable to reproduce this. If someone can provide a reasonable reproducer, we'd be more than happy to open it up again.

[sed-i]

Encountered this again after

juju deploy --trust --channel=edge cos-lite --overlay ./offers-overlay.yaml --overlay ./tls-overlay.yaml

with

Model        Controller  Cloud/Region        Version  SLA          Timestamp
welcome-k8s  microk8s    microk8s/localhost  3.5.1    unsupported  17:28:54-04:00

App           Version  Status  Scale  Charm                     Channel      Rev  Address         Exposed  Message
alertmanager  0.27.0   active      1  alertmanager-k8s          latest/edge  119  10.152.183.185  no       
ca                     active      1  self-signed-certificates  latest/edge  144  10.152.183.69   no       
catalogue              active      1  catalogue-k8s             latest/edge   43  10.152.183.79   no       
grafana       9.5.3    active      1  grafana-k8s               latest/edge  114  10.152.183.90   no       
loki          2.9.6    active      1  loki-k8s                  latest/edge  151  10.152.183.229  no       
prometheus    2.50.1   active      1  prometheus-k8s            latest/edge  189  10.152.183.83   no       
traefik       v2.11.0  active      1  traefik-k8s               latest/edge  194  10.152.183.153  no       Serving at 10.196.135.67

On 8cpu16gb multipass vm (jammy).

Might be related to: #64

[lucabello]

We should try to reproduce this reliably and consistently first!

[sabaini]

Just as a datapoint, I've now encountered this on a fresh install after rebooting the VM COS-lite was deployed to

Model      Controller  Cloud/Region        Version  SLA          Timestamp
cos-model  microk8s    microk8s/localhost  3.5.3    unsupported  07:17:10Z

App           Version  Status  Scale  Charm             Channel        Rev  Address         Exposed  Message
alertmanager  0.27.0   active      1  alertmanager-k8s  latest/stable  119  10.152.183.29   no       
catalogue              active      1  catalogue-k8s     latest/stable   43  10.152.183.112  no       
grafana       9.5.3    active      1  grafana-k8s       latest/stable  114  10.152.183.168  no       
loki          2.9.6    active      1  loki-k8s          latest/stable  151  10.152.183.93   no       
prometheus    2.52.0   active      1  prometheus-k8s    latest/stable  189  10.152.183.119  no       
traefik       v2.11.0  active      1  traefik-k8s       latest/stable  194  10.152.183.233  no       Serving at 240.22.0.229

Unit             Workload  Agent  Address     Ports  Message
alertmanager/0*  active    idle   10.1.54.92         
catalogue/0*     active    idle   10.1.54.97         
grafana/0*       active    idle   10.1.54.93         
loki/0*          active    idle   10.1.54.98         
prometheus/0*    active    idle   10.1.54.91         
traefik/0*       active    idle   10.1.54.96         Serving at 240.22.0.229

Offer                            Application   Charm             Rev  Connected  Endpoint              Interface                Role
alertmanager-karma-dashboard     alertmanager  alertmanager-k8s  119  0/0        karma-dashboard       karma_dashboard          provider
grafana-dashboards               grafana       grafana-k8s       114  1/1        grafana-dashboard     grafana_dashboard        requirer
loki-logging                     loki          loki-k8s          151  0/0        logging               loki_push_api            provider
prometheus-receive-remote-write  prometheus    prometheus-k8s    189  1/1        receive-remote-write  prometheus_remote_write  provider


snap list|grep microk8s
microk8s  v1.31.0         7114   1.31-strict/stable  canonical**  -

[sabaini]

Fwiw I've seen this in several test runs now. Please let me know if you need any diagnostics/logs