ops.pebble.APIError: user: unknown user discourse

[cjohnston1158]

Bug Description

Seeing this error when deploying discouse-k8s

To Reproduce

juju add-model postgresql
juju deploy postgresql-k8s --channel 14/edge --trust -n 3
juju deploy discourse-k8s --channel latest/edge
juju deploy redis-k8s
juju deploy nginx-ingress-integrator --trust
juju relate redis-k8s discourse-k8s
juju relate discourse-k8s postgresql-k8s
juju relate discourse-k8s:nginx-route nginx-ingress-integrator

Environment

Ubuntu 22.04
Juju 3.1.5
Microk8s 1.28-strict

Relevant log output

unit-discourse-k8s-0: 23:56:01 ERROR unit.discourse-k8s/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-discourse-k8s-0/charm/venv/ops/pebble.py", line 1599, in _request_raw
    response = self.opener.open(request, timeout=self.timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./src/charm.py", line 635, in <module>
    main(DiscourseCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-discourse-k8s-0/charm/venv/ops/main.py", line 439, in main
    framework.reemit()
  File "/var/lib/juju/agents/unit-discourse-k8s-0/charm/venv/ops/framework.py", line 851, in reemit
    self._reemit()
  File "/var/lib/juju/agents/unit-discourse-k8s-0/charm/venv/ops/framework.py", line 930, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 461, in _set_up_discourse
    migration_process: ExecProcess = container.exec(
  File "/var/lib/juju/agents/unit-discourse-k8s-0/charm/venv/ops/model.py", line 2525, in exec
    return self._pebble.exec(
  File "/var/lib/juju/agents/unit-discourse-k8s-0/charm/venv/ops/pebble.py", line 2374, in exec
    resp = self._request('POST', '/v1/exec', body=body)
  File "/var/lib/juju/agents/unit-discourse-k8s-0/charm/venv/ops/pebble.py", line 1564, in _request
    response = self._request_raw(method, path, query, headers, data)
  File "/var/lib/juju/agents/unit-discourse-k8s-0/charm/venv/ops/pebble.py", line 1610, in _request_raw
    raise APIError(body, code, status, message)
ops.pebble.APIError: user: unknown user discourse

Additional context

No response

[nrobinaubertin]

I couldn't repro...
It's a bit strange because in the trace, it says that it comes from here: "File "./src/charm.py", line 461, in _set_up_discourse".
https://github.com/canonical/discourse-k8s-operator/blob/main/src/charm.py#L461
That's a command executed in the container under the user _daemon_ and not discourse like we see in the error: "user: unknown user discourse"

We changed the user in rev40.
Is it possible that you've encountered this as part of a terraform apply ?
There's a bug with the terraform provider: it doesn't refresh the resources of a charm automatically. So you could see this with the new charm on an old container.

A way to check this is to get the owner of the /srv/discourse directory in the discourse charm container. It should be _daemon_.

[mthaddon]

@cjohnston1158 can you confirm the charm revisions you were/are using? I've just tried this and am able to deploy everything fine locally. The output of juju status is as follows:

$ juju status
Model   Controller          Cloud/Region        Version  SLA          Timestamp
d-test  microk8s-localhost  microk8s/localhost  3.1.5    unsupported  11:06:35+02:00

App                       Version               Status  Scale  Charm                     Channel      Rev  Address         Exposed  Message
discourse-k8s             2.8.14                active      1  discourse-k8s             latest/edge   41  10.152.183.88   no       
nginx-ingress-integrator  25.3.0                active      1  nginx-ingress-integrator  stable        81  10.152.183.162  no       Ingress IP(s): 127.0.0.1, Service IP(s): 10.152.183.163
postgresql-k8s            14.9                  active      3  postgresql-k8s            14/edge      138  10.152.183.122  no       
redis-k8s                 ubuntu/redis@691f315  active      1  redis-k8s                 stable         7                  no       

Unit                         Workload  Agent  Address       Ports     Message
discourse-k8s/0*             active    idle   10.1.129.146            
nginx-ingress-integrator/0*  active    idle   10.1.129.147            Ingress IP(s): 127.0.0.1, Service IP(s): 10.152.183.163
postgresql-k8s/0*            active    idle   10.1.129.143            Primary
postgresql-k8s/1             active    idle   10.1.129.145            
postgresql-k8s/2             active    idle   10.1.129.144            
redis-k8s/0*                 active    idle   10.1.129.149  6379/TCP