omdb: `db sleds` forbidden

[smklein]

What was observed:

support@oxz_switch:/tmp$ ./omdb db sleds
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:108::3]:32221,[fd00:1122:3344:10c::3]:32221,[fd00:1122:3344:102::3]:32221,[fd00:1122:3344:107::3]:32221,[fd00:1122:3344:109::3]:32221/omicron?sslmode=disable
WARN: found schema version 3.0.3, expected 5.0.0
It's possible the database is running a version that's different from what this
tool understands.  This may result in errors or incorrect output.
Error: listing sleds

Caused by:
    Forbidden

What was expected:

Not seeing "Forbidden".

Why we saw this:

Quoting @askfongjojo

So if any query is using the privileged user, it no longer has the fleet admin role in the customer environment because [one of our deployments] did something like this https://docs.oxide.computer/guides/system/completing-rack-config#_create_local_users
i.e. making a PUT request to directly assign recovery user (or some other admin user they created) the fleet admin role
that's only done for initial setup, not that they would do so as a regular configuration process
The reliance on the built-in users/silos is in conflict with #2305. We probably need to think about which way the ticket should go.

[davepacheco]

Is it possible to get the output of the command with --log-level trace?

[smklein]

omdb-db-sleds-trace.txt