Auth: Fix missing snapshots and backups from storage pool used-by URLs #14324

markylaing · 2024-10-22T12:56:29Z

The underlying cause of this bug was that general filtering of used-by URLs makes the assumption that the can_view entitlement is available for all entity types. It is a fair assumption, but wasn't true for storage volume or instance backups or snapshots.

To fix this, four new entity types have been added to the authorization model:

instance_backup
instance_snapshot
storage_volume_backup
storage_volume_snapshot

Each has associated entitlements:

can_edit
can_view
can_delete

It is still not possible to grant these entitlements via the API. Instead, they are granted via can_manage_snapshots or can_manage_backups on the associated instance or storage volume.

The OpenFGADatastore implementation has been updated to handle instance and storage_volume relations between the parent and it's snapshots/backups.

Update OpenFGADatastore comments - it says an instance is not a relation but after this PR it is.

Closes #14291

github-actions · 2024-10-22T12:56:44Z

Heads up @mionaalex - the "Documentation" label was applied to this issue.

markylaing · 2024-10-22T12:56:54Z

CC @mas-who @edlerd

tomponline · 2024-10-22T13:30:48Z

tests are sad

lxd/db/openfga/openfga.go

markylaing · 2024-10-23T07:18:23Z

@tomponline tests are mostly green except for one: https://github.com/canonical/lxd/actions/runs/11463430112/job/31913802942#step:12:52804

I'm not certain why this is failing as it doesn't seem to have anything to do with this PR. It is potentially related to #14315 since lxc profile assign calls PUT /1.0/instances/{name} which does some Profile.ToAPI work. I also don't understand why it only failed with the dir storage backend.

Edit: Note that this also doesn't fail locally. I'll have to get a tmate session running.

tomponline · 2024-10-23T07:36:04Z

I'm not certain why this is failing as it doesn't seem to have anything to do with this PR. It is potentially related to #14315 since lxc profile assign calls PUT /1.0/instances/{name} which does some Profile.ToAPI work. I also don't understand why it only failed with the dir storage backend.

@hamistao please can you check this out, thanks

Seems like a panic.

markylaing · 2024-10-24T06:32:08Z

@tomponline @hamistao The CI passed on the third attempt. I'll investigate a bit more though as I don't want to introduce any races, especially when they may be causing a panic.

markylaing · 2024-10-24T08:12:37Z

@tomponline @hamistao The CI passed on the third attempt. I'll investigate a bit more though as I don't want to introduce any races, especially when they may be causing a panic.

I've been investigating this for an hour or so with no progress. It would be very useful to surface panics in the test logs. I'm trying to figure out a way to do this.

tomponline · 2024-10-24T08:24:55Z

@tomponline @hamistao The CI passed on the third attempt. I'll investigate a bit more though as I don't want to introduce any races, especially when they may be causing a panic.

I've been investigating this for an hour or so with no progress. It would be very useful to surface panics in the test logs. I'm trying to figure out a way to do this.

Did you identify which commit introduced it yet?

Did you try reverting the earlier profiles PR?

markylaing · 2024-10-24T11:46:19Z

Did you identify which commit introduced it yet?

Did you try reverting the earlier profiles PR?

With it being intermittent I didn't think reverting the profiles PR would tell me very much (e.g. I'll need to figure out where that panic is occurring in either case). I've added a commit to check LXD logs for panics. It's failing on standalone tests but not in the cluster tests which is a bit odd. Still investigating.

markylaing · 2024-10-25T10:28:39Z

I've re-run the test 8 times now and the panic only occurred on the first two runs. I've added a PR to handle panics a bit more cleanly in the future (#14346). If it happens again it should be obvious where it occurred.

markylaing · 2024-10-25T12:25:57Z

Of course it fails again as soon as I move the panic checker work into another PR 🤦

Signed-off-by: Mark Laing <[email protected]>

The auth.ValidateEntitlement function validates all entitlements that can be granted via the API. Since the new entitlements on snapshots and backups cannot be granted via the API, this check fails. The OpenFGA server will return an error if an invalid query is performed based on it's own understanding of the authorization model. Signed-off-by: Mark Laing <[email protected]>

Signed-off-by: Mark Laing <[email protected]>

Previously the only entities that had inherited relations were project and server. Now that we are linking instances and storage volumes to their snapshots and backups, the OpenFGADatastore implementation needs to handle these relations. On Read, we can connect a snapshot or backup to its parent instance or storage volume using the information stored in its URL. For example, the storage volume backup URL: /1.0/storage-pools/default/volumes/custom/vol1/backups/backup1?project=project1 is related to its parent: /1.0/storage-pools/default/volumes/custom/vol1?project=project1 via the `storage_volume relation`. Signed-off-by: Mark Laing <[email protected]>

…tartingWithUser. Previously the only entities that had inherited relations were project and server. Now that we are linking instances and storage volumes to their snapshots and backups, the OpenFGADatastore implementation needs to handle these relations. On ReadStartingWithUser, the function needs to return all backups or snapshots that are related to a parent instance or storage volume. This is used in the `ListObjects` call to the OpenFGA server, which is used by `(auth.Authorizer).GetPermissionChecker`. To do this, I have naively queried for all snapshots or backups in the project, and filtered out those that don't have the correct parent. This keeps the implementation simple and makes use of `GetEntityURLs`, which performs as few queries as possible. Further optimisation may be needed. Signed-off-by: Mark Laing <[email protected]>

We can now use the `can_view`, `can_edit`, and `can_delete` entitlements with instance backups and snapshots. We should do this so that our checks more accurately reflect the authorization model. Signed-off-by: Mark Laing <[email protected]>

The access handler was performing some logic to determine the location of the storage volume for use in the access check. This was based on whether the storage pool is remote, and if not, the cluster member where the volume is located. This commit removes that logic and adds a "location" field to `storageVolumeDetails` so that it can be used in the handlers. The logic for determining the location is modified to suit the call site. It is only set when the pool is not remote. Signed-off-by: Mark Laing <[email protected]>

The storage volume snapshot and backup access handlers need to share almost identical logic to the storage volume access handler. Including getting the storage pool, understanding if the storage volume is located on another cluster member, and so forth. This commit parameterises the function so that it can be used by the snapshot and backup entity types as well; creating and checking against the correct URL when called. Signed-off-by: Mark Laing <[email protected]>

Signed-off-by: Mark Laing <[email protected]>

We can now check `can_view`, `can_edit`, and `can_delete` against the backup/snapshot itself. We should do so to more accurately reflect the authorization model. Signed-off-by: Mark Laing <[email protected]>

Signed-off-by: Mark Laing <[email protected]>

test/main.sh

A suspected panic occurred in #14324. This happened twice but is intermittent and does not occur locally. The test suite has now passed for that PR 6/8 times. To surface panics more cleanly when investigating the issue, I added a python script to search LXD logs for a panic. Panics are logged at info level because the net/http package has a default `recover()` when a handler is run. The script is run after each test suite to ensure the test fails if it panics. The script is only run when `LXD_VERBOSE` or `LXD_DEBUG` are set however. This is because the panic log entry is at info level, and is not written if `--verbose` or `--debug` are unset.

tomponline · 2024-11-20T09:24:19Z

lxd/auth/drivers/openfga_model.openfga

Out of interest, what prevents this?

These entitlements cannot be assigned to identities, service accounts, or group members.

This is enforced by openfga. Notice that normally an entitlement has [identity, service_account, group#member] next to it. This means that those types can have that relation. When generating the map of entity type to entitlement, that's what we look for. So auth.ValidateEntitlement is checking that the entitlement can be granted to a group member.

In this case we're defined the relations such that they can only be granted via the parent instance. In fact, no types can be directly related to an instance snapshot except for instances.

tomponline · 2024-11-20T09:25:36Z

lxd/auth/drivers/openfga.go

@@ -134,11 +134,6 @@ func (e *embeddedOpenFGA) CheckPermission(ctx context.Context, entityURL *api.UR
 		return fmt.Errorf("Failed to parse entity URL: %w", err)
 	}

-	err = auth.ValidateEntitlement(entityType, entitlement)


do we have a test for trying to assign an invalid entitlement? (to check openfga does indeed block this)

Do you mean a check for say:

! lxc auth group permission add instance_snapshot c1-snap can_view || false # Cannot be assigned via API

Or a check where we manipulate the auth_groups_permissions table to insert a permission that is invalid from the perspective of OpenFGA?

Signed-off-by: Mark Laing <[email protected]>

tomponline · 2024-11-20T09:31:17Z

lxd/storage_volumes.go

 	defer func() {
+		// Only set the location if the pool is not remote.
+		if details.pool.Driver() != nil && !details.pool.Driver().Info().Remote {


just a nit, but you're calling .pool.Driver() twice here, would be more efficient to call it once and assign to a var locally right?

tomponline

LGTM, great commit and code comments. Thanks!

Addresses two final comments on #14324

markylaing added the Bug Confirmed to be a bug label Oct 22, 2024

markylaing added this to the lxd-6.2 milestone Oct 22, 2024

markylaing self-assigned this Oct 22, 2024

markylaing requested a review from tomponline October 22, 2024 12:56

github-actions bot added the Documentation Documentation needs updating label Oct 22, 2024

markylaing force-pushed the used-by-bug branch from a51ac04 to 86d9a32 Compare October 22, 2024 15:25

tomponline reviewed Oct 22, 2024

View reviewed changes

lxd/db/openfga/openfga.go Outdated Show resolved Hide resolved

markylaing mentioned this pull request Oct 24, 2024

Auth: Prune pending TLS identities #14261

Merged

markylaing force-pushed the used-by-bug branch 4 times, most recently from 4b56433 to bea0552 Compare October 25, 2024 10:22

markylaing mentioned this pull request Oct 25, 2024

Test: Add panic checker #14346

Merged

markylaing marked this pull request as draft October 25, 2024 12:26

markylaing force-pushed the used-by-bug branch 5 times, most recently from 320408e to 6b8b335 Compare October 31, 2024 15:02

markylaing added 13 commits November 18, 2024 10:28

lxd/auth: Run make update-auth.

1c84225

Signed-off-by: Mark Laing <[email protected]>

metadata: Run make update-metadata.

e9fd600

Signed-off-by: Mark Laing <[email protected]>

lxd/db/openfga: Use entity types for parent-child relations.

aaef11a

Signed-off-by: Mark Laing <[email protected]>

lxd: Update calls to the storage volume access handler.

90bdff3

Signed-off-by: Mark Laing <[email protected]>

lxd: Update storage volume snapshot and backup access checks.

e554262

We can now check `can_view`, `can_edit`, and `can_delete` against the backup/snapshot itself. We should do so to more accurately reflect the authorization model. Signed-off-by: Mark Laing <[email protected]>

test/suites: Add tests for storage pool used-by filtering.

44e4a30

Signed-off-by: Mark Laing <[email protected]>

Fix linter errors (revive: redefines-builtin-id).

0d3d40e

Signed-off-by: Mark Laing <[email protected]>

markylaing force-pushed the used-by-bug branch 2 times, most recently from ca1786e to 16fc177 Compare November 18, 2024 10:59

github-advanced-security bot found potential problems Nov 18, 2024

View reviewed changes

test/main.sh Fixed Show fixed Hide fixed

test/main.sh Fixed Show fixed Hide fixed

markylaing force-pushed the used-by-bug branch 2 times, most recently from a3a40ab to 0f45b26 Compare November 19, 2024 15:14

tomponline reviewed Nov 20, 2024

View reviewed changes

test/includes: Set ulimit before spawning LXD.

a5de620

Signed-off-by: Mark Laing <[email protected]>

markylaing force-pushed the used-by-bug branch from 0f45b26 to a5de620 Compare November 20, 2024 09:29

lxd/db/openfga: Update datastore comments to reflect change in model.

908c933

Signed-off-by: Mark Laing <[email protected]>

tomponline reviewed Nov 20, 2024

View reviewed changes

markylaing marked this pull request as ready for review November 20, 2024 09:31

tomponline approved these changes Nov 20, 2024

View reviewed changes

tomponline merged commit cd22ce3 into canonical:main Nov 20, 2024
10 of 11 checks passed

markylaing mentioned this pull request Nov 20, 2024

API: Storage pool used-by list follow up (#14324) #14500

Merged

tomponline added a commit that referenced this pull request Nov 20, 2024

API: Storage pool used-by list follow up (#14324) (#14500)

762d67a

Addresses two final comments on #14324

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Auth: Fix missing snapshots and backups from storage pool used-by URLs #14324

Auth: Fix missing snapshots and backups from storage pool used-by URLs #14324

markylaing commented Oct 22, 2024 •

edited

Loading

github-actions bot commented Oct 22, 2024

markylaing commented Oct 22, 2024

tomponline commented Oct 22, 2024

markylaing commented Oct 23, 2024 •

edited

Loading

tomponline commented Oct 23, 2024 •

edited

Loading

markylaing commented Oct 24, 2024

markylaing commented Oct 24, 2024

tomponline commented Oct 24, 2024

markylaing commented Oct 24, 2024

markylaing commented Oct 25, 2024

markylaing commented Oct 25, 2024

tomponline Nov 20, 2024

markylaing Nov 20, 2024

tomponline Nov 20, 2024

tomponline Nov 20, 2024

markylaing Nov 20, 2024

tomponline Nov 20, 2024

tomponline left a comment

Auth: Fix missing snapshots and backups from storage pool used-by URLs #14324

Auth: Fix missing snapshots and backups from storage pool used-by URLs #14324

Conversation

markylaing commented Oct 22, 2024 • edited Loading

github-actions bot commented Oct 22, 2024

markylaing commented Oct 22, 2024

tomponline commented Oct 22, 2024

markylaing commented Oct 23, 2024 • edited Loading

tomponline commented Oct 23, 2024 • edited Loading

markylaing commented Oct 24, 2024

markylaing commented Oct 24, 2024

tomponline commented Oct 24, 2024

markylaing commented Oct 24, 2024

markylaing commented Oct 25, 2024

markylaing commented Oct 25, 2024

tomponline Nov 20, 2024

Choose a reason for hiding this comment

markylaing Nov 20, 2024

Choose a reason for hiding this comment

tomponline Nov 20, 2024

Choose a reason for hiding this comment

tomponline Nov 20, 2024

Choose a reason for hiding this comment

markylaing Nov 20, 2024

Choose a reason for hiding this comment

tomponline Nov 20, 2024

Choose a reason for hiding this comment

tomponline left a comment

Choose a reason for hiding this comment

markylaing commented Oct 22, 2024 •

edited

Loading

markylaing commented Oct 23, 2024 •

edited

Loading

tomponline commented Oct 23, 2024 •

edited

Loading