want way to read data from a Disk through the API

[faithanalog]

I believe that right now, if I wanted to download a Disk, I would need to launch a VM with the Disk attached, and a guest image that will let me read the Disk data. That'll function, but it'd be nice to be able to download bytes from the Disk directly.

Crucible Pantry has the /crucible/pantry/0/volume/{id}/bulk-read endpoint, but it's not wired up to anything in the API right now.

I believe that we would need

• a way to put the Disk into an "export" state, a state I just invented, to tell pantry to attach to the Disk in read-mode
• an endpoint that maps to pantry's bulk-read to read a range of blocks from the Disk
• a way to take the Disk out of "export" state, to tell pantry to detach

It may also be possible to implementing reading data from an Image, but this would need some extra logic to decide when pantry attaches/detaches from the Image. Though without a convenience in the API, you could get the job done by creating a Disk from an Image, exporting the Disk, and then deleting it afterwards.

It would be pretty convenient if this exposed the Disk as an endpoint you can GET to download the whole file, while also responding to standard HTTP range-requests. That, instead of having to make multiple requests with a bespoke interface and put it back together client side (which would be similar to what uploading does). But I don't yet know enough about the API workings to know if that's feasible; I suppose if there's limits on API response sizes then it maybe isn't.

I'm intentionally vague saying "a way" for the disk states. I've noticed we already have 3 endpoints that all serve the role of transitioning a disk between pantry-related states

• /v1/disks/{disk}/bulk-write-start
• /v1/disks/{disk}/bulk-write-stop
• /v1/disks/{disk}/finalize

I'm not sure if we'd want to keep tacking on extra state-transition endpoints (bulk-read-start, bulk-read-stop) or do something else

[rmustacc]

I believe that right now, if I wanted to download a Disk, I would need to launch a VM with the Disk attached, and a guest image that will let me read the Disk data. That'll function, but it'd be nice to be able to download bytes from the Disk directly.

This has been a long standing ask for snapshots as part of backup efforts. For what you're doing if we it so you could just do streaming / range gets as you described on a snapshot how far would that get you?

It may also be possible to implementing reading data from an Image, but this would need some extra logic to decide when pantry attaches/detaches from the Image. Though without a convenience in the API, you could get the job done by creating a Disk from an Image, exporting the Disk, and then deleting it afterwards.

I think if we end up with snapshot download, this should be pretty straightforward to do for images as well at least conceptually from the user perspective. Both of these are appealing because they're just static things that may want to be accessed and often are connected due to read-only parents anyways.

FWIW, the thing that you described about a raw read API is not dissimilar to what I once pitched @jmpesp in terms of a raw write API as a way for creating images (create a raw disk, attach it in an API sense, and do your writes. Then snapshot and promote to image like the main normal path). I think if we are going to do something like this for reads, then we should be doing it for writes. The idea of something like this isn't dissimilar to what the EBS Direct APIs are and has been asked about before by some folks.

I don't know if the existing APIs are necessarily quite right as they have different semantics given it's basically an import-only API, where as I think what you want and what the write form of this really wants is effectively mounting it via one call via HTTP and then being able to do arbitrary read/writes.

[faithanalog]

Interesting. So what I have learned is that today while we do have random-write functionality, it's only available as the first thing you do with a disk.

The way importing works today is that you have to create the disk with the IMPORTING_BLOCKS source, which puts it into Import mode. You cannot put a disk into import mode from detached mode.

However, while it is in import mode, you do have full random-write capabilities. We already take advantage of this to skip writing zeroes by just not sending writes for those blocks. This is also the thing that makes our CLI uploads work: our CLI distributes the upload across 8 worker threads, with no synchronization between them, so it's entirely possible for writes to land out of order.

I believed you could put a disk back into this import state from Detached, but no:

echo '{}' | oxide api -X POST --input - '/v1/disks/e2d69a29-eb89-4618-a769-9db653403a74/bulk-write-start'
error; status code: 400 Bad Request
{
  "error_code": "InvalidRequest",
  "message": "cannot set disk in state Detached to \"importing_from_bulk_writes\"",
  "request_id": "82d49f43-36af-4959-b6c2-5c907e049fe0"
}

---

As far as I understand it, a snapshot is basically a ZFS snapshot of a crucible region without a running downstairs, while a Disk or an Image have a Downstairs associated with them. (it's something that can be attached to). In other words, to serve data from a snapshot on the API, we would need to start up a Downstairs. I believe this is what the "Create Image" button on a snapshot does today- turning on a Downstairs process to read the region data, so that it can be used as the read-only parent of other disks.

So we could duplicate that process and start our own Downstairs. Or we could make it so reading happens from an Image instead of a Snapshot. In either case, we would need that state tracking to decide when pantry should detach from the Downstairs. And if we are reading from a Snapshot, not an Image, we'll have created a Downstairs that we need to destroy too.

So if I understand this right, implementing exports from an Image seems like it would be a little bit simpler than a Snapshot. But I think either way is about the same.

But yes, those would both work for me. Mostly I am interested in downloading VM disks so I can boot them somewhere else, or so that I can verify certain things about their contents. Doing that through a snapshot or image would do the trick just fine.

[rmustacc]

As far as I understand it, a snapshot is basically a ZFS snapshot of a crucible region without a running downstairs, while a Disk or an Image have a Downstairs associated with them. (it's something that can be attached to). In other words, to serve data from a snapshot on the API, we would need to start up a Downstairs. I believe this is what the "Create Image" button on a snapshot does today- turning on a Downstairs process to read the region data, so that it can be used as the read-only parent of other disks.

At the moment a snapshot is, I guess, but the longer term design here is to serialize a snapshot into its own volume to decouple and simplify accounting. Otherwise you're limited in the number of snapshots you can have based on churn in the disk and surrounding allocation capacity. So at that point it probably doesn't look too different from images (albeit that isn't how things work right at this moment).

But yes, those would both work for me. Mostly I am interested in downloading VM disks so I can boot them somewhere else, or so that I can verify certain things about their contents. Doing that through a snapshot or image would do the trick just fine.

Thanks for that context. That overlaps a lot with the basic snapshot download use cases we got.