Stream snapshots and restores

[neilalexander]

Observed behavior

Currently stream snapshots and restores work by archiving what's in the store directory for the stream on disk and restoring in-place. While this approach is simple, it has some problems.

When restoring a stream, we need to decompress the tar into a staging folder. It's not possible to apply limits at this point, as we can't know the full uncompressed size until we are in the process of decompressing it, and we also can't know which files may not be relevant for that version (i.e. as on-disk metadata has changed through the versions).

We also don't know that the layout of the restored files is optimal in the face of large numbers of interior deletes either, i.e. we may end up restoring lots of blocks with tombstones that could be compressed down into a contiguous range represented by a single delete range.

We also don't know if there'll be enough disk space free to satisfy both the JetStream reservations as well as the temporary space required for the restore staging directory.

Expected behavior

Ideally this wants a complete redesign, such that the stream snapshot is actually a stream of messages and their metadata, rather than the on-disk representation.

That way we can safely ingest messages into a new stream whilst obeying limits, updating the subject state etc inline correctly for the ingesting version, as well as removing any possibility of the archive containing information that isn't relevant to the new server.

Server and client version

All

Host environment

No response

Steps to reproduce

No response