Reverse incremental ZFS receive

[wrouesnel]

System information

Type	Version/Name
Distribution Name	Centos
Distribution Version	8.1
Linux Kernel	4.18.0-147.8.1.el8_1.x86_64
Architecture	x86_64
ZFS Version	0.8.2-1

Describe the problem you're observing

When synchronizing two machines with ZFS snapshots, (A and B), if an accident or outage causes A to wind up without a common snapshot ancestor with B, then there is no way short of juggling filesystems to recover the incremental send behavior nicely.

What I would like to be able to do is zfs receive a full filesystem stream (i.e. the result of zfs send /rpool@snapshot1) and have ZFS receive receive it as a snapshot of the same filesystem on the remote side, without wiping out the other snapshots on the system (particularly useful if you're serving data off of snapshot clones).

Basically with two filesystems looking like:

$ zfs list -t snapshot
NAME              USED  AVAIL     REFER  MOUNTPOINT
apool@snapshot1     0B      -       24K  -
apool@snapshot2     0B      -       24K  -
apool@snapshot3     0B      -       24K  -
bpool@snapshot0     0B      -       24K  -

The command

zfs send apool@snapshot3 | zfs receive bpool

would result in:

$ zfs list -t snapshot
NAME              USED  AVAIL     REFER  MOUNTPOINT
apool@snapshot1     0B      -       24K  -
apool@snapshot2     0B      -       24K  -
apool@snapshot3     0B      -       24K  -
bpool@snapshot0     0B      -       24K  -
bpool@snapshot3     0B      -       24K  -

Based on my understanding of how snapshots are implemented under the hood for ZFS, this would functionally have to be implemented as though there was a totally empty snapshot which then received a bunch of blocks. But, it would mean that a single filesystem chain can be maintained, which vastly simplifies the use-case of serving fixed content (backup snapshots) out of the system.

As things currently stand, if you wind up in this situation the only recovery mechanism is to delete all the snapshots on the destination filesystem (which you may not want to do if they represent part of an offsite rollback or backup strategy that have since been cleared from the apool side) or juggle filesystems (which leaves you with separate filesystem trees).

[ahrens]

This should be possible today:

man zfs-receive
...
     zfs receive [-FhMnsuv] [-o origin=snapshot] [-o property=value]
         [-x property] filesystem|volume|snapshot
       -o origin=snapshot
           Forces the stream to be received as a clone of the given snapshot.
           If the stream is a full send stream, this will create the filesys‐
           tem described by the stream as a clone of the specified snapshot.

So in your example, you can do zfs send apool@snapshot3 | zfs receive -o origin=bpol@snapshot0 bpool/recvd. Ideally you'd then zfs promote bool/recvd, rename it to replace the original and then destroy the old filesystem. In your example, that might be a bit messy since you are talking about the topmost filesystem. Hopefully in practice you have bpool/fs@snapshot0.

[wrouesnel]

Trying this out with, this doesn't appear to work:

So the situation on my workstation...

$ zfs list -t snapshot
NAME                            USED  AVAIL     REFER  MOUNTPOINT
rpool/home/will/tmp/fs1@0         0B      -       96K  -
rpool/home/will/tmp/fs1@1         0B      -       96K  -

$ sudo zfs list | grep tmp/fs
rpool/home/will/tmp/fs1         96K   249G       96K  /home/will/tmp/fs1
rpool/home/will/tmp/fs2        104K   249G      104K  /home/will/tmp/fs2

Based on the man pages this is the command I would have expected to work (which prompts the original question - it feels like it implies that I should just be able to "receive to snapshot")

$ sudo zfs send -Lce rpool/home/will/tmp/fs1@0 | sudo zfs receive -o origin=rpool/home/will/tmp/fs2@0 rpool/home/will/tmp/fs2/rec0
cannot receive: specified origin fs (rpool/home/will/tmp/fs2@0) does not exist

Taking that error message, I then get this:

$ sudo zfs send -Lce rpool/home/will/tmp/fs1@0 | sudo zfs receive -o origin=rpool/home/will/tmp/fs2 rpool/home/will/tmp/fs2/rec0
cannot receive new filesystem stream: incompatible embedded data stream feature with encrypted receive.

Taking that error message I then get this:

$ sudo zfs send rpool/home/will/tmp/fs1@0 | sudo zfs receive -o origin=rpool/home/will/tmp/fs2 rpool/home/will/tmp/fs2/rec0
cannot receive new filesystem stream: invalid backup stream

Basically as far as I can tell, this is not currently possible. The use of the -o origin=<> flag is a bit mysterious too.

[ahrens]

I'm not sure about the later 2 errors, but I understand the first one:

$ sudo zfs send -Lce rpool/home/will/tmp/fs1@0 | sudo zfs receive -o origin=rpool/home/will/tmp/fs2@0 rpool/home/will/tmp/fs2/rec0
cannot receive: specified origin fs (rpool/home/will/tmp/fs2@0) does not exist

According to the output you provided from zfs list -t snapshot , there is no snapshot rpool/home/will/tmp/fs2@0, so the error message is accurate. You would need to create the snapshot with zfs snapshot rpool/home/will/tmp/fs2@0 and then re-run the command.