New copy 'placement' dataset property

[tonyhutter]

@behlendorf and I got to talking about #16073, and we had an idea for a more elegant way to handle backing up special device metadata to the pool. Imagine if we had a dataset property called placement that could control where each of the 3 DVA copies landed.

placement:   Set the dataset allocation class placement preference for each of the three DVA copies.

placement=[ALL_COPIES]|[[COPY1[:COPY2[:COPY3]]]

Where the values for ALL_COPIES and COPIES* are string values:
	auto: Let ZFS decide where the data should land (the current behavior in `master`)
	pool: Write to the normal pool
        special: Write to the special allocation class device

	auto = auto:auto:auto - The current behavior in `master`
	pool = pool:pool:pool - Write to the main pool
	special = special:special:special - Write to special/dedup, even if files are enormous.  
	
	auto:pool:auto - Backup all metadata to the pool (since we know that all metadata internally has copies>=2).
	resilient - Alias for "auto:pool:auto" above
        
        auto:pool:pool - Backup all metadata to the pool, but if copies=3, it puts at least two copies on the pool.

Examples:

       # Write the first copy of the data to the special allocation device, the 2nd copy to the pool, and let ZFS
       # decide where the 3rd copy should go
       zfs set placement=special:pool:auto tank/important_dir1
       zfs set placement=special:pool tank/important_dir1

       # Write everything to the special device because I want it to be fast
       zfs set placement=special tank/www

       # Let ZFS decide where to put my data, but always backup metadata to the pool
       zfs set placement=resilient tank/home

These values are only a "preference", and ZFS can override them. Like, if if you do placement=special and there's no more special disk space, then write to the pool instead (which copies our current behavior in master).

To create a pool with all the metadata backed-up to the main pool:

zpool create -O placement=resilient tank /dev/sda special /dev/nvme1n1

placement would override special_small_blocks for a dataset it unless it's set to "auto" ("auto" would honor special_small_blocks and use all our normal data placement logic). So if you set placement=special and then wrote multi-GB ISOs into the directory, it would all go to special, regardless of what special_small_blocks was set to. This could be used as a way to get certain datasets to go completely on special devices (to reduce load times for games, serving a busy directory on a webserver that doesn't fit in RAM, etc).

Alternate names for "placement": "copy_placement" "allocation_preference" "placement_preference" "copy_preference"

[tonyhutter]

So then the next question becomes, what happens if my special device completely fails?

As a prerequisite, let say we make sure 100% of the critical metadata for importing the pool always has at least one copy on the pool. @behlendorf thinks this metadata may only be a few megabytes. I'm not talking about the file metadata here, I'm talking about just enough critical metadata to have '/tank' imported with literally no files in it.

However, all is not lost, as every dataset that has placement=resilient (aka, has a metadata copy on the pool) will also show up after import.

Let's run though an example:

$ zpool create tank /dev/sda special /dev/nvme1n1

<install Fedora on pool>

# Mark the directories we really really care about with
# placement=resilient.  We know that their metadata
# will have 2 copies written by default, and since we set
# placement=resilient then we know one copy will be
# on the pool.
zfs set placement=resilient tank/home
zfs set placement=resilient tank/etc

$ ls /tank
afs  bin  boot  dev  etc  home  junk  lib  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tank  tmp  usr  var

< we add users, write data to the pool, years pass >

< special disks fail >

$ zpool import tank
Warning, your special allocation vdevs are missing but you have
feature@copy_placement enabled, so some datasets that have a copy
of their data on the pool may be recoverable.  Pass -f to import
anyway.

$ zpool import -f tank

# The two directories we marked with placement=resilient show up, but nothing else
$ ls /tank
etc  home

[tonyhutter]

The advantages of this over #16073:

1. It still gives us the ability to backup the special device metadata to disk
2. No clunkly "all or nothing" pool or vdev properities.
3. No need to suffer dRAID write inflation on the entire pool. You can just select the datasets you really care about for backup.
4. Gives users a way to allocate datasets specificially on a special class disk.
5. placement would also work on zvols since they use dataset properities.

[rincebrain]

I don't hate this as a premise, but it feels like you're going to wind up with confusing outcomes.

1. how does this interact with e.g. special_small_blocks?
2. does this introduce a feature flag to block importing on older systems since they might violate the policy?
3. if you have this policy, what does it mean if you hit BRT/DDT with an entry? Conceptually, you don't care since it's not an allocation, but if you set something like resilient, and your DDT is all on a separate vdev, you could argue that the ddt vdev failing breaking your pool would be a very surprising outcome (but what's the alternative, exactly, keep the entire DDT redundant on the spinning rust?), and that segues into...
4. I said metadata is all copies>=2, and that's true...unless you change redundant_metadata, so auto:pool:auto still doesn't save you in that case, and could lead to surprising outcomes for people who might have expected this to override that. (I'm not trying to weigh in on whether they're right to expect that, just that I would expect that kind of complaint to come from something like this.)

Conceptually, my two remarks would be:

1. If we think about the idea of "policy per DMU object type" and "policy for all DMU objects" as a scale with two extremes, I'd suggest sliding the notch toward the former slightly - e.g., at a minimum, we'd probably want ways to specify the behavior for {all metadata, dedup metadata, L0 volume, L0 file} (since we've discussed people sometimes wanting to have distinct policies for volumes versus filesystems), and maybe even {volume metadata, filesystem metadata, pool metadata, dedup metadata, L0 volume data, L0 file}. (This is nasty and requires a bunch of knowledge about internals to make use of effectively, but tragically, I don't think that's avoidable - the fundamental premise is trying to know enough about the innards and telling it how to make effective use of the storage available for the use case, the knowledge is unavoidable.)
2. I keep going back to wanting something more involved, e.g. take a few of the bp bits available, use them to indicate "DVA[1/2] means something different", like with encryption, and then specify an encoding for "DVA[1] is in this lookaside table", possibly with "with ID so and so", and then async do the DVA[1] write and enter it there, so we can dance around the immutability requirement. (Alternately, I don't hate the notion of having a bit for something like that but instead we encode it as something like a ZIL entry, so we allocate space on the slow vdev but don't actually block on the write going out before we return...)

Mostly because I think a lot more people would find this useful if it didn't block the write path, and I don't think it's completely impossible to make that dance work well enough for many people.

[rincebrain]

More generally, I feel like ZFS is increasingly looking like it's going to want some kind of zconfig nonsense for managing more complex interdependent structured configurations like this, or perhaps if you wanted different compression types for different recordsizes, or anything else where fitting it in a single line is gross.

(It'd also be useful, conceptually, if we had a standardized structure for tunables across platforms, e.g. zconfig tunable arc_max 100G rather than /sys/module/zfs/parameters/... or sysctl ......)

I mention that mostly because it seems like this would be a good argument for it - you basically want a k-v of type-setting, of variable length, which is not a thing that maps well to the current interface as a readable thing (IMO).

[tonyhutter]

1. how does this interact with e.g. special_small_blocks?

So if placement=auto, then it would honor special_small_blocks, but if placement=special, then it would override special_small_blocks.

2. does this introduce a feature flag to block importing on older systems since they might violate the policy?

Yes it would. You could still mount it read-only on older systems though.

4. I said metadata is all copies>=2, and that's true...unless you change redundant_metadata, so auto:pool:auto

Yea, I agree it could be confusing for users who set placement=resiliant ("look I'm being nice and safe!") and also redundant_metadata=none. We could print a warning if they happen to pass them both at the zfs set time, but that's only going to catch a handful of cases. We can cross reference redundant_metadata and placement in the man pages to increase awareness. Also, if you're the kind of user who is changing redundant_metadata, you probably have read the docs about it.

3. if you have this policy, what does it mean if you hit BRT/DDT with an entry?

This is a really good point. The BRT & DDT are pool-wide metadata so we can't control them on a per-dataset basis.

It gets even trickier:

• You can control if the DDT is considered special data with the zfs_ddt_data_is_special module option (default on).
• There's no equivalent module option for BRT.
• You can control if the file metadata is considered special data with the zfs_user_indirect_is_special module option (default on).

zfs_user_indirect_is_special is hard since it sort-of conflicts with the meaning of placement. Like, what does it mean to have a dataset with placement=special and also the zfs_user_indirect_is_special=0 module option set.

Between placement, copies, special_small_blocks, redundant_metadata, zfs_ddt_data_is_special, and zfs_user_indirect_is_special we basically have a patchwork of policies that don't mesh well. I'm not sure if it's insurmountable, but I need to give it some more thought...

[Haravikk]

We could print a warning if they happen to pass them both at the zfs set time, but that's only going to catch a handful of cases.

Could you not have zfs set redundant_metadata=none check the value for placement, and zfs set placement=auto:<copy2>:<copy3> check the setting for redundant_metadata? This should allow it to give a warning if either of these is set separately (rather than at the same time), but to make it come up in as many cases as possible it might need to be checked recursively so that changes to a parent dataset will produce a confirmation step if it could put child datasets at risk. Usually it should just be a case of encouraging the user to set placement at a higher level.

Regarding BRT and DDT, could we issue a similar warning/confirmation step if these features are enabled at a level incompatible with the placement setting (and vice versa)? i.e- if you try to set placement on an encrypted dataset that doesn't share its key (not just encryption root) with any others, then IIRC this isn't a problem because it isn't possible for its data to be cloned elsewhere anyway (unless you clone the dataset and change the settings). However if you do the same for an unencrypted dataset or one that shares its key with others (not currently possible, but was discussed as a future option IIRC?) then the user would need to confirm they aware of the risk of the change.

Do you think it's feasible to build a generic way to handle such incompatible (or at least risky) settings? Some mechanism where setting a particular value checks other settings, and if any of them is a mismatch (optionally with recursion up or down as required) then the change will either be rejected (not possible) or a confirmation is required (user is told which options are in conflict, possibly why it's risky, and given the choice to continue or cancel). If something like that doesn't already exist it seems like it could be useful for other potential properties.

Long term we'll definitely want a way to control placement of BRT and DDT as well, ideally in a consistent way. Perhaps if there were a brt_placement and ddt_placement setting which zfs_ddt_data_is_special etc. could just become proxies for (and eventually retired)? Part of the challenge though is going to be the fact that either of these features could have been setup in a way that was unsafe, and later changed, so the settings alone may not be enough to inform the user.

It could always just be a case of making the documentation as complete as possible, and advising that settings like resilient be used by default in the dataset root, and overridden as required. As users should be best placed to know if they need to change it or not.

Also wanted to add that while I'm generally in favour of resilient being the default for a newly created pool's root dataset, it might be worth adding a check to zpool add <pool> special and zpool attach <pool> <special vdev> to remind the user to change their placement setting(s) if the special device is created with/gains redundancy. For example, "As your special device [now] has redundancy, the extra protection of placement=resilient may no longer be required" so similar?

That said, all of this is extra sugar to help inform users, in terms of what is required for the addition of this feature, it may be enough to just ensure the documentation is as complete as possible, and set the default to placement=auto so it's an opt-in feature, as this will be a lot easier to implement (just the basic feature + documentation).

[tonyhutter]

So ultimately I don't think that a per-dataset property is going to work as a replacement for #16073, due to the dedup data being per-pool.

I've decided to go with a new, simplied version of my old PR instead. The new PR is here: #16185