Feature Proposal: AAE Exclude Bucket Property

[martinsumner]

Background

AAE is currently all-or-nothing in a cluster - enabling TictacAAE will turn on AAE for all buckets.

There may be certain buckets where data resilience is less important, and so it would be beneficial to exclude these buckets from AAE - so that the AAE caches do not contain knowledge of these buckets, and those buckets could have other properties that would otherwise conflict with intra-cluster AAE and inter-cluster full-sync.

This would mean that real-time replication could be optional for those buckets, as a failure to replicate would not lead a full-sync failure. So if a specific bucket was to be local, whereas the rest of the buckets were to be replicated - AAE Exclude bucket property could be enforced for that bucket, the bucket could be excluded from the real-time repl config - and full-sync would still work.

This would also allow for a TTL to be applied for that specific bucket (without the need for an eraser job), as this would not cause AAE failures after tree rebuilds

Proposal

Add an aae_exclude bucket property, which is false by default.

Keys could still be repaired by read_repair, and still included in hinted handoffs (and transfers). So this is suited to short-lived data (i.e. less than 3 months), where long-term entropy is not an issue.

As an extension there could be a ttl option possible on a bucket, which may be enabled if and only the aae_exclude option is set - as keys could now be auto-expired, without impacting AAE.

Design

Work ongoing:

• In parallel mode AAE, should the key still be added to the keystore, so that stats-style AAE queries would still work (i.e. the keys are excluded from the cache but not the store).
• In this case how should such results be filtered from fetch_clocks style queries, in the fold, or should it be the responsibility of the client to exclude. This might be a configurable option for the fold.
• Should there be a persistent term to add excluded_buckets, which is potentially updated every time there is an update to bucket properties (if the list of excluded buckets has changed). This would allow exclusion in folds to be a simple lists:member/1 check.

Alternative Design Ideas

.If a per-bucket TTL property is to be added, then it will probably need to be a leveled-only feature. For bitcask, an expiry setting is store-wide, and there is not an obvious and simple way of making it more specific - and certainly getting the same behaviour as with leveled (for example setting expiry per object at PUT time).

For this reason, it may be preferable to stick with eraser jobs as an answer to expiry, rather than try and make significant changes to bitcask. Currently eraser depends on operator prompting, but perhaps instead there should be a riak_kv_eraser_manager - that would look for buckets with expiry enabled, and periodically trigger eraser activity to garbage collect these buckets.

Testing

tba

Caveats

...

Pull Requests

tba

Planned Release for Inclusion

tba

[martinsumner]

With regards to naming, there is the question of whether the bucket property should be named by "what it does" or " what we expect it to be used for"?

For example, this is a feature intended for temporary objects that don't need the usual anti-entropy (and potentially inter-cluster resilience): so the bucket property could be {temporary, true|TTL} - what it is expected to be used for. This might, by some, be deemed more user friendly.

But perhaps for expert users, that keep a mental model of how things work, a name which is more explicitly about what it does might be preferable (e.g. the proposed aae_exclude). Also such a bucket property might be used as a workaround for some other issue in the future, in which case it would be confusing for it to be named after what we expected it to be used for.

[tburghart]

I think there's an argument to be made for "both" - aae_exclude does precisely what it says, while temporary might serve as a) a readable flag; b) an umbrella that implicitly turns on aae_exclude and potentially other settings; c) a hint to a bucket-type-aware backend as to how to store the object; d) etc ...