Support PersistentVolumeClaimRetentionPolicy for statefulsets

[SatyaKuppam]

As of Kubernetes v1.32 PersistentVolumeClaimRetention policy is enabled by default for statefulsets and is behind a feature gate for v1.31. This feature provides a K8s native way to manage orphaned PVCs, that is pretty similar to the existing deleteOrphanPvc config in the Druid CRD. By using this policy, we can control how orphaned PVCs will be treated:

persistentVolumeClaimRetentionPolicy:
   whenDelete: <true|false>
   whenScaled: <true|false>

Some folks might prefer persistentVolumeClaimRetention over deleteOrphanPvc because:

• allows more granular control because its at the statefulset level, instead of deleteOrphanPvc which affects the entire Druid cluster. For example:
  • we might want to Retain historical PVCs because loading data from S3 might be expensive
  • but we might want to delete middlemanagers' PVCs on scale-downs
• dont have to be bound by the deleteOrphanPvc's delayed deletion of 60s (ref)

I want to add this functionality in the nodeSpec if folks think its useful and I want to align on the following:

• this makes sense only for statefulsets and not deployments
• if the deleteOrphanPvc is set, then the persistentVolumeClaimRetention configuration is automatically ignored to allow for backward compatibility
• this configuration only makes sense if StatefulSetAutoDeletePVC feature flag is set (ref).
  • I am actually not sure how to check for this. Any ideas?
• this feature is only available from v1.31, but from the compatibility matrix (ref), the latest we support is v.1.30.1.
  • What do I have to do to use the > v.1.30 API?