stateful deployments: use TaskGroupVolumeClaim table to associate volume requests with volume IDs

[pkazmierczak]

We introduce an alternative solution to the one presented in #24960 which is based on the state store and not previous-next allocation tracking in the reconciler. This new solution reduces cognitive complexity of the scheduler code at the cost of slightly more boilerplate code, but also opens up new possibilities in the future, e.g., allowing users to explicitly "un-stick" volumes with workloads still running.

The diagram below illustrates the new logic:

     SetVolumes()                                               upsertAllocsImpl()          
     sets ns, job                             +-----------------checks if alloc requests    
     tg in the scheduler                      v                 sticky vols and consults    
            |                  +-----------------------+        state. If there is no claim,
            |                  | TaskGroupVolumeClaim: |        it creates one.             
            |                  | - namespace           |                                    
            |                  | - jobID               |                                    
            |                  | - tg name             |                                    
            |                  | - vol ID              |                                    
            v                  | uniquely identify vol |                                    
     hasVolumes()              +----+------------------+                                    
     consults the state             |           ^                                           
     and returns true               |           |               DeleteJobTxn()              
     if there's a match <-----------+           +---------------removes the claim from      
     or if there is no                                          the state                   
     previous claim                                                                         
|                             | |                                                      |    
+-----------------------------+ +------------------------------------------------------+    
                                                                                            
           scheduler                                  state store                           

Supersedes #24960
Fixes issues found in #24869

[tgross]

This is looking great, @pkazmierczak!

[tgross]

Approach looks good! One more item to clean up in the feasibility check, I think.

[tgross]

Somewhere along the way we dropped that this set of claims should only be the ones for this task group. This would be all claims for all task groups, which we don't care about.

[tgross]

Looks great! Just a couple small items left to address and I think this is ready to merge.

[tgross]

LGTM! 👍