Feature: Make draining configurable

[mschnepf]

In cases where COBalD reduces the demand below the current supply, TARDIS reduce the number of drones by killing requested but not started drones as well as draining running drones. As a result, no further jobs run on the drones, and the empty drones get shut down. However, by resource providers such as HPC clusters, the number of drones get reduced over time when no further drones are requested. For these resources, it is useful not to drain the allocated resources, so that drones which are partly used could accept further jobs over their requested runtime.

Three draining modes would be useful:

• no draining: not required drones accept further jobs
• draining: drone does not accept further jobs (current implementation)
• forced draining: drone does not accept further jobs and goes directly into DisintegrateState state

[giffels]

Generally a good idea. This needs potentially also adjustments to COBalD. Assuming COBalD is setting the demand of a resource to 0. The resource is moved from the hatchery to the mortuary, hence it is not considered anymore for calculating allocation and utilisation. Basically, COBalD forgets about it. https://github.com/MatterMiners/cobald/blob/5cf8bdb4ab31fb474cc48130817801c32d3783f7/src/cobald/composite/factory.py#L132

I would suggest to re-activate resource in the mortuary first in case new resources are needed because of growing demands. @maxfischer2781 can you comment about that, please?

[giffels]

Talking to @maxfischer2781 offline: The newest COBalD version already supports part of it. Resources in the mortuary are accounted for allocation, utilisation and supply.