Updated handling of SLURM scripts

[jarlsondre]

Summary

Currently there are a lot of duplicate bash scripts for submitting SLURM jobs. This is inefficient and hard to maintain, so this should be rewritten. As these scripts benefit from reusability and thus slightly more complex scripting, the suggestion is to do this in Python instead. Python is generally better suited for more complex scripting tasks and everyone in this project is familiar with Python.

Using libraries like PySlurm is possible, but this would create a slight learning curve for anyone not familiar with this library, and it is far from an industry standard. Therefore I propose that we use write the code for this ourselves. This also means that we require one less dependency, which is always an advantage.

Wanted features:

• Ability to specify your slurm job in a yaml file, so that everything can be changed in a single spot
• Centralized commands that are common to the different use cases

Ideas (food for thought):

• In each use-case folder there should be a config file that you use to specify your job
• There should be some way to override functions for a specific use case, if need be
• Avoiding over-abstraction is important. The main goal here is a higher level of reusability, not an entire SLURM library.

[matbun]

I would suggest to add in the future plans:

• removal of config "template" from the itwinai's library (src/itwinai/slurm/slurm_config.yaml), which could be potentially embedded in a module as a dictionary.
• integration of SLURM config file with the existing "pipeline" config file, to control everything from the same place.
• integration with itwinai CLI (somehow related with the above)