Update SLURM builder

[jarlsondre]

Summary

Updates to the itwinai SLURM builder. See issue #288 for more information.

Brief overview:

• Add SLURM Builder to CLI
• Use tempdir when --no-save-script is active
• Add some documentation for the builder in the existing SLURM command documentation
• Add some tests for the SLURM builder
• Use pytest's tmp_path instead of tempfile's TemporaryDirectory wherever possible
• Remove old slurm.sh and runall.sh files from use cases
• Add training command with overridable string, removing the need for separate slurm.py files in EURAC and Virgo
• General clean up such as changing the type hints in the TorchTrainer etc.
• Possibly some more small stuff 😁

Related issues : #288, #316 #318, #319, #320, #324, #325

[matbun]

our formatters will keep fighting on this forever hahaha

[jarlsondre]

Haha yeah I think I have my line length a bit shorter than yours. Mine doesn't use the project-wide one I think

[matbun]

it would be nice to replicate arguments and options of the argument parser here as well. This would generate a nice help page, helping the users to understand what options they can use and what type they receive. You can take inspiration from exec_pipeline

[jarlsondre]

This is possible and might be better since the Typer help page is pretty, but since I am using the argparser, doing -h actually works atm and will tell you all the arguments that exist. The format will be slightly different from Typer's, though.

[matbun]

I see two limitations with this solution. First, I guess that if you use --help instead of -h, this help page will not appear -- a bit inconsistent and may generate confusion. Second, the CLI reference is not generated in the dedicated docs page

[jarlsondre]

Regarding the first limitation: This is possible to change by simply setting add_help_option=False.

The CLI reference might not be as easy to fix. We could just add them as arguments, but there are 23 of them in total. Looking at the other Annotated variables they take between 1 and 8 lines each, where most seem to take 5-6. This would result in around 100-150 lines of variables. Certainly doable and might be the best fix, but certainly worth thinking about. What do you think? Should I add them? 😄

[jarlsondre]

I suppose the biggest downside would be that we would have all the descriptions in two places, as well as all the variables. We could solve this by removing the descriptions from the argparser and maybe having an automatic test (in tests/) that checks that the params are the same between the argparser and the function? Just thinking out loud here.

Another option is to change the parsing mechanism to use the newest parser. That way we wouldn't have the whole double thing. The disadvantage is that it could potentially be a lot of work for something that already works the way it is.