Enabling persistent state across controller restarts

[casassg]

This is a bit of a stretch but we have been trying to see if there's some way via disks to do persistent state across restarts. I initially tried using controller-v5 disk_auto_delete: false hoping keeping the boot disk around would help but it seems setup.py doesnt overwrite cloud.conf if it finds it before (I assume, though there's no logs, the cloud.conf is the same as the previous restart).

I see theres some examples showing usage of additional disk, so I wonder if there's a setup where you can mount those into spool directories for slurm and to the DBD storage such that on restart slurm can just pick up where it left off.

[harshthakkar01]

Hi, Thanks for your question. It would be helpful if you can provide more details on what state you want to preserve.
For example,
If you are trying to persist Slurm file-based state then recommend option would be to go for Filestore with sufficient IOPS.
If you want to persist user data, then similarly a distributed file system e.g. Filestore, would be a better choice.
For Slurmdb, the recommendation would be cloudsql db.

[casassg]

My end goal would be to allow to restart controller and jobs would be picked up again after restart.

Why filestore vs a persistent disk which gets remounted on restart? Seems second you get faster IOPS and cheaper as well. Also do you happen to have any reference on how slurm-gcp structures these files? I know slurm itself uses mostly the spool folder if I remember correctly, but wondering wether the GCP version stores stuff elsewhere as well which should be kept?

User data we already have filestore as HOME + GCSFuse.

SlurmDB makes sense cloudsql though would be nice if we could have it on disk as well (cheaper for some use cases and simpler) though that also requires to tweak how mariaDB stores to make sure it uses a persistent disk?

[cboneti]

Are you just rebooting the controller? that should already work with any additional configuration.

[casassg]

@cboneti not rebooting, we delete and recreate (for example if we need to add a new startup script to the underlying image).

In theory from my knowledge of Slurm. If state files are kept on a persistent disk/nfs/whatever which is kept between instance deletions, and we dont do a slurm upgrade, Slurm controller should be able to pick up where it left off before it was restarted, right?

[jvilarru]

Hi, if you set the disk_auto_delete to false in the controller_instance_config (here) this will effectively never remove the disk of the controller, thus not removing the files stored in the statesavelocation, also the setup.py will see that the file /slurm/slurm_configured_do_not_remove is there and in consequence it will not try to run again. Just tested this with the simple_cloud example and it works as expected

[casassg]

@jvilarru I did try this in my cluster but it doesnt work if you need to add partitions (aka bc setup.py is skipped cloud.conf is left intact)

[mr0re1]

@casassg , I've just tried it with SlurmGCP-V6 blueprint, and attached disk survives (ghpc deploy / destroy / deploy cycle) as expected (with disk_auto_delete: false).
Could you please clarify what you're doing and your expectations.