HPC Episode

[multimeric]

• Added an optional HPC episode, which uses Slurm as an example
• This has been written using the current plan writing system. Once/if Code Re-Use #19 is merged, some of this will need re-writing to remove redundancy

[github-actions]

🆗 Pre-flight checks passed 😃

This pull request has been checked and contains no modified workflow files or spoofing.

Results of any additional workflows will appear here when they are done.

[edoyango]

A couple comments after having gone through the preceding episodes.

[joelnitta]

@multimeric

I realized that I needed to update renv.lock in main, so I did that and cherry-picked the commits here (4eb5f06, 9489bea), as well as make one more update to renv.lock (I think due to using CRAN instead of the carpentries r-universe, e012aaf).

Can you check if you can still build the HPC lesson locally? I can't because I don't have access to a computer with slurm.

(sorry - I realized cherry-pick may not have been the best approach here. Feel free to roll those back and re-base instead if you want)

[multimeric]

I can do that, but can we also test if it's working on the CI? I'm not sure if it actually runs the lesson build for pull requests at the moment?

[joelnitta]

Sure - I assume you mean implementing suggestion 3?

Add slurm to the CI server, e.g. https://github.com/koesterlab/setup-slurm-action

[multimeric]

Oh, I already added that to .github/workflows/sandpaper-main.yaml, but I'm not sure if it will even run that workflow for a pull request. Also it doesn't like the fact that I've changed the content and the workflow in the same PR.

[joelnitta]

How about this: I just set up another repo and merged hpc into main over there, let's see how it goes.

GH action is running now... https://github.com/joelnitta/targets-hpc/actions/runs/9867906104

hm, seems to be hung up, I predict it will git killed by time-out. I've actually seen this behavior before, I'm not sure what's causing it but I don't think it is because of this PR...

[multimeric]

No, I think it is a fault in my PR. It's failing at the second workflow which uses different memory requirements, which is likely the issue. I've reduced the memory requirements, but I now realise that it's guaranteed to fail at the GPU workflow since the mini cluster isn't going to have a GPU node.

[joelnitta]

ah... so perhaps CI is not an option then?

[multimeric]

Well not for the last workflow, but that's inside a challenge solution so even if I just paste in the pre-computed result there I don't think that's too bad.

[joelnitta]

Sounds good, let's try that then. If you push the change here, I can try it again over on my other fork.

[joelnitta]

@multimeric

markdown build seems to be hanging

[multimeric]

Yeah, feel free to cancel it. I can't.

[joelnitta]

canceled

[multimeric]

I believe this failed because the lesson involves submitting a job that requires 2 GB of RAM, but the koesterlab/setup-slurm-action only provides a single node with 2000 MB of RAM and so it hangs forever while waiting for sufficient memory to become available. I'll fork the action to allow increased memory.

[multimeric]

Actually can you give me the permission to cancel builds so that I can play around with this?

[multimeric]

Okay, putting this on hold until koesterlab/setup-slurm-action#7 is merged. Technically we could use my fork of the action, but there's no hurry.