[catalystproject] Increase size of nodepool disk to increase space in /tmp

[jmunroe]

For bioinformatics workloads on both the latam and africa catalyst hubs, i think we should provide more local disk space per node. I'll suggest 400 Gb is a better number than the current 80Gb.

@jnywong and I can then work on guidance/tutorial for Catalyst communities that /tmp is the right place to store intermediate files which will be deleted automatically when the node shuts down.

The workflow I think we should recommend is
a) users should stage their 'input' datasets in object storage buckets
b) if workflows support reading directly from object storage then use that else make a local copy from object storage to /tmp
c) use /tmp for any intermediate files created during a workflow pipeline
d) push 'output' data sets to object storage for persistence
e) strongly encourage community users to keep home directory storage to under 1GB per user
f) discourage use of shared expect for smaller datasets (100Gb total per community)

Engineering Tasklist

• Increase disk size on GCP catalystproject-latam to 400 catalystproject-latam: Increase disk size to 400 #4215
• Increase disk size on AWS catalystproject-africa to 400 Bump disk size of the catalystproject-africa nodes #4225

[yuvipanda]

I've done this for catalystproject-latam, but someone else would need to do it for catalystproject-af

[sgibson91]

The catalystproject-africa cluster has now been bumped too. There was a user on a node though, so I tainted it and brought up a new equivalent. I will wait for the tainted node to drain and delete it before closing this issue.