GPU optimization

[colindaven]

From @maxdeest

runners * chunks_per_runner * chunk_size ~= 100000 * [max GPU memory in GB] * 2

For Ampere A100 GPU, max out the 40 GB GPU mem
--chunk_size 3000 --gpu_runners_per_device 8 --chunks_per_runner 512

[BeneKenobi]

Why do you use less gpu_runners_per_device in

guppy_on_slurm/runbatch_gpu_guppy.sh

Line 42 in d373459

gpu_params='--compress_fastq --num_callers 4 --gpu_runners_per_device 4 --chunks_per_runner 512 --chunk_size 3000'

? Where there problems with more runners?

[colindaven]

I don't think there was a problem with more runners, it was maybe just set for a different GPU. You need to play with the settings to reach the optimal speed for your GPU, and not everyone has a A100 with 40 GB GPU RAM.

Just monitor the GPU device usage and RAM using nvidia-smi.

Remember IO has a big impact, so storing fast5 files on SSD or other fast storage can decrease runtime a lot (36h to 24h for a human genome).