benchmarks with snakemake

[ctb]

from code added in 2b78b96, this:

srun -p high2 --time=5:00:00 --nodes=1 --cpus-per-task 32 --mem 50GB --pty /bin/bash
snakemake -j 32 --snakefile benchmarking.snakefile 
csvtk concat benchmarks/*.txt | csvtk cut -t -f "h:m:s",cpu_time,max_rss

yields:

h:m:s   cpu_time        max_rss
0:23:14 40626.79        16106.77
0:23:14 38643.81        17553.49
0:22:37 37936.37        18257.87
0:25:05 39209.39        16107.48
0:26:36 37791.57        14046.44

[ctb]

that's for 1000 queries against 10,000 metagenomes (random queries, random metagenomes).

so about 32 hours for 1000 queries against 800,000 metagenomes using 32 threads.

with a standing memory usage of under 20 GB.

[ctb]

note to self: pick largest 10k metagenomes, see what happens :)

[ctb]

further note to self: meaning of columns in snakemake benchmarks: https://www.google.com/search?client=firefox-b-1-d&q=snakemake+benchmarking

[ctb]

note to self: pick largest 10k metagenomes, see what happens :)

h:m:s   cpu_time        max_rss
9:54:20 1022695.90      59988.88